Skip to content

Create a new version of MAGIC; called Magic v2#24163

Open
leonerd wants to merge 2 commits intoPerl:bleadfrom
leonerd:magic-v2-phase1
Open

Create a new version of MAGIC; called Magic v2#24163
leonerd wants to merge 2 commits intoPerl:bleadfrom
leonerd:magic-v2-phase1

Conversation

@leonerd
Copy link
Contributor

@leonerd leonerd commented Feb 3, 2026

The existing MAGIC and MGVTBL structures are insufficiently extensible, because they lack any embedded type information, or any version numbering by which new abilities can be added in a backwards-compatible way. Commit 6d97c86 managed to free a single bit in the mg_flags field; it is this bit that we use to indicate that the MAGIC structure refers to version 2 of Magic.

The new structures are defined in a way that should be much easier to permit updates in future; adding new structure shapes as well as new trigger functions to the various shapes entirely. In this way, we hope to be able to extend the abilities of magic in the future.

--

This PR is verymuch in "first stage" development, as I'm sure there's plenty of things that need discussion and adjustment before we want to actually merge it.

  • This set of changes requires a perldelta entry, and I need help writing it.

@guest20
Copy link

guest20 commented Feb 3, 2026

Can it be called "sorcery"?

@leonerd leonerd force-pushed the magic-v2-phase1 branch 3 times, most recently from f68b250 to a332da2 Compare February 6, 2026 16:40
@leonerd
Copy link
Contributor Author

leonerd commented Feb 6, 2026

A TODO list of items that still need addressing:

(none)

@leonerd leonerd marked this pull request as ready for review February 6, 2026 21:48
@leonerd
Copy link
Contributor Author

leonerd commented Feb 6, 2026

The few TODO items notwithstanding, I think this is now justabout ready for a review; at least to get some more eyes on it and gain a rough feel for what people think of it.

@leonerd
Copy link
Contributor Author

leonerd commented Feb 9, 2026

In particular, two of these TODO items are really more questions for other people, as I don't have particularly good ideas:

The ver field

This wants to be a number, that later versions of perl that add more fields to the functions structures give bigger numbers to. How do we want to allocate those numbers? We could just put 1, 2, 3,... or we could somehow tie it against the perl version number (e.g. 0x0544 or some variant thereon), or what. There's no technical reason to prefer any style over any other, provided it increases when it needs to.

Find home for more docs

Where would people expect to find the remaining text that's currently in perlhook.pod. I don't know where to put it.

@tonycoz
Copy link
Contributor

tonycoz commented Feb 9, 2026

The ver field

I don't care much what the actual numbers are, but they should have names, and there shouldn't be a 'latest' name visible outside of core.

Perhaps the names could be functional:

  • PERL_MG_VERSION_BASE
  • PERL_MG_VERSION_ARRAY_TIE_ADDED

but that could get silly.

I don't see much point to using random numbers for versions so probably a simple sequence (1,2,3,4,...) is best
I though about using a bit mask, but if we've implemented up to 0x7 (3 features) how is the structure laid out for 0x5?

A version outside of the implemented values should panic. (prevent someone doing 100 if we use 1,2,3..., or ~0U if we use a bitmask).

This wants to be a number, that later versions of perl that add more fields to the functions structures give bigger numbers to. How do we want to allocate those numbers? We could just put 1, 2, 3,... or we could somehow tie it against the perl version number (e.g. 0x0544 or some variant thereon), or what. There's no technical reason to prefer any style over any other, provided it increases when it needs to.

I don't think it needs to be tied to the perl version number - AFAICT this is for source compatibility, do I need to rewrite when I go from 5.44 to 5.46?

Find home for more docs

Where would people expect to find the remaining text that's currently in perlhook.pod. I don't know where to put it.

Probably perlguts, along with the older magic discussion.

It might be worth splitting it perlguts up, but I don't know if you want to do that.

@leonerd
Copy link
Contributor Author

leonerd commented Feb 10, 2026

@tonycoz Yes I think you're right, in that it shouldn't follow the perl version number, as then there would be many "pointless" bumps if there were new versions that didn't change the layout of the struct in any meaningful way. I also don't think a bitfield of "features" is all that useful - as you say it's not useful for there to be holes on it. I think I will just go with simple integers 1, 2, 3, ... Starting at 1, to avoid the "uninitialised as zero" problem.

@leonerd
Copy link
Contributor Author

leonerd commented Feb 10, 2026

Actually, no. I will start at 2, because "magic v2 version 1" would be too confusing. "magic v2" is less v2, and more "not version 1" - if ever we hit a version 3, maybe we'd rename the flags to be "extended" magic or something. Though not to be confused with extmagic and the sv_magicext function from v1. Ugh. Naming is hard. Version numbers are hard too.

@leonerd leonerd force-pushed the magic-v2-phase1 branch 2 times, most recently from 15ba904 to 264b78e Compare February 10, 2026 16:56
@leonerd
Copy link
Contributor Author

leonerd commented Feb 10, 2026

@tonycoz

Probably perlguts, along with the older magic discussion.

It might be worth splitting it perlguts up, but I don't know if you want to do that.

That seems a reasonable first cut, I've moved it there and adjusted/expanded on some of the existing words.

I think it does suggest that the entire "magic" section could be pulled out into a new "pod/perlmagic.pod" document. I don't think I will do that quite yet, but it would definitely be something to consider when we start expanding on the structure here by adding new abilities or whatever; those things would want more documentation anyway.

@leonerd
Copy link
Contributor Author

leonerd commented Feb 10, 2026

Right then. I believe that's all my current TODO list sorted. :)

Copy link
Contributor

@tonycoz tonycoz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any sufficiently advanced technology is indistinguishable from ...

@leonerd
Copy link
Contributor Author

leonerd commented Feb 10, 2026

Thanks @tonycoz :)

As this is quite a large change, with lots of additional docs and tests and so on, I'll still leave it a couple more days for folks to gather any additional comments or thoughts before merging it.

@Leont
Copy link
Contributor

Leont commented Feb 11, 2026

I don't think we should merge this at this stage of the release cycle. There's little advantage to doing so (the first phase doesn't enable us to do anything we currently can't do), but there is a risk that we got interfaces wrong.

It's probably better to merge this in 5.45.1, so we can merge the follow-up before 5.46 happens.

@leonerd leonerd added the defer-next-dev This PR should not be merged yet, but await the next development cycle label Feb 11, 2026
leonerd added 2 commits March 2, 2026 12:57
The existing `MAGIC` and `MGVTBL` structures are insufficiently
extensible, because they lack any embedded type information, or any
version numbering by which new abilities can be added in a
backwards-compatible way. Commit 6d97c86 managed to free a single bit
in the `mg_flags` field; it is this bit that we use to indicate that the
`MAGIC` structure refers to version 2 of Magic.

The new structures are defined in a way that should be much easier to
permit updates in future; primarily by the presence of a version number
in the `ver` field, and the set of trigger functions in the `shape`
field. In this way, we hope to be able to extend the abilities of magic
in the future by adding new structure shapes as well as new trigger
functions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

defer-next-dev This PR should not be merged yet, but await the next development cycle

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants