Fix shared library crash on s390x (RHEL/Fedora) by Zaneham · Pull Request #14514 · ocaml/ocaml

Zaneham · 2026-02-03T02:10:09Z

Explicitly initialise global root skiplists at runtime using a constructor function, rather than relying solely on static initialisation.

On some s390x systems (RHEL/Fedora), the BSS section of shared libraries may not be properly zeroed, causing the skiplist forward pointers to contain garbage instead of NULL. This leads to a segmentation fault in caml_skiplist_insert during startup.

I don't have access to s390x RHEL/Fedora hardware. Could someone with access to the affected platform verify this fix?

Explicitly initialize global root skiplists at runtime using a constructor function, rather than relying solely on static initialization. On some s390x systems (RHEL/Fedora), the BSS section of shared libraries may not be properly zeroed, causing the skiplist forward pointers to contain garbage instead of NULL. This leads to a segmentation fault in caml_skiplist_insert during startup. Fixes ocaml#13693

runtime/globroots.c

MisterDA · 2026-02-03T09:17:06Z

BSS section of shared libraries may not be properly zeroed

Wouldn't that totally break the semantics of C, which requires objects of static storage duration to be empty-initialized? and besides, these variables have an explicit initializer, though it seems that variables that are explicitly initialized to zero are also put in the bss (see GCC's -fno-zero-initialized-in-bss).
Out of curiosity, how can I double-check your claim?

Zaneham · 2026-02-03T09:31:32Z

BSS section of shared libraries may not be properly zeroed

Wouldn't that totally break the semantics of C, which requires objects of static storage duration to be empty-initialized? and besides, these variables have an explicit initializer, though it seems that variables that are explicitly initialized to zero are also put in the bss (see GCC's -fno-zero-initialized-in-bss). Out of curiosity, how can I double-check your claim?

Fair point, "BSS not zeroed" was an oversimplification. It's more likely an initialisation ordering issue with shared
libraries on RHEL/Fedora specifically. I can't reproduce locally since I don't have s390x RHEL/Fedora. The constructor approach should fix any initialisation ordering issues, but I'd appreciate someone with access verifying it.

Co-authored-by: Antonin Décimo <antonin.decimo@gmail.com>

xavierleroy · 2026-02-05T16:39:33Z

Thanks for pointing out the potential issue with shared library initialization on s390x, and for the proposed fix.

I have the impression there are other global variables in the OCaml runtime system that rely on proper static initialization... How can we find all the global variables that could be affected by the s390x bug?

If the bug is fixed in more recent Linux distributions, or in updated packages, we could also elect not to work around it in OCaml. Any details you could provide on this bug (e.g. a discussion on a RedHat or Ubuntu bug tracker) would be useful.

Zaneham · 2026-02-06T23:59:21Z

Thanks for pointing out the potential issue with shared library initialization on s390x, and for the proposed fix.

I have the impression there are other global variables in the OCaml runtime system that rely on proper static initialization... How can we find all the global variables that could be affected by the s390x bug?

If the bug is fixed in more recent Linux distributions, or in updated packages, we could also elect not to work around it in OCaml. Any details you could provide on this bug (e.g. a discussion on a RedHat or Ubuntu bug tracker) would be useful.

Hello @xavierleroy I couldn't find a single upstream bug report that matches exactly, but it looks like an IFUNC resolver ordering issue specific to s390x. glibc's "delayed relocation" fix for IFUNC resolvers was https://patchwork.ozlabs.org/project/glibc/patch/20180606140223.4D11F439942E1@oldenburg.str.redhat.com/. s390x got a no-op stub. Some related bugs showing the pattern:

https://bugzilla.redhat.com/show_bug.cgi?id=1398716
https://bugzilla.redhat.com/show_bug.cgi?id=1312462

Since it's a gap in glibc's s390x support rather than a fixed bug, I don't think we can count on distro updates
resolving it. I think the constructor approach seems like the safest workaround.

For auditing other affected globals, grep -rn 'SKIPLIST_STATIC_INITIALIZER' runtime/ should catch the main ones.

runtime/globroots.c

xavierleroy · 2026-02-10T14:12:27Z

For auditing other affected globals, grep -rn 'SKIPLIST_STATIC_INITIALIZER' runtime/ should catch the main ones

Thanks for the tip, but this is not the question I was asking.

The OCaml runtime system has a bunch of global variables that are statically initialized. Some are skiplists, as in

ocaml/runtime/globroots.c

Lines 46 to 51 in 3b09092

    
           struct skiplist caml_global_roots = SKIPLIST_STATIC_INITIALIZER; 
        
                             /* mutable roots, don't know whether old or young */ 
        
           struct skiplist caml_global_roots_young = SKIPLIST_STATIC_INITIALIZER; 
        
                             /* generational roots pointing to minor or major heap */ 
        
           struct skiplist caml_global_roots_old = SKIPLIST_STATIC_INITIALIZER; 
        
                             /* generational roots pointing to major heap */

and others are not, such as

ocaml/runtime/signals.c

Lines 157 to 160 in 3b09092

    
           CAMLexport void (*caml_enter_blocking_section_hook)(void) = 
        
              caml_enter_blocking_section_default; 
        
           CAMLexport void (*caml_leave_blocking_section_hook)(void) = 
        
              caml_leave_blocking_section_default;

Your patch replaces static initialization by initialization by a constructor function in the first example above, but not in the second example above. Why? Are global variables of type struct skiplist the only ones that are affected by the s390x bug on static initialization? What's so special about those global variables?

it looks like an IFUNC resolver ordering issue specific to s390x. glibc's "delayed relocation" fix for IFUNC resolvers was https://patchwork.ozlabs.org/project/glibc/patch/20180606140223.4D11F439942E1@oldenburg.str.redhat.com/. s390x got a no-op stub.

For reference, here is the description of the IFUNC feature: https://sourceware.org/glibc/wiki/GNU_IFUNC . It's about load-time resolution of function symbols. I don't see the connection with static initialization of global variables. Please explain like I'm 6.

Since it's a gap in glibc's s390x support rather than a fixed bug, I don't think we can count on distro updates
resolving it. I think the constructor approach seems like the safest workaround.

We're talking about (possibly) incorrect static initialization of global variables at dynamic loading time. That's a standard feature of C since K&R. If it was really broken, it would affect lots of programs and be treated as a highest-priority issue.

Zaneham · 2026-02-10T14:24:46Z

I was working with what I know from my albeit hobbyist work with mainframes and my interpretation of the docs. I can see my interpretation is wrong, I saw the issue, took a swing at it, and evidently missed. If you'd like I am happy to close this PR and I can look into this further later unless someone has the means to reproduce the bug.

xavierleroy · 2026-02-11T17:11:42Z

No problem. Would you be interested in getting access to our System-Z / RHEL test machine to investigate the #13693 issue further? If so, please e-mail me (https://xavierleroy.org/contact.html).

nojb · 2026-03-02T10:02:55Z

Superseded by #14547

Zaneham mentioned this pull request Feb 3, 2026

libasmrun_shared.so segfaulting on s390x Fedora/RHEL #13693

Closed

MisterDA reviewed Feb 3, 2026

View reviewed changes

runtime/globroots.c Show resolved Hide resolved

Update runtime/globroots.c

abcc5f1

Co-authored-by: Antonin Décimo <antonin.decimo@gmail.com>

Zaneham requested a review from MisterDA February 3, 2026 09:36

Merge branch 'trunk' into s390x

889210d

nojb assigned damiendoligez Feb 4, 2026

Merge branch 'trunk' into s390x

88dbba4

dra27 reviewed Feb 8, 2026

View reviewed changes

runtime/globroots.c Outdated Show resolved Hide resolved

dra27 and others added 2 commits February 8, 2026 21:16

Apply suggestion from @dra27

813f08b

Merge branch 'trunk' into s390x

d08d28e

Zaneham mentioned this pull request Feb 12, 2026

s390x: use GOT-indirect calls to fix heap corruption with shared runt… #14547

Merged

nojb closed this Mar 2, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix shared library crash on s390x (RHEL/Fedora)#14514

Fix shared library crash on s390x (RHEL/Fedora)#14514
Zaneham wants to merge 6 commits intoocaml:trunkfrom
Zaneham:s390x

Zaneham commented Feb 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

MisterDA commented Feb 3, 2026 •

edited

Loading

Uh oh!

Zaneham commented Feb 3, 2026

Uh oh!

xavierleroy commented Feb 5, 2026

Uh oh!

Zaneham commented Feb 6, 2026

Uh oh!

Uh oh!

xavierleroy commented Feb 10, 2026

Uh oh!

Zaneham commented Feb 10, 2026

Uh oh!

xavierleroy commented Feb 11, 2026

Uh oh!

nojb commented Mar 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

Zaneham commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

MisterDA commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Zaneham commented Feb 3, 2026

Uh oh!

xavierleroy commented Feb 5, 2026

Uh oh!

Zaneham commented Feb 6, 2026

Uh oh!

Uh oh!

xavierleroy commented Feb 10, 2026

Uh oh!

Zaneham commented Feb 10, 2026

Uh oh!

xavierleroy commented Feb 11, 2026

Uh oh!

nojb commented Mar 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Zaneham commented Feb 3, 2026 •

edited

Loading

MisterDA commented Feb 3, 2026 •

edited

Loading