Skip to content

Segfaults on exit when the sigaltstack is changed by the program #11489

@rajdakin

Description

@rajdakin

The issue

If the user's program changes the sigaltstack (through a C library, for example) with the stack overflow detection active (HAS_STACK_OVERFLOW_DETECTION is defined), the program will crash with an invalid free when exiting.

The source

The function (caml_)setup_stack_overflow_detection calls sigaltstack with a stack placed from a malloc, that is then supposedly freed in caml_stop_stack_overflow_detection.
This function only checks if the flags indicates that the sigaltstack was disabled, and if not, frees it.
However, this is not valid in general, when other libraries may want more memory space (in fact, Wasmer, which is a Rust library, requires more than 8k, instead allocating 64k of stack space). In this case, the library changes the sigaltstack (with an enabled stack), but using a different method (for example, an mmap). Then, OCaml detects the sigaltstack is enabled, ad tries to free it.

The solution

The solution would be to remember the ss_sp variable (that was malloc'ed), then when freeing, we also need to check if this ss_sp was changed. If not, the case I just described did not happen, and we can free it.
Otherwise, we left everything as-is (because the external library might restore the stack and/or use it for its own purpose), potentially leading to a memory leak.

Affected version

Version 4.10 is guaranteed to fail, but the incriminating piece of code stays up until version 4.14.
At first glance, it does not seems like version 5.0 is affected (but I may be wrong, I have not tested it).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions