[libc] allocator initialization re-entrancy

[SchrodingerZhu]

I create this meta issue because in microsoft/snmalloc#762, we investigated several edge cases where layering allocator on top of system libc leads to deadlocks (found by me) and memory leaks (credits @mjp41).

These issues are actually quite interesting. Even though they are hard to trigger in glibc in practice because it is unlikely to create too many atexit hooks and it is likely that allocator (and its TLS) is initialized already by libc's internal allocation.

However, given that LLVM-libc has been specifically designed to support overlay usage, it seems to be more relevant to these edge cases.

To mitigate these problems, in libc, we should:

• Add some static buffers for atexit hook.
• Handle re-entrance to our best efforts in atexit and __cxa_thread_atexit_impl and pthread_setspecific.
  I see that in public packaging mode, current implemention (and TODOs) tend to use bloblist, but that is actually vulnerable. Based on our observation, a bulk allocation can make the pthread_setspecific overrides its structure due to re-entrance caused by allocation, while single cell allocation is good to go.
• Explore other solutions:
  A common practice by BSDs is to expose _malloc_thread_cleanup as a weak symbol that can be overwritten by custom allocators. This largely avoids complexity to maintain the correctness of cleanup. I really think it is a good thing to have for LLVM-libc.

[llvmbot]

@llvm/issue-subscribers-libc

Author: Schrodinger ZHU Yifan (SchrodingerZhu)

I create this meta issue because in https://github.com/microsoft/snmalloc/pull/762, we investigated several edge cases where layering allocator on top of system libc leads to deadlocks (found by me) and memory leaks (credits @mjp41).

These issues are actually quite interesting. Even though they are hard to trigger in glibc in practice because it is unlikely to create too many atexit hooks and it is likely that allocator (and its TLS) is initialized already by libc's internal allocation.

However, given that LLVM-libc has been specifically designed to support overlay usage, it seems to be more relevant to these edge cases.

To mitigate these problems, in libc, we should:

• Add some static buffers for atexit hook.
• Handle re-entrance to our best efforts.
• Explore other solutions:
  A common practice by BSDs is to expose _malloc_thread_cleanup as a weak symbol that can be overwritten by custom allocators. This largely avoids complexity to maintain the correctness of cleanup. I really think it is a good thing to have for LLVM-libc.