[libclang] Always Dup in createRef(StringRef) by vitalybuka · Pull Request #125020 · llvm/llvm-project

vitalybuka · 2025-01-30T03:59:32Z

We can't guaranty that underlying string is
0-terminated and [String.size()] is even in the
same allocation.

https://lab.llvm.org/buildbot/#/builders/94/builds/4152/steps/17/logs/stdio

==c-index-test==1846256==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0  in clang::cxstring::createRef(llvm::StringRef) llvm-project/clang/tools/libclang/CXString.cpp:96:36
    #1  in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:521:39
    #2  in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:674:7
    #3  in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:674:7
    #4  in DumpCXComment llvm-project/clang/tools/c-index-test/c-index-test.c:685:3
    #5  in PrintCursorComments llvm-project/clang/tools/c-index-test/c-index-test.c:768:7

  Memory was marked as uninitialized
    #0  in __msan_allocated_memory llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1023:5
    #1  in Allocate llvm-project/llvm/include/llvm/Support/Allocator.h:172:7
    #2  in Allocate llvm-project/llvm/include/llvm/Support/Allocator.h:216:12
    #3  in Allocate llvm-project/llvm/include/llvm/Support/AllocatorBase.h:53:43
    #4  in Allocate<char> llvm-project/llvm/include/llvm/Support/AllocatorBase.h:76:29
    #5  in convertCodePointToUTF8 llvm-project/clang/lib/AST/CommentLexer.cpp:42:30
    #6  in clang::comments::Lexer::resolveHTMLDecimalCharacterReference(llvm::StringRef) const llvm-project/clang/lib/AST/CommentLexer.cpp:76:10
    #7  in clang::comments::Lexer::lexHTMLCharacterReference(clang::comments::Token&) llvm-project/clang/lib/AST/CommentLexer.cpp:615:16
    #8  in consumeToken llvm-project/clang/include/clang/AST/CommentParser.h:62:9
    #9  in clang::comments::Parser::parseParagraphOrBlockCommand() llvm-project/clang/lib/AST/CommentParser.cpp
    #10 in clang::comments::Parser::parseFullComment() llvm-project/clang/lib/AST/CommentParser.cpp:925:22
    #11 in clang::RawComment::parse(clang::ASTContext const&, clang::Preprocessor const*, clang::Decl const*) const llvm-project/clang/lib/AST/RawCommentList.cpp:221:12
    #12 in clang::ASTContext::getCommentForDecl(clang::Decl const*, clang::Preprocessor const*) const llvm-project/clang/lib/AST/ASTContext.cpp:714:35
    #13 in clang_Cursor_getParsedComment llvm-project/clang/tools/libclang/CXComment.cpp:36:35
    #14 in PrintCursorComments llvm-project/clang/tools/c-index-test/c-index-test.c:756:25

Created using spr 1.3.4

llvmbot · 2025-01-30T04:00:07Z

@llvm/pr-subscribers-clang

Author: Vitaly Buka (vitalybuka)

Changes

We can't guaranty that underlying string is
0-terminated and [String.size()] is even in the
same allocation.

Full diff: https://github.com/llvm/llvm-project/pull/125020.diff

1 Files Affected:

(modified) clang/tools/libclang/CXString.cpp (+1-13)

diff --git a/clang/tools/libclang/CXString.cpp b/clang/tools/libclang/CXString.cpp
index 5e427957a1092b..aaa8f8eeb67a12 100644
--- a/clang/tools/libclang/CXString.cpp
+++ b/clang/tools/libclang/CXString.cpp
@@ -87,19 +87,7 @@ CXString createRef(StringRef String) {
   if (String.empty())
     return createEmpty();
 
-  // If the string is not nul-terminated, we have to make a copy.
-
-  // FIXME: This is doing a one past end read, and should be removed! For memory
-  // we don't manage, the API string can become unterminated at any time outside
-  // our control.
-
-  if (String.data()[String.size()] != 0)
-    return createDup(String);
-
-  CXString Result;
-  Result.data = String.data();
-  Result.private_flags = (unsigned) CXS_Unmanaged;
-  return Result;
+  return createDup(String);
 }
 
 CXString createDup(StringRef String) {

vitalybuka · 2025-01-30T04:09:46Z

It does not look nice for memory usage, but I don't know how critical it's here.
It's it's unacceptable, then __attribute__((no_sanitize("memory"))) is an option.

vitalybuka · 2025-02-03T22:21:43Z

ping

vitalybuka · 2025-02-07T17:51:02Z

ping

efriedma-quic

If CXString createRef(StringRef) is essentially equivalent to CXString createDup(StringRef), should we just get rid of CXString createRef(StringRef)?

vitalybuka · 2025-02-11T00:04:03Z

If CXString createRef(StringRef) is essentially equivalent to CXString createDup(StringRef), should we just get rid of CXString createRef(StringRef)?

I guess this is some C API which we need to maintain? No idea how it's used, and if we can remove it.
I'd prefer to land this as-is, and remove createRef in a follow up PR. In case someone find a good reason too keep both we don't want to revert UB fix. WDYT?

#126683

Created using spr 1.3.4

AaronBallman · 2025-02-11T13:21:05Z

If CXString createRef(StringRef) is essentially equivalent to CXString createDup(StringRef), should we just get rid of CXString createRef(StringRef)?

I guess this is some C API which we need to maintain?

Correct, libclang is a stable API. We would have to deprecate the API and wait a decent amount of time before removing it.

AaronBallman

Please add a release note so that users know about the fix and about the expected performance costs, but otherwise LGTM

efriedma-quic · 2025-02-11T22:10:49Z

The public libclang interface needs to be stable, yes... but in this context, that's just the interfaces declared in clang/include/clang-c/CXString.h. We can do whatever we want with the internal interface in clang/tools/libclang/CXString.h.

AaronBallman · 2025-02-12T13:26:36Z

The public libclang interface needs to be stable, yes... but in this context, that's just the interfaces declared in clang/include/clang-c/CXString.h. We can do whatever we want with the internal interface in clang/tools/libclang/CXString.h.

Oh, good catch! Agreed. :-)

Created using spr 1.3.4

vitalybuka · 2025-02-13T06:02:18Z

Please add a release note so that users know about the fix and about the expected performance costs, but otherwise LGTM

done

Created using spr 1.3.4

llvm-ci · 2025-02-13T07:16:54Z

LLVM Buildbot has detected a new failure on builder sanitizer-aarch64-linux-bootstrap-asan running on sanitizer-buildbot7 while building clang at step 2 "annotate".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/24/builds/5252

Here is the relevant piece of the build log for the reference

Step 2 (annotate) failure: 'python ../sanitizer_buildbot/sanitizers/zorg/buildbot/builders/sanitizers/buildbot_selector.py' (failure)
...
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/wasm-ld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using ld.lld: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/ld.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/wasm-ld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/main.py:72: note: The test suite configuration requested an individual test timeout of 0 seconds but a timeout of 900 seconds was requested on the command line. Forcing timeout to be 900 seconds.
-- Testing: 86445 tests, 72 workers --
Testing:  0.. 10
FAIL: Clang :: Index/USR/structural-value-tpl-arg.cpp (12308 of 86445)
******************** TEST 'Clang :: Index/USR/structural-value-tpl-arg.cpp' FAILED ********************
Exit Code: 2

Command Output (stderr):
--
RUN: at line 1: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/c-index-test -test-load-source-usrs local -std=c++20 -- /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/USR/structural-value-tpl-arg.cpp | /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/USR/structural-value-tpl-arg.cpp
+ /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/USR/structural-value-tpl-arg.cpp
+ /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/c-index-test -test-load-source-usrs local -std=c++20 -- /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/USR/structural-value-tpl-arg.cpp
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/USR/structural-value-tpl-arg.cpp

--

********************
Testing:  0.. 10
FAIL: Clang :: Index/annotate-attribute.cpp (12311 of 86445)
******************** TEST 'Clang :: Index/annotate-attribute.cpp' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 1: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/c-index-test -test-load-source all /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp | /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp
+ /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp
+ /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/c-index-test -test-load-source all /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp
/home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp:25:11: error: CHECK: expected string not found in input
// CHECK: ClassDecl=Test:3:7 (Definition) Extent=[3:1 - 17:2]
          ^
<stdin>:1:1: note: scanning from here
// CHECK: <invalid loc>:2:9: macro definition=__llvm__
^
<stdin>:1:30: note: possible intended match here
// CHECK: <invalid loc>:2:9: macro definition=__llvm__
                             ^

Input file: <stdin>
Check file: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp

-dump-input=help explains the following input dump.
Step 11 (stage2/asan check) failure: stage2/asan check (failure)
...
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/wasm-ld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using ld.lld: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/ld.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using lld-link: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/lld-link
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using ld64.lld: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/ld64.lld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/llvm/config.py:506: note: using wasm-ld: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/wasm-ld
llvm-lit: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/utils/lit/lit/main.py:72: note: The test suite configuration requested an individual test timeout of 0 seconds but a timeout of 900 seconds was requested on the command line. Forcing timeout to be 900 seconds.
-- Testing: 86445 tests, 72 workers --
Testing:  0.. 10
FAIL: Clang :: Index/USR/structural-value-tpl-arg.cpp (12308 of 86445)
******************** TEST 'Clang :: Index/USR/structural-value-tpl-arg.cpp' FAILED ********************
Exit Code: 2

Command Output (stderr):
--
RUN: at line 1: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/c-index-test -test-load-source-usrs local -std=c++20 -- /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/USR/structural-value-tpl-arg.cpp | /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/USR/structural-value-tpl-arg.cpp
+ /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/USR/structural-value-tpl-arg.cpp
+ /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/c-index-test -test-load-source-usrs local -std=c++20 -- /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/USR/structural-value-tpl-arg.cpp
FileCheck error: '<stdin>' is empty.
FileCheck command line:  /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/USR/structural-value-tpl-arg.cpp

--

********************
Testing:  0.. 10
FAIL: Clang :: Index/annotate-attribute.cpp (12311 of 86445)
******************** TEST 'Clang :: Index/annotate-attribute.cpp' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 1: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/c-index-test -test-load-source all /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp | /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp
+ /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/FileCheck /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp
+ /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm_build_asan/bin/c-index-test -test-load-source all /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp
/home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp:25:11: error: CHECK: expected string not found in input
// CHECK: ClassDecl=Test:3:7 (Definition) Extent=[3:1 - 17:2]
          ^
<stdin>:1:1: note: scanning from here
// CHECK: <invalid loc>:2:9: macro definition=__llvm__
^
<stdin>:1:30: note: possible intended match here
// CHECK: <invalid loc>:2:9: macro definition=__llvm__
                             ^

Input file: <stdin>
Check file: /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/test/Index/annotate-attribute.cpp

-dump-input=help explains the following input dump.

We can't guaranty that underlying string is 0-terminated and [String.size()] is even in the same allocation. https://lab.llvm.org/buildbot/#/builders/94/builds/4152/steps/17/logs/stdio ``` ==c-index-test==1846256==WARNING: MemorySanitizer: use-of-uninitialized-value #0 in clang::cxstring::createRef(llvm::StringRef) llvm-project/clang/tools/libclang/CXString.cpp:96:36 llvm#1 in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:521:39 llvm#2 in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:674:7 llvm#3 in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:674:7 llvm#4 in DumpCXComment llvm-project/clang/tools/c-index-test/c-index-test.c:685:3 llvm#5 in PrintCursorComments llvm-project/clang/tools/c-index-test/c-index-test.c:768:7 Memory was marked as uninitialized #0 in __msan_allocated_memory llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1023:5 llvm#1 in Allocate llvm-project/llvm/include/llvm/Support/Allocator.h:172:7 llvm#2 in Allocate llvm-project/llvm/include/llvm/Support/Allocator.h:216:12 llvm#3 in Allocate llvm-project/llvm/include/llvm/Support/AllocatorBase.h:53:43 llvm#4 in Allocate<char> llvm-project/llvm/include/llvm/Support/AllocatorBase.h:76:29 llvm#5 in convertCodePointToUTF8 llvm-project/clang/lib/AST/CommentLexer.cpp:42:30 llvm#6 in clang::comments::Lexer::resolveHTMLDecimalCharacterReference(llvm::StringRef) const llvm-project/clang/lib/AST/CommentLexer.cpp:76:10 llvm#7 in clang::comments::Lexer::lexHTMLCharacterReference(clang::comments::Token&) llvm-project/clang/lib/AST/CommentLexer.cpp:615:16 llvm#8 in consumeToken llvm-project/clang/include/clang/AST/CommentParser.h:62:9 llvm#9 in clang::comments::Parser::parseParagraphOrBlockCommand() llvm-project/clang/lib/AST/CommentParser.cpp llvm#10 in clang::comments::Parser::parseFullComment() llvm-project/clang/lib/AST/CommentParser.cpp:925:22 llvm#11 in clang::RawComment::parse(clang::ASTContext const&, clang::Preprocessor const*, clang::Decl const*) const llvm-project/clang/lib/AST/RawCommentList.cpp:221:12 llvm#12 in clang::ASTContext::getCommentForDecl(clang::Decl const*, clang::Preprocessor const*) const llvm-project/clang/lib/AST/ASTContext.cpp:714:35 llvm#13 in clang_Cursor_getParsedComment llvm-project/clang/tools/libclang/CXComment.cpp:36:35 llvm#14 in PrintCursorComments llvm-project/clang/tools/c-index-test/c-index-test.c:756:25 ```

This reverts commit e76739e.

Reverts #125020 https://lab.llvm.org/buildbot/#/builders/24/builds/5252/steps/12/logs/stdio ``` ==c-index-test==2512295==ERROR: AddressSanitizer: heap-use-after-free on address 0xe19338c27992 at pc 0xc66be4784830 bp 0xe0e33660df00 sp 0xe0e33660d6e8 READ of size 23 at 0xe19338c27992 thread T1 #0 0xc66be478482c in printf_common(void*, char const*, std::__va_list) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors_format.inc:563:9 #1 0xc66be478643c in vprintf /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1699:1 #2 0xc66be478643c in printf /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1757:1 #3 0xc66be4839384 in FilteredPrintingVisitor /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/c-index-test/c-index-test.c:1359:5 #4 0xe4e3454f12e8 in clang::cxcursor::CursorVisitor::Visit(CXCursor, bool) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CIndex.cpp:227:11 #5 0xe4e3454f48a8 in bool clang::cxcursor::CursorVisitor::visitPreprocessedEntities<clang::PreprocessingRecord::iterator>(clang::PreprocessingRecord::iterator, clang::PreprocessingRecord::iterator, clang::PreprocessingRecord&, clang::FileID) CIndex.cpp 0xe19338c27992 is located 82 bytes inside of 105-byte region [0xe19338c27940,0xe19338c279a9) freed by thread T1 here: #0 0xc66be480040c in free /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:51:3 #1 0xc66be4839728 in GetCursorSource c-index-test.c #2 0xc66be4839368 in FilteredPrintingVisitor /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/c-index-test/c-index-test.c:1360:12 #3 0xe4e3454f12e8 in clang::cxcursor::CursorVisitor::Visit(CXCursor, bool) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CIndex.cpp:227:11 #4 0xe4e3454f48a8 in bool clang::cxcursor::CursorVisitor::visitPreprocessedEntities<clang::PreprocessingRecord::iterator>(clang::PreprocessingRecord::iterator, clang::PreprocessingRecord::iterator, clang::PreprocessingRecord&, clang::FileID) CIndex.cpp previously allocated by thread T1 here: #0 0xc66be4800680 in malloc /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:67:3 #1 0xe4e3456379b0 in safe_malloc /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/include/llvm/Support/MemAlloc.h:26:18 #2 0xe4e3456379b0 in createDup /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CXString.cpp:95:40 #3 0xe4e3456379b0 in clang::cxstring::createRef(llvm::StringRef) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CXString.cpp:90:10 ```

…27076) Reverts llvm/llvm-project#125020 https://lab.llvm.org/buildbot/#/builders/24/builds/5252/steps/12/logs/stdio ``` ==c-index-test==2512295==ERROR: AddressSanitizer: heap-use-after-free on address 0xe19338c27992 at pc 0xc66be4784830 bp 0xe0e33660df00 sp 0xe0e33660d6e8 READ of size 23 at 0xe19338c27992 thread T1 #0 0xc66be478482c in printf_common(void*, char const*, std::__va_list) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors_format.inc:563:9 #1 0xc66be478643c in vprintf /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1699:1 #2 0xc66be478643c in printf /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1757:1 #3 0xc66be4839384 in FilteredPrintingVisitor /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/c-index-test/c-index-test.c:1359:5 #4 0xe4e3454f12e8 in clang::cxcursor::CursorVisitor::Visit(CXCursor, bool) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CIndex.cpp:227:11 #5 0xe4e3454f48a8 in bool clang::cxcursor::CursorVisitor::visitPreprocessedEntities<clang::PreprocessingRecord::iterator>(clang::PreprocessingRecord::iterator, clang::PreprocessingRecord::iterator, clang::PreprocessingRecord&, clang::FileID) CIndex.cpp 0xe19338c27992 is located 82 bytes inside of 105-byte region [0xe19338c27940,0xe19338c279a9) freed by thread T1 here: #0 0xc66be480040c in free /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:51:3 #1 0xc66be4839728 in GetCursorSource c-index-test.c #2 0xc66be4839368 in FilteredPrintingVisitor /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/c-index-test/c-index-test.c:1360:12 #3 0xe4e3454f12e8 in clang::cxcursor::CursorVisitor::Visit(CXCursor, bool) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CIndex.cpp:227:11 #4 0xe4e3454f48a8 in bool clang::cxcursor::CursorVisitor::visitPreprocessedEntities<clang::PreprocessingRecord::iterator>(clang::PreprocessingRecord::iterator, clang::PreprocessingRecord::iterator, clang::PreprocessingRecord&, clang::FileID) CIndex.cpp previously allocated by thread T1 here: #0 0xc66be4800680 in malloc /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:67:3 #1 0xe4e3456379b0 in safe_malloc /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/include/llvm/Support/MemAlloc.h:26:18 #2 0xe4e3456379b0 in createDup /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CXString.cpp:95:40 #3 0xe4e3456379b0 in clang::cxstring::createRef(llvm::StringRef) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CXString.cpp:90:10 ```

We can't guaranty that underlying string is 0-terminated and [String.size()] is even in the same allocation. https://lab.llvm.org/buildbot/#/builders/94/builds/4152/steps/17/logs/stdio ``` ==c-index-test==1846256==WARNING: MemorySanitizer: use-of-uninitialized-value #0 in clang::cxstring::createRef(llvm::StringRef) llvm-project/clang/tools/libclang/CXString.cpp:96:36 #1 in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:521:39 #2 in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:674:7 llvm#3 in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:674:7 llvm#4 in DumpCXComment llvm-project/clang/tools/c-index-test/c-index-test.c:685:3 llvm#5 in PrintCursorComments llvm-project/clang/tools/c-index-test/c-index-test.c:768:7 Memory was marked as uninitialized #0 in __msan_allocated_memory llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1023:5 #1 in Allocate llvm-project/llvm/include/llvm/Support/Allocator.h:172:7 #2 in Allocate llvm-project/llvm/include/llvm/Support/Allocator.h:216:12 llvm#3 in Allocate llvm-project/llvm/include/llvm/Support/AllocatorBase.h:53:43 llvm#4 in Allocate<char> llvm-project/llvm/include/llvm/Support/AllocatorBase.h:76:29 llvm#5 in convertCodePointToUTF8 llvm-project/clang/lib/AST/CommentLexer.cpp:42:30 llvm#6 in clang::comments::Lexer::resolveHTMLDecimalCharacterReference(llvm::StringRef) const llvm-project/clang/lib/AST/CommentLexer.cpp:76:10 llvm#7 in clang::comments::Lexer::lexHTMLCharacterReference(clang::comments::Token&) llvm-project/clang/lib/AST/CommentLexer.cpp:615:16 llvm#8 in consumeToken llvm-project/clang/include/clang/AST/CommentParser.h:62:9 llvm#9 in clang::comments::Parser::parseParagraphOrBlockCommand() llvm-project/clang/lib/AST/CommentParser.cpp llvm#10 in clang::comments::Parser::parseFullComment() llvm-project/clang/lib/AST/CommentParser.cpp:925:22 llvm#11 in clang::RawComment::parse(clang::ASTContext const&, clang::Preprocessor const*, clang::Decl const*) const llvm-project/clang/lib/AST/RawCommentList.cpp:221:12 llvm#12 in clang::ASTContext::getCommentForDecl(clang::Decl const*, clang::Preprocessor const*) const llvm-project/clang/lib/AST/ASTContext.cpp:714:35 llvm#13 in clang_Cursor_getParsedComment llvm-project/clang/tools/libclang/CXComment.cpp:36:35 llvm#14 in PrintCursorComments llvm-project/clang/tools/c-index-test/c-index-test.c:756:25 ```

Reverts llvm#125020 https://lab.llvm.org/buildbot/#/builders/24/builds/5252/steps/12/logs/stdio ``` ==c-index-test==2512295==ERROR: AddressSanitizer: heap-use-after-free on address 0xe19338c27992 at pc 0xc66be4784830 bp 0xe0e33660df00 sp 0xe0e33660d6e8 READ of size 23 at 0xe19338c27992 thread T1 #0 0xc66be478482c in printf_common(void*, char const*, std::__va_list) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors_format.inc:563:9 #1 0xc66be478643c in vprintf /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1699:1 #2 0xc66be478643c in printf /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1757:1 llvm#3 0xc66be4839384 in FilteredPrintingVisitor /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/c-index-test/c-index-test.c:1359:5 llvm#4 0xe4e3454f12e8 in clang::cxcursor::CursorVisitor::Visit(CXCursor, bool) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CIndex.cpp:227:11 llvm#5 0xe4e3454f48a8 in bool clang::cxcursor::CursorVisitor::visitPreprocessedEntities<clang::PreprocessingRecord::iterator>(clang::PreprocessingRecord::iterator, clang::PreprocessingRecord::iterator, clang::PreprocessingRecord&, clang::FileID) CIndex.cpp 0xe19338c27992 is located 82 bytes inside of 105-byte region [0xe19338c27940,0xe19338c279a9) freed by thread T1 here: #0 0xc66be480040c in free /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:51:3 #1 0xc66be4839728 in GetCursorSource c-index-test.c #2 0xc66be4839368 in FilteredPrintingVisitor /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/c-index-test/c-index-test.c:1360:12 llvm#3 0xe4e3454f12e8 in clang::cxcursor::CursorVisitor::Visit(CXCursor, bool) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CIndex.cpp:227:11 llvm#4 0xe4e3454f48a8 in bool clang::cxcursor::CursorVisitor::visitPreprocessedEntities<clang::PreprocessingRecord::iterator>(clang::PreprocessingRecord::iterator, clang::PreprocessingRecord::iterator, clang::PreprocessingRecord&, clang::FileID) CIndex.cpp previously allocated by thread T1 here: #0 0xc66be4800680 in malloc /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:67:3 #1 0xe4e3456379b0 in safe_malloc /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/include/llvm/Support/MemAlloc.h:26:18 #2 0xe4e3456379b0 in createDup /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CXString.cpp:95:40 llvm#3 0xe4e3456379b0 in clang::cxstring::createRef(llvm::StringRef) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CXString.cpp:90:10 ```

Reverts #127076 to reland #125020. Use-after-free should be fixed here #127063

We can't guaranty that underlying string is 0-terminated and [String.size()] is even in the same allocation. https://lab.llvm.org/buildbot/#/builders/94/builds/4152/steps/17/logs/stdio ``` ==c-index-test==1846256==WARNING: MemorySanitizer: use-of-uninitialized-value #0 in clang::cxstring::createRef(llvm::StringRef) llvm-project/clang/tools/libclang/CXString.cpp:96:36 llvm#1 in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:521:39 llvm#2 in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:674:7 llvm#3 in DumpCXCommentInternal llvm-project/clang/tools/c-index-test/c-index-test.c:674:7 llvm#4 in DumpCXComment llvm-project/clang/tools/c-index-test/c-index-test.c:685:3 llvm#5 in PrintCursorComments llvm-project/clang/tools/c-index-test/c-index-test.c:768:7 Memory was marked as uninitialized #0 in __msan_allocated_memory llvm-project/compiler-rt/lib/msan/msan_interceptors.cpp:1023:5 llvm#1 in Allocate llvm-project/llvm/include/llvm/Support/Allocator.h:172:7 llvm#2 in Allocate llvm-project/llvm/include/llvm/Support/Allocator.h:216:12 llvm#3 in Allocate llvm-project/llvm/include/llvm/Support/AllocatorBase.h:53:43 llvm#4 in Allocate<char> llvm-project/llvm/include/llvm/Support/AllocatorBase.h:76:29 llvm#5 in convertCodePointToUTF8 llvm-project/clang/lib/AST/CommentLexer.cpp:42:30 llvm#6 in clang::comments::Lexer::resolveHTMLDecimalCharacterReference(llvm::StringRef) const llvm-project/clang/lib/AST/CommentLexer.cpp:76:10 llvm#7 in clang::comments::Lexer::lexHTMLCharacterReference(clang::comments::Token&) llvm-project/clang/lib/AST/CommentLexer.cpp:615:16 llvm#8 in consumeToken llvm-project/clang/include/clang/AST/CommentParser.h:62:9 llvm#9 in clang::comments::Parser::parseParagraphOrBlockCommand() llvm-project/clang/lib/AST/CommentParser.cpp llvm#10 in clang::comments::Parser::parseFullComment() llvm-project/clang/lib/AST/CommentParser.cpp:925:22 llvm#11 in clang::RawComment::parse(clang::ASTContext const&, clang::Preprocessor const*, clang::Decl const*) const llvm-project/clang/lib/AST/RawCommentList.cpp:221:12 llvm#12 in clang::ASTContext::getCommentForDecl(clang::Decl const*, clang::Preprocessor const*) const llvm-project/clang/lib/AST/ASTContext.cpp:714:35 llvm#13 in clang_Cursor_getParsedComment llvm-project/clang/tools/libclang/CXComment.cpp:36:35 llvm#14 in PrintCursorComments llvm-project/clang/tools/c-index-test/c-index-test.c:756:25 ```

Reverts llvm#125020 https://lab.llvm.org/buildbot/#/builders/24/builds/5252/steps/12/logs/stdio ``` ==c-index-test==2512295==ERROR: AddressSanitizer: heap-use-after-free on address 0xe19338c27992 at pc 0xc66be4784830 bp 0xe0e33660df00 sp 0xe0e33660d6e8 READ of size 23 at 0xe19338c27992 thread T1 #0 0xc66be478482c in printf_common(void*, char const*, std::__va_list) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors_format.inc:563:9 llvm#1 0xc66be478643c in vprintf /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1699:1 llvm#2 0xc66be478643c in printf /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:1757:1 llvm#3 0xc66be4839384 in FilteredPrintingVisitor /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/c-index-test/c-index-test.c:1359:5 llvm#4 0xe4e3454f12e8 in clang::cxcursor::CursorVisitor::Visit(CXCursor, bool) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CIndex.cpp:227:11 llvm#5 0xe4e3454f48a8 in bool clang::cxcursor::CursorVisitor::visitPreprocessedEntities<clang::PreprocessingRecord::iterator>(clang::PreprocessingRecord::iterator, clang::PreprocessingRecord::iterator, clang::PreprocessingRecord&, clang::FileID) CIndex.cpp 0xe19338c27992 is located 82 bytes inside of 105-byte region [0xe19338c27940,0xe19338c279a9) freed by thread T1 here: #0 0xc66be480040c in free /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:51:3 llvm#1 0xc66be4839728 in GetCursorSource c-index-test.c llvm#2 0xc66be4839368 in FilteredPrintingVisitor /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/c-index-test/c-index-test.c:1360:12 llvm#3 0xe4e3454f12e8 in clang::cxcursor::CursorVisitor::Visit(CXCursor, bool) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CIndex.cpp:227:11 llvm#4 0xe4e3454f48a8 in bool clang::cxcursor::CursorVisitor::visitPreprocessedEntities<clang::PreprocessingRecord::iterator>(clang::PreprocessingRecord::iterator, clang::PreprocessingRecord::iterator, clang::PreprocessingRecord&, clang::FileID) CIndex.cpp previously allocated by thread T1 here: #0 0xc66be4800680 in malloc /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:67:3 llvm#1 0xe4e3456379b0 in safe_malloc /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/llvm/include/llvm/Support/MemAlloc.h:26:18 llvm#2 0xe4e3456379b0 in createDup /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CXString.cpp:95:40 llvm#3 0xe4e3456379b0 in clang::cxstring::createRef(llvm::StringRef) /home/b/sanitizer-aarch64-linux-bootstrap-asan/build/llvm-project/clang/tools/libclang/CXString.cpp:90:10 ```

[𝘀𝗽𝗿] initial version

10af910

Created using spr 1.3.4

llvmbot added clang Clang issues not falling into any other category clang:as-a-library libclang and C++ API labels Jan 30, 2025

vitalybuka requested review from Arnaud-de-Grandmaison-ARM, egorzhdan and gribozavr January 30, 2025 04:01

vitalybuka requested a review from AaronBallman January 30, 2025 04:13

vitalybuka changed the title ~~[libclang] Always Dup in createRef~~ [libclang] Always Dup in createRef(StringRef) Jan 30, 2025

vitalybuka requested review from alexey-bataev and efriedma-quic February 8, 2025 01:55

efriedma-quic reviewed Feb 10, 2025

View reviewed changes

vitalybuka requested a review from efriedma-quic February 11, 2025 00:04

rebase

fe55503

Created using spr 1.3.4

AaronBallman approved these changes Feb 11, 2025

View reviewed changes

relnote

3ece7bc

Created using spr 1.3.4

vitalybuka added 2 commits February 12, 2025 22:03

quotes

278a41b

Created using spr 1.3.4

the

5fd15be

Created using spr 1.3.4

vitalybuka merged commit e76739e into main Feb 13, 2025
7 of 9 checks passed

vitalybuka deleted the users/vitalybuka/spr/libclang-always-dup-in-createref branch February 13, 2025 06:05

vitalybuka restored the users/vitalybuka/spr/libclang-always-dup-in-createref branch February 13, 2025 15:39

vitalybuka deleted the users/vitalybuka/spr/libclang-always-dup-in-createref branch February 13, 2025 15:39

vitalybuka added a commit that referenced this pull request Feb 13, 2025

Revert "[libclang] Always Dup in createRef(StringRef) (#125020)"

3548823

This reverts commit e76739e.

vitalybuka mentioned this pull request Feb 13, 2025

Revert "[libclang] Always Dup in createRef(StringRef)" #127076

Merged

vitalybuka mentioned this pull request Feb 13, 2025

Reland "[libclang] Always Dup in createRef(StringRef)" #127078

Merged

vitalybuka added a commit that referenced this pull request Feb 20, 2025

Reland "[libclang] Always Dup in createRef(StringRef)" (#127078)

a16fa3a

Reverts #127076 to reland #125020. Use-after-free should be fixed here #127063

vitalybuka mentioned this pull request Feb 25, 2025

[libclang] Replace createRef with createDup #126683

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[libclang] Always Dup in createRef(StringRef)#125020

[libclang] Always Dup in createRef(StringRef)#125020
vitalybuka merged 5 commits intomainfrom
users/vitalybuka/spr/libclang-always-dup-in-createref

vitalybuka commented Jan 30, 2025 •

edited

Loading

Uh oh!

llvmbot commented Jan 30, 2025

Uh oh!

vitalybuka commented Jan 30, 2025

Uh oh!

vitalybuka commented Feb 3, 2025

Uh oh!

vitalybuka commented Feb 7, 2025

Uh oh!

efriedma-quic left a comment

Uh oh!

vitalybuka commented Feb 11, 2025 •

edited

Loading

Uh oh!

AaronBallman commented Feb 11, 2025

Uh oh!

AaronBallman left a comment

Uh oh!

efriedma-quic commented Feb 11, 2025

Uh oh!

AaronBallman commented Feb 12, 2025

Uh oh!

vitalybuka commented Feb 13, 2025

Uh oh!

Uh oh!

llvm-ci commented Feb 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

vitalybuka commented Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

llvmbot commented Jan 30, 2025

Uh oh!

vitalybuka commented Jan 30, 2025

Uh oh!

vitalybuka commented Feb 3, 2025

Uh oh!

vitalybuka commented Feb 7, 2025

Uh oh!

efriedma-quic left a comment

Choose a reason for hiding this comment

Uh oh!

vitalybuka commented Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AaronBallman commented Feb 11, 2025

Uh oh!

AaronBallman left a comment

Choose a reason for hiding this comment

Uh oh!

efriedma-quic commented Feb 11, 2025

Uh oh!

AaronBallman commented Feb 12, 2025

Uh oh!

vitalybuka commented Feb 13, 2025

Uh oh!

Uh oh!

llvm-ci commented Feb 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vitalybuka commented Jan 30, 2025 •

edited

Loading

vitalybuka commented Feb 11, 2025 •

edited

Loading