[D148266] [clang][driver] Linking to just-built libc++.dylib when bootstrapping libc++ with clang

[ldionne]

This issue tracks picking up https://reviews.llvm.org/D148266 from the Phabricator archive.

Also related to rdar://107060541.

[llvmbot]

@llvm/issue-subscribers-clang-driver

Author: Louis Dionne (ldionne)

This issue tracks picking up https://reviews.llvm.org/D148266 from the Phabricator archive.

Also related to rdar://107060541.

[hoyhoy]

I just built with your patch, but I'm getting...

 /opt/llvm18/bin/clang++ -v  ~/backup/cmake_bootstrap.cxx -o cmake_bootstrap_5096_test
clang version 18.1.0rc
Target: x86_64-apple-darwin23.4.0
Thread model: posix
InstalledDir: /opt/llvm18/bin
Configuration file: /opt/llvm18/etc/x86_64-apple-darwin23.4.0-clang++.cfg
System configuration file directory: /opt/llvm18/etc
 "/opt/llvm18/bin/clang-18" -cc1 -triple x86_64-apple-macosx14.4.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -dumpdir cmake_bootstrap_5096_test- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name cmake_bootstrap.cxx -mrelocation-model pic -pic-level 2 -mframe-pointer=all -ffp-contract=on -fno-rounding-math -funwind-tables=2 -fcompatibility-qualified-id-block-type-checking -fvisibility-inlines-hidden-static-local-var -fbuiltin-headers-in-system-modules -fdefine-target-os-macros -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -fdebug-compilation-dir=/Users/jperrie/llvm-project-18.1.0.src/build -v -fcoverage-compilation-dir=/Users/jperrie/llvm-project-18.1.0.src/build -resource-dir /opt/llvm18/lib/clang/18 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -I/opt/llvm18/include -internal-isystem /opt/llvm18/bin/../include/c++/v1 -internal-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /opt/llvm18/lib/clang/18/include -internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -fdeprecated-macro -ferror-limit 19 -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcolor-diagnostics -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o -x c++ /Users/jperrie/backup/cmake_bootstrap.cxx
clang -cc1 version 18.1.0rc based upon LLVM 18.1.0rc default target x86_64-apple-darwin23.4.0
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /opt/llvm18/include
 /opt/llvm18/bin/../include/c++/v1
 /opt/llvm18/lib/clang/18/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
 "/opt/llvm18/bin/ld64.lld" -demangle -dynamic -arch x86_64 -platform_version macos 14.4.0 14.4.0 -L /opt/llvm18/bin/../lib -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -mllvm -enable-linkonceodr-outlining -o cmake_bootstrap_5096_test -L/opt/llvm18/lib/darwin -L/opt/llvm18/lib -L/opt/llvm18/lib -L/opt/llvm18/lib/clang/18/lib/darwin -rpath /opt/llvm18/lib /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o -lc++ -lSystem /opt/llvm18/lib/clang/18/lib/darwin/libclang_rt.osx.a
ld64.lld: error: undefined symbol: operator new(unsigned long)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol check_cxx17()+0xe)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol main+0x15)

ld64.lld: error: undefined symbol: __cxa_throw
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol std::__1::__throw_bad_optional_access[abi:ne180100]()+0x38)

ld64.lld: error: undefined symbol: __cxa_allocate_exception
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol std::__1::__throw_bad_optional_access[abi:ne180100]()+0xe)

ld64.lld: error: undefined symbol: vtable for std::exception
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol std::exception::exception[abi:ne180100]()+0xf)

ld64.lld: error: undefined symbol: std::terminate()
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol __clang_call_terminate+0xa)

ld64.lld: error: undefined symbol: __cxa_begin_catch
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol __clang_call_terminate+0x5)

ld64.lld: error: undefined symbol: operator delete(void*)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol std::__1::default_delete<int>::operator()[abi:ne180100](int*) const+0x27)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol std::__1::default_delete<Class>::operator()[abi:ne180100](Class*) const+0x27)

ld64.lld: error: undefined symbol: __gxx_personality_v0
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(symbol EH_Frame+0x13)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(__compact_unwind+0x130)
>>> referenced by /var/folders/sg/by4ymb_1491658nz5rdt1k940000gn/T/cmake_bootstrap-9f5e8c.o:(__compact_unwind+0x70)
>>> referenced 3 more times
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

I even rebuilt the runtimes with a second stage using the installed llvm, but that didn't work either.

[hoyhoy]

Working llvm 17.0.6 vs non-working llvm 18.1.0...

# working llvm 17.0.6
$ sudo dtruss -a /opt/llvm17/bin/clang++  cmake_bootstrap.cxx -o cmake_test 2>&1 | grep 'open.*\.dylib'
86197/0x14cb7a:      1167      27     25 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1201      15     14 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1247      14     13 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1345      11     10 open("@rpath/libc++.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86197/0x14cb7a:      1394      17     16 open("/opt/llvm17/lib/libc++.1.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1441      14     13 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1510       4      3 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86197/0x14cb7a:      1560      17     16 open("/opt/llvm17/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:      1591      13     12 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:     15717      45     43 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:     33877      42     38 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86197/0x14cb7a:     34208      19     18 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:       899      23     21 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:       930      15     13 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:       958      13     12 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:      1040       4      2 open("@rpath/libc++.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86198/0x14cb7c:      1085      16     15 open("/opt/llvm17/lib/libc++.1.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:      1109      13     12 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:      1170       4      3 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86198/0x14cb7c:      1217      16     15 open("/opt/llvm17/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:      1241      13     12 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:     14174      27     25 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:     30621      22     20 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86198/0x14cb7c:     30886      16     14 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0

# not working llvm 18.0.1
$ sudo dtruss -a /opt/llvm18/bin/clang++  cmake_bootstrap.cxx -o cmake_test 2>&1 | grep 'open.*\.dylib'
86205/0x14cbfd:      1114      74     36 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1157      16     15 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1208      15     14 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1316      11     10 open("@rpath/libc++.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86205/0x14cbfd:      1354      17     16 open("/opt/llvm17/lib/libc++.1.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1402      14     13 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1468       4      3 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86205/0x14cbfd:      1505      17     16 open("/opt/llvm17/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:      1537      14     13 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:     29702      42     40 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:     47315      39     35 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86205/0x14cbfd:     47616      19     17 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:       837      23     22 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:       869      13     11 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:       947      35     34 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:      1021       3      2 open("@rpath/libc++.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86206/0x14cc08:      1050      14     13 open("/opt/llvm17/lib/libc++.1.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:      1071      11     10 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:      1119       3      2 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86206/0x14cc08:      1148      13     12 open("/opt/llvm17/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:      1169      11     10 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:     26020      32     30 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:     41340      22     20 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86206/0x14cc08:     41585      15     14 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:       947      25     23 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:       981      15     14 open("/usr/local/lib/libzstd.1.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:      1008      14     12 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:      1106       4      3 open("@rpath/libc++.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86207/0x14cc0a:      1143      17     16 open("/opt/llvm17/lib/libc++.1.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:      1169      14     12 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:      1229       4      3 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
86207/0x14cc0a:      1266      17     16 open("/opt/llvm17/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:      1292      14     12 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:     26033      28     26 open("/usr/local/lib/libzstd.1.5.5.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:     42373      91     36 open("/opt/llvm17/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:     42637      17     15 open("/opt/llvm17/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
86207/0x14cc0a:     49265      16     14 open("/opt/llvm18/bin/../lib/libc++.dylib\0", 0x1000000, 0x0)		 = 3 0
86207/0x14cc0a:     70134      17     16 open("/opt/llvm18/bin/../lib/libunwind.dylib\0", 0x1000000, 0x0)		 = 3 0

[hoyhoy]

Seems like something with libcxx is messed up. Or maybe because I configured libunwind this time?

[hoyhoy]

I also tried manually changing libcxx's rpath out to the llvm 18.1.0 binary ...

llvm-install-name-tool -delete_rpath /opt/llvm17/lib /opt/llvm18/bin/clang-18
llvm-install-name-tool -delete_rpath /opt/llvm17/lib /opt/llvm18/bin/lld
llvm-install-name-tool -change @rpath/libc++.1.dylib /opt/llvm18/lib/libc++.1.dylib /opt/llvm18/bin/clang-18
llvm-install-name-tool -change @rpath/libc++abi.1.dylib /opt/llvm18/lib/libc++abi.1.dylib /opt/llvm18/bin/clang-18
llvm-install-name-tool -change @rpath/libc++.1.dylib /opt/llvm18/lib/libc++.1.dylib /opt/llvm18/bin/lld
llvm-install-name-tool -change @rpath/libc++abi.1.dylib /opt/llvm18/lib/libc++abi.1.dylib /opt/llvm18/bin/lld

But, when I do that, (or, if I deploy my working llvm17 to /opt/llvm18, compile llvm18 with the /opt/llvm18/bin/clang-17, and then overwrite the previous /opt/llvm18 installation directory with llvm 18.1.0), I get this...

 $ /opt/llvm18/bin/clang++ ~/backup/cmake_bootstrap.cxx
dyld[4913]: Symbol not found: ___cxa_pure_virtual
  Referenced from: <4C4C4476-5555-3144-A184-84B6267412E1> /opt/llvm18/bin/clang-18
  Expected in:     <4C4C4429-5555-3144-A139-01A02C015AD3> /opt/llvm18/lib/libc++.1.0.dylib
Abort trap: 6

[hoyhoy]

Looks like libunwind is the problem actually.

5187/0x15e640:      1596       4      3 open("@rpath/libunwind.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
 5187/0x15e640:      1598       2      1 open("@rpath\0", 0x100000, 0x0)		 = -1 Err#2
 5187/0x15e640:      1604       3      2 stat64("/opt/llvm17/lib/libunwind.1.dylib\0", 0x7FF7BCFB6DA0, 0x0)		 = -1 Err#2
 5187/0x15e640:      1615       4      2 stat64("/System/Volumes/Preboot/Cryptexes/OS/opt/llvm17/lib/libunwind.1.dylib\0", 0x7FF7BCFB6DA0, 0x0)		 = -1 Err#2
 5187/0x15e640:      1626       9      8 stat64("/opt/llvm18/lib/libunwind.1.dylib\0", 0x7FF7BCFB6DA0, 0x0)		 = 0 0
 5187/0x15e640:      1632       5      4 stat64("/opt/llvm18/lib/libunwind.1.dylib\0", 0x7FF7BCFB6800, 0x0)		 = 0 0
 5187/0x15e640:      1659      26     25 open("/opt/llvm18/lib/libunwind.1.dylib\0", 0x0, 0x0)		 = 3 0
 5187/0x15e640:      1665       7      5 mmap(0x0, 0x106C0, 0x1, 0x40002, 0x3, 0x0)		 = 0x109EA5000 0
 5187/0x15e640:      1667       2      1 fcntl(0x3, 0x32, 0x7FF7BCFB6910)		 = 0 0
 5187/0x15e640:      1669       1      0 close(0x3)		 = 0 0
 5187/0x15e640:      1687      14     13 open("/opt/llvm18/lib/libunwind.1.0.dylib\0", 0x0, 0x0)		 = 3 0
 5187/0x15e640:      1692       5      3 fcntl(0x3, 0x62, 0x7FF7BCFB65C8)		 = 0 0
 5187/0x15e640:      1704      11      9 mmap(0x109EB6000, 0xC000, 0x5, 0x40012, 0x3, 0x0)		 = 0x109EB6000 0
 5187/0x15e640:      1709       4      2 mmap(0x109EC2000, 0x1000, 0x3, 0x40012, 0x3, 0xC000)		 = 0x109EC2000 0
 5187/0x15e640:      1712       3      2 mmap(0x109EC3000, 0x1000, 0x3, 0x40012, 0x3, 0xD000)		 = 0x109EC3000 0
 5187/0x15e640:      1714       3      1 mmap(0x109EC4000, 0x26C0, 0x1, 0x40012, 0x3, 0xE000)		 = 0x109EC4000 0
 5187/0x15e640:      1716       1      0 close(0x3)		 = 0 0
 5187/0x15e640:      1724       3      2 munmap(0x109EA5000, 0x106C0)		 = 0 0
 5187/0x15e640:      1900     107     43 open("/dev/dtracehelper\0", 0x2, 0x0)		 = 3 0
 5187/0x15e640:      2271     375    369 ioctl(0x3, 0x80086804, 0x7FF7BCFB68A8)		 = 0 0
 5187/0x15e640:      2275       4      2 close(0x3)		 = 0 0
 5187/0x15e640:      2506       6      4 write(0x2, "dyld[5187]: \0", 0xC)		 = 12 0
 5187/0x15e640:      2508       2      0 write(0x2, "Symbol not found: ___cxa_pure_vi\0", 0x20)		 = 32 0
 5187/0x15e640:      2509       1      0 write(0x2, "rtual\n  Referenced from: <4C4C44\0", 0x20)		 = 32 0
 5187/0x15e640:      2511       1      0 write(0x2, "76-5555-3144-A184-84B6267412E1> \0", 0x20)		 = 32 0
 5187/0x15e640:      2512       1      0 write(0x2, "/opt/llvm18/bin/clang-18\n  Expec\0", 0x20)		 = 32 0
 5187/0x15e640:      2514       2      0 write(0x2, "ted in:     <4C4C4429-5555-3144-\0", 0x20)		 = 32 0
 5187/0x15e640:      2515       1      0 write(0x2, "A139-01A02C015AD3> /opt/llvm18/l\0", 0x20)		 = 32 0
 5187/0x15e640:      2517       1      0 write(0x2, "ib/libc++.1.0.dylib\n\0", 0x14)		 = 20 0
 5187/0x15e640:      2519       2      0 sigprocmask(0x2, 0x7FF7BCFB703C, 0x0)		 = 0x0 0
 5187/0x15e640:      2526       8      5 abort_with_payload(0x6, 0x4, 0x7FF7BCFB7490)		 = 0 0

[hoyhoy]

I tried disabling libunwind, and this happens...

8986/0x17cdff:      1411      14     13 open("/opt/llvm18/lib/libc++.1.0.dylib\0", 0x0, 0x0)		 = 3 0
38986/0x17cdff:      1416       5      4 fcntl(0x3, 0x62, 0x7FF7BE9B9768)		 = 0 0
38986/0x17cdff:      1429      11      9 mmap(0x108C1F000, 0xA4000, 0x5, 0x40012, 0x3, 0x0)		 = 0x108C1F000 0
38986/0x17cdff:      1435       4      2 mmap(0x108CC3000, 0x5000, 0x3, 0x40012, 0x3, 0xA4000)		 = 0x108CC3000 0
38986/0x17cdff:      1438       4      2 mmap(0x108CC8000, 0x2000, 0x3, 0x40012, 0x3, 0xA9000)		 = 0x108CC8000 0
38986/0x17cdff:      1440       3      1 mmap(0x108CD0000, 0x56C88, 0x1, 0x40012, 0x3, 0xAB000)		 = 0x108CD0000 0
38986/0x17cdff:      1442       1      0 close(0x3)		 = 0 0
38986/0x17cdff:      1451       4      2 munmap(0x108B1D000, 0x101C88)		 = 0 0
38986/0x17cdff:      1472       4      3 open("@rpath/libc++abi.1.dylib\0", 0x0, 0x0)		 = -1 Err#2
38986/0x17cdff:      1474       2      1 open("@rpath\0", 0x100000, 0x0)		 = -1 Err#2
38986/0x17cdff:      1485       9      7 stat64("/opt/llvm18/lib/libc++abi.1.dylib\0", 0x7FF7BE9B9E60, 0x0)		 = 0 0
38986/0x17cdff:      1491       5      4 stat64("/opt/llvm18/lib/libc++abi.1.dylib\0", 0x7FF7BE9B98C0, 0x0)		 = 0 0
38986/0x17cdff:      1517      26     25 open("/opt/llvm18/lib/libc++abi.1.dylib\0", 0x0, 0x0)		 = 3 0
38986/0x17cdff:      1528      11      9 mmap(0x0, 0x4B7F0, 0x1, 0x40002, 0x3, 0x0)		 = 0x1084A1000 0
38986/0x17cdff:      1530       2      0 fcntl(0x3, 0x32, 0x7FF7BE9B99D0)		 = 0 0
38986/0x17cdff:      1531       2      0 close(0x3)		 = 0 0
38986/0x17cdff:      1549      14     13 open("/opt/llvm18/lib/libc++abi.1.0.dylib\0", 0x0, 0x0)		 = 3 0
38986/0x17cdff:      1554       4      3 fcntl(0x3, 0x62, 0x7FF7BE9B9688)		 = 0 0
38986/0x17cdff:      1566      11     10 mmap(0x108901000, 0x2A000, 0x5, 0x40012, 0x3, 0x0)		 = 0x108901000 0
38986/0x17cdff:      1571       3      2 mmap(0x10892B000, 0x4000, 0x3, 0x40012, 0x3, 0x2A000)		 = 0x10892B000 0
38986/0x17cdff:      1574       3      1 mmap(0x10892F000, 0x1000, 0x3, 0x40012, 0x3, 0x2E000)		 = 0x10892F000 0
38986/0x17cdff:      1576       3      1 mmap(0x108930000, 0x1C7F0, 0x1, 0x40012, 0x3, 0x2F000)		 = 0x108930000 0
38986/0x17cdff:      1578       1      0 close(0x3)		 = 0 0
38986/0x17cdff:      1586       3      2 munmap(0x1084A1000, 0x4B7F0)		 = 0 0
38986/0x17cdff:      1747     106     43 open("/dev/dtracehelper\0", 0x2, 0x0)		 = 3 0
38986/0x17cdff:      2104     358    356 ioctl(0x3, 0x80086804, 0x7FF7BE9B99C8)		 = 0 0
38986/0x17cdff:      2108       4      2 close(0x3)		 = 0 0
38986/0x17cdff:      2332       6      4 write(0x2, "dyld[38986]: \0", 0xD)		 = 13 0
38986/0x17cdff:      2334       2      1 write(0x2, "Symbol not found: ___cxa_pure_vi\0", 0x20)		 = 32 0
38986/0x17cdff:      2336       1      0 write(0x2, "rtual\n  Referenced from: <4C4C44\0", 0x20)		 = 32 0
38986/0x17cdff:      2337       1      0 write(0x2, "76-5555-3144-A184-84B6267412E1> \0", 0x20)		 = 32 0
38986/0x17cdff:      2339       1      0 write(0x2, "/opt/llvm18/bin/clang-18\n  Expec\0", 0x20)		 = 32 0
38986/0x17cdff:      2340       1      0 write(0x2, "ted in:     <4C4C444F-5555-3144-\0", 0x20)		 = 32 0
38986/0x17cdff:      2341       1      0 write(0x2, "A1AE-E57703862411> /opt/llvm18/l\0", 0x20)		 = 32 0
38986/0x17cdff:      2343       1      0 write(0x2, "ib/libc++.1.0.dylib\n\0", 0x14)		 = 20 0
38986/0x17cdff:      2345       2      0 sigprocmask(0x2, 0x7FF7BE9BA11C, 0x0)		 = 0x0 0
38986/0x17cdff:      2353       9      7 abort_with_payload(0x6, 0x4, 0x7FF7BE9BA570)		 = 0 0

[hoyhoy]

OK, with your patch, everything works! Thanks a lot.

I ended up having some issues where I was pulling in binaries from /usr/local compiled with previous versions of clang. I just recompiled with the latest XCode 15.3 on macOS 14.4 with mostly everything set to default using libcxx and libcxxabi.

Here's my build script for anyone else who is seeing these issues. My problems were RPATH related. I had to remove a libzstd that was linked to a libc++ that no longer existed.

#!/bin/sh

BUILD_DIR=build

rm -Rf ${BUILD_DIR}
mkdir ${BUILD_DIR}

export PATH="/usr/bin:/bin"
INSTALL_PATH="/opt/llvm18"

MACOS_SDK_ROOT=$(xcrun --show-sdk-path)
MACOS_VERSION=$(sw_vers -productVersion)

unset ASAN_OPTIONS
unset MSAN_OPTIONS
unset TSAN_OPTIONS
unset DYLD_LIBRARY_PATH
unset LD_LIBRARY_PATH
unset LIBPATH
unset LIBRARY_PATH
unset CPLUS_INCLUDE_PATH
unset CPATH

cmake_args=(
  -DCMAKE_BUILD_TYPE:STRING=Release
  -DLLVM_ENABLE_RUNTIMES:STRING="libcxx;libcxxabi;compiler-rt"
  -DLLVM_ENABLE_PROJECTS:STRING="clang;clang-tools-extra;lldb;lld"
  
  -DDEFAULT_SYSROOT:PATH=${MACOS_SDK_ROOT}

  -DLLVM_ENABLE_LIBCXX:BOOL=ON
  -DLIBCXXABI_USE_COMPILER_RT:BOOL=ON
  -DCLANG_DEFAULT_CXX_STDLIB:STRING=libc++

  -DCLANG_CONFIG_FILE_SYSTEM_DIR:PATH="${INSTALL_PATH}/etc"

  -DCMAKE_INSTALL_PREFIX:PATH="${INSTALL_PATH}"
  -DLLVM_INSTALL_BINUTILS_SYMLINKS:BOOL=ON
  -DLLVM_INSTALL_CCTOOLS_SYMLINKS:BOOL=ON

  -DCLANG_INCLUDE_TESTS:BOOL=OFF
  -DLLVM_INCLUDE_TESTS:BOOL=OFF
  -DLLDB_ENABLE_LIBEDIT:BOOL=ON
  -DLLDB_ENABLE_CURSES:BOOL=ON
  -DLLVM_ENABLE_TERMINFO:BOOL=ON

  -DLLVM_BUILD_TOOLS:BOOL=ON

  -DHAVE_CXX_ATOMICS64_WITHOUT_LIB:BOOL=ON
  -DHAVE_CXX_ATOMICS_WITHOUT_LIB:BOOL=ON

  -DLLVM_PARALLEL_COMPILE_JOBS:STRING=17
  -DLLVM_PARALLEL_LINK_JOBS:STRING=2

  -DLLVM_ENABLE_LLD:BOOL=ON

  -DCMAKE_OSX_DEPLOYMENT_TARGET:STRING="${MACOS_VERSION}"

  -DCOMPILER_RT_BUILD_BUILTINS:BOOL=ON
  -DCOMPILER_RT_ENABLE_IOS:BOOL=OFF
  -DCOMPILER_RT_ENABLE_WATCHOS:BOOL=OFF
  -DCOMPILER_RT_ENABLE_TVOS:BOOL=OFF
  -DCOMPILER_RT_ENABLE_MACCATALYST:BOOL=OFF

  -DCOMPILER_RT_BUILD_LIBFUZZER:BOOL=ON
  -DCOMPILER_RT_BUILD_MEMPROF:BOOL=ON
  -DCOMPILER_RT_BUILD_PROFILE:BOOL=ON
  -DCOMPILER_RT_BUILD_SANITIZERS:BOOL=ON
  -DCOMPILER_RT_BUILD_XRAY:BOOL=ON

  -DLIBCXX_CXX_ABI:STRING=libcxxabi
  -DLIBCXX_USE_COMPILER_RT:BOOL=ON
  -DLIBCXXABI_USE_COMPILER_RT:BOOL=ON
  -DLIBCXX_HAS_GCC_S_LIB:BOOL=OFF

  -DLLVM_ENABLE_RTTI:BOOL=ON
  -DLLVM_ENABLE_FFI:BOOL=ON
)

cmake "${cmake_args[@]}" -S llvm -G Ninja -B "${BUILD_DIR}" && ninja -C "${BUILD_DIR}"

[hoyhoy]

Also, if your clang-18 is having libc++ linking issues, you can just override the load path with the following...

install_name_tool -change /usr/lib/libc++.1.dylib /opt/llvm18/lib/libc++.1.dylib /opt/llvm18/bin/clang-18 

(where /opt/llvm18 is your CMAKE_INSTALL_PREFIX, normally /usr/local).

This is the issue that @ldionne was talking about. With the bootstrapped build (i.e. building the runtimes and the projects with a single ninja or cmake --build), clang-18 load path points to the MacOS platform libc++ (i.e. /usr/lib/libc++.1.dylib), and NOT llvm-18.x's libc++ (i.e. ${CMAKE_INSTALL_PREFIX}/lib/libc++.1.dylib).

If you look at the binary with otool -L, it refer to the libc++ built from the llvm source code. clang-18 and lld's LC_RPATH should point to your CMAKE_INSTALL_PREFIX as well.

 $ otool -L /opt/llvm18/bin/clang-18
/opt/llvm18/bin/clang-18:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.12)
	/opt/xcode15/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.5)
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	/opt/llvm18/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1700.255.0)

$ otool -l /opt/llvm18/bin/clang-18
# relevant lines only...
Load command 18
  cmd LC_LOAD_DYLIB
  name /opt/llvm18/lib/libc++.1.dylib (offset 24)
  compatibility version 1.0.0
cmd LC_RPATH
  cmdsize 32
  path /opt/llvm18/lib (offset 12)
Load command 22
  cmd LC_RPATH
  path /opt/llvm18/lib/darwin (offset 12)

The other odd thing I've been seeing is that the boostrap build doesn't deploy lib/darwin RPATH. I end up having to manually symbolically link that after ninja install.

ln -s /opt/llvm18/lib/clang/18/lib/darwin /opt/llvm18/lib/darwin

If I compile the runtimes separately and install them separately, the installation does deploy to lib/darwin, but not lib/clang/18/lib/darwin. Not sure if there is a cmake setting for that or not. Internally, clang seems to need both installed directories for some reason.

[hoyhoy]

Just compiled 18.1.3, and it still has issues. Is x86_64 not supported on macOS at this point? It's nearly impossible to get this to install correctly. Also, this diff isn't in 18.1.3. What's the best way to build at this point? Use XCode 15 to build clang 18.1.3, and then clang 18.1.3 to compile the runtimes in two steps?