Skip to content

Conversation

JesseLovelace
Copy link

@JesseLovelace JesseLovelace commented Sep 4, 2025

Rationale for this change

The currently bundled version of GRPC is several years out of date

What changes are included in this PR?

There's a few things going on here so let me explain:

  • I've upgraded grpc
  • Upgrading grpc requires updating protobuf, so I've done that as well
  • Updating protbuf requires updating abseil, so I've done that as well
  • The upgraded version of abseil has an updated dependency tree, so I've updated the build_absl macro
  • The new version of protobuf introduces a dependency on abseil. So, I had to move the build_absl macro up farther in the cmake file so that it's defined when protobuf references it
  • The new version of protobuf also exports the utf8_range and utf8_validity libraries, as well as the upb library, which are needed at runtime, so I added those to the static library list
  • The new version of grpc no longer exports a library called "upb", instead it exports a bunch of smaller upb libraries. However, these libraries are identical to the one exported by protobuf, so including them both would create conflicts. Instead I just linked the grpc libraries to the upb library exported by protobuf, which works. The same goes for utf8_range
  • I added -DCMAKE_CXX_EXTENSIONS=OFF to the common cmake args. I think this is actually intended to be the case anyway, based on this, but that wasn't propagating to the ThirdPartyToolchain build. This was causing all bundled libraries to default to building with std=gnu++17, whereas the arrow libraries were being build with std=c++17, which was causing runtime issues with the new version of absl
  • I removed the check for protobuf versions greater than 3.22 from the build_opentelemetry macro. It isn't needed anymore as of Support protobuf 3.22 or upper open-telemetry/opentelemetry-cpp#2163, and https://github.com/apache/arrow/pull/46509/files#diff-39a645630afbfb1702af73bcb3fcdb13d87be3d78fcf501f2dcf93000f4aa738 bumped it to the version with that fix
  • I changed the protobuf version check in https://github.com/apache/arrow/pull/47408/files from 32.0 to 22.0. I'm pretty sure it was a typo and should have been 22.0 in the first place, and I was running into the issue without it. I also specifically added a check for when protobuf is vendored, since Protobuf_VERSION is not populated at build time if that's the case

I think that's everything. Github is weirdly showing a bunch of things that I didn't actually touch in the diff, I think it maybe has something to do with the fact that I moved some of the macros higher up in the file. If you open it in VScode it shows a much more sensible diff

Are these changes tested?

I ran the ORC tests, Flight tests, and Telemetry tests, since those seem to the parts of the repo that have dependencies on grpc, protobuf, and abseil. Everything passed. Let me know if I need to do anything else

Are there any user-facing changes?

No

@github-actions github-actions bot added the awaiting review Awaiting review label Sep 4, 2025
Copy link

github-actions bot commented Sep 4, 2025

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@JesseLovelace JesseLovelace changed the title [C++] Upgrade vendored grpc, protobuf, abseil GH-45063: [C++] Upgrade vendored grpc, protobuf, abseil Sep 4, 2025
Copy link
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the great work here, there seems to be some CI issues for some specific builds, seems to be related to some missing targets

Comment on lines -3180 to -3181
set_property(TARGET absl::bad_optional_access
PROPERTY INTERFACE_LINK_LIBRARIES absl::config absl::raw_logging_internal)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could this removal be the reason for some of the CI failures?

CMake Error at cmake_modules/BuildUtils.cmake:142 (file):
  Error evaluating generator expression:

    $<TARGET_FILE:absl::bad_optional_access>

  Target "absl::bad_optional_access" is not an executable or library.
Call Stack (most recent call first):
  src/arrow/CMakeLists.txt:1030 (arrow_create_merged_static_lib)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bad_optional_access doesn't exist in this updated version of absl, but it looks like I missed removing the declaration of it even though I removed this link libraries. Not sure if the script didn't catch it or if I made a mistake, but I'll fully remove it, hopefully that fixes the issue

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Sep 5, 2025
@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 5, 2025
@JesseLovelace
Copy link
Author

@raulcd I think I fixed the issue, can you run the CI again?

@JesseLovelace
Copy link
Author

@raulcd Looking at the failure for C GLib & Ruby / AMD64 Ubuntu 22.04 GLib & Ruby (pull_request), I see this:

/usr/include/nlohmann/json.hpp:2020:36: note: no known conversion for argument 1 from 'absl::lts_20250512::string_view' {aka 'std::basic_string_view<char>'} to 'nlohmann::basic_json<>::size_type' {aka 'long unsigned int'} 2020 | reference operator[](size_type idx)

It seems that the Ubuntu build uses a verison of nlohmann_json which is installed on the local machine (in /usr/include) whereas the other environments install it from source. The version that this ubuntu build has locally is 3 years out of date. I think this may be the reason for the failure. Is it possible to reconfigure that build to install nlohmann_json from source?

@JesseLovelace
Copy link
Author

The failure from AMD64 Ubuntu 22.04 C++ ASAN UBSAN looks like it's a failure of arrow-azurefs-test which should be unrelated to this PR but i'll look into if i can reproduce it locally

@JesseLovelace
Copy link
Author

JesseLovelace commented Sep 22, 2025

The failure from AMD64 Windows 2022 AVX2 says that it's failing to install the patch for ORC. I don't think anything in this change should affect that, and the patch works in other builds

@JesseLovelace
Copy link
Author

I'm not entirely sure how to make sense of the R failures because I have no familiarity with R. Does the R library link to the cpp library in some way?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants