Skip to content

Bump confluent-kafka to 2.11.1 #21259

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Sep 11, 2025
Merged

Bump confluent-kafka to 2.11.1 #21259

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
13f3f6f
Bump confluent-kafka
Kyle-Neale Sep 4, 2025
bc794e9
Add changelog
Kyle-Neale Sep 4, 2025
83c6ad8
Merge branch 'master' into kyle.neale/bump-kafka-dep
Kyle-Neale Sep 4, 2025
335d79a
Merge branch 'master' into kyle.neale/bump-kafka-dep
Kyle-Neale Sep 9, 2025
a9b3669
Merge branch 'master' into kyle.neale/bump-kafka-dep
Kyle-Neale Sep 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .builders/images/linux-aarch64/build_script.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ if [[ "${DD_BUILD_PYTHON_VERSION}" == "3" ]]; then
LDFLAGS="${LDFLAGS} -L/usr/local/lib -lkrb5 -lgssapi_krb5 -llmdb" \
DOWNLOAD_URL="https://github.com/confluentinc/librdkafka/archive/refs/tags/v{{version}}.tar.gz" \
VERSION="${kafka_version}" \
SHA256="5bd1c46f63265f31c6bfcedcde78703f77d28238eadf23821c2b43fc30be3e25" \
SHA256="a2c87186b081e2705bb7d5338d5a01bc88d43273619b372ccb7bb0d264d0ca9f" \
RELATIVE_PATH="librdkafka-{{version}}" \
bash install-from-source.sh --enable-sasl --enable-curl
always_build+=("confluent-kafka")
Expand Down
2 changes: 1 addition & 1 deletion .builders/images/linux-x86_64/build_script.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ if [[ "${DD_BUILD_PYTHON_VERSION}" == "3" ]]; then
LDFLAGS="${LDFLAGS} -L/usr/local/lib -lkrb5 -lgssapi_krb5 -llmdb" \
DOWNLOAD_URL="https://github.com/confluentinc/librdkafka/archive/refs/tags/v{{version}}.tar.gz" \
VERSION="${kafka_version}" \
SHA256="5bd1c46f63265f31c6bfcedcde78703f77d28238eadf23821c2b43fc30be3e25" \
SHA256="a2c87186b081e2705bb7d5338d5a01bc88d43273619b372ccb7bb0d264d0ca9f" \
RELATIVE_PATH="librdkafka-{{version}}" \
bash install-from-source.sh --enable-sasl --enable-curl
always_build+=("confluent-kafka")
Expand Down
2 changes: 1 addition & 1 deletion .builders/images/macos/extra_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ if [[ "${DD_BUILD_PYTHON_VERSION}" == "3" ]]; then
LDFLAGS="${LDFLAGS} -L${DD_PREFIX_PATH}/lib -lgssapi_krb5 -llmdb" \
DOWNLOAD_URL="https://github.com/confluentinc/librdkafka/archive/refs/tags/v{{version}}.tar.gz" \
VERSION="${kafka_version}" \
SHA256="5bd1c46f63265f31c6bfcedcde78703f77d28238eadf23821c2b43fc30be3e25" \
SHA256="a2c87186b081e2705bb7d5338d5a01bc88d43273619b372ccb7bb0d264d0ca9f" \
RELATIVE_PATH="librdkafka-{{version}}" \
bash install-from-source.sh --prefix="${DD_PREFIX_PATH}" --enable-sasl --enable-curl

Expand Down
2 changes: 1 addition & 1 deletion .builders/images/windows-x86_64/build_script.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Write-Host "Will build librdkafka $kafka_version"
Get-RemoteFile `
-Uri "https://github.com/confluentinc/librdkafka/archive/refs/tags/v${kafka_version}.tar.gz" `
-Path "librdkafka-${kafka_version}.tar.gz" `
-Hash '5bd1c46f63265f31c6bfcedcde78703f77d28238eadf23821c2b43fc30be3e25'
-Hash 'a2c87186b081e2705bb7d5338d5a01bc88d43273619b372ccb7bb0d264d0ca9f'
7z x "librdkafka-${kafka_version}.tar.gz" -o"C:\"
7z x "C:\librdkafka-${kafka_version}.tar" -o"C:\librdkafka"
Remove-Item "librdkafka-${kafka_version}.tar.gz"
Expand Down
2 changes: 1 addition & 1 deletion .ddev/ci/scripts/kafka_consumer/linux/32_install_kerberos.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ sudo apt install -y --no-install-recommends build-essential libkrb5-dev wget sof
# Install librdkafka from source since no binaries are available for the distribution we use on the CI:
git clone https://github.com/confluentinc/librdkafka
cd librdkafka
git checkout v2.8.0
git checkout v2.11.1
sudo ./configure --install-deps --prefix=/usr
make
sudo make install
Expand Down
2 changes: 1 addition & 1 deletion agent_requirements.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ cachetools==6.2.0
clickhouse-cityhash==1.0.2.4
clickhouse-driver==0.2.9
cm-client==45.0.4
confluent-kafka==2.8.0
confluent-kafka==2.11.1
cryptography==45.0.6
ddtrace==3.12.3
dnspython==2.7.0
Expand Down
1 change: 1 addition & 0 deletions kafka_consumer/changelog.d/21259.added
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Bump `confluent-kafka` to 2.11.1
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor but still, please, add newline

2 changes: 1 addition & 1 deletion kafka_consumer/hatch.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# If you bump the `confluent-kafka` version, also bump the `librdkafka` version in the `32_install_kerberos.sh` file
post-install-commands = [
"python -m pip uninstall -y confluent-kafka",
"python -m pip install --no-binary confluent-kafka confluent-kafka==2.8.0",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably need to bring a newer one now that we are using confluent kafka and dropped bitnami?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: I assumed that they might have a python client with versions matching the confluent platform version we are using (as is commonly done), but looking at the client versions this is not the case. 2.11.1 seems to be the latest one so we should be good there. It was a wild guess about why tests could be failing.

"python -m pip install --no-binary confluent-kafka confluent-kafka==2.11.1",
]

[[envs.default.matrix]]
Expand Down
2 changes: 1 addition & 1 deletion kafka_consumer/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ license = "BSD-3-Clause"

[project.optional-dependencies]
deps = [
"confluent-kafka==2.8.0",
"confluent-kafka==2.11.1",
"fastavro==1.12.0",
"protobuf==6.32.0",
]
Expand Down
Loading