-
Notifications
You must be signed in to change notification settings - Fork 14.6k
MINOR: Add metrics corresponding to consumer rebalance listener metrics #20464
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: trunk
Are you sure you want to change the base?
MINOR: Add metrics corresponding to consumer rebalance listener metrics #20464
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds latency metrics for rebalance listener callbacks in Kafka Streams to provide better observability of task rebalancing operations. The implementation follows the existing Streams ThreadMetrics pattern and introduces three new metrics for measuring the time taken during task revocation, assignment, and loss operations.
Key changes:
- Created RebalanceMetrics class with factory methods for creating sensors
- Modified DefaultStreamsRebalanceListener to record latency metrics during rebalance operations
- Updated StreamThread to pass required metrics dependencies to the rebalance listener
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
File | Description |
---|---|
RebalanceMetrics.java | New metrics class providing static factory methods for creating rebalance latency sensors |
DefaultStreamsRebalanceListener.java | Enhanced to record latency metrics for tasks revoked, assigned, and lost operations |
StreamThread.java | Updated constructor calls to pass streamsMetrics and threadId to DefaultStreamsRebalanceListener |
DefaultStreamsRebalanceListenerTest.java | Added comprehensive tests for metrics recording in both success and exception scenarios |
RebalanceMetricsTest.java | Added unit tests to verify sensor creation logic and metric configuration |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
public Optional<Exception> onTasksAssigned(final StreamsRebalanceData.Assignment assignment) { | ||
final long start = time.milliseconds(); | ||
try { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The start time is captured outside the try block, but if an exception occurs during the method execution, the sensor will still record the latency in the finally block. This could lead to recording metrics for failed operations that didn't complete normally, potentially skewing the latency measurements.
Copilot uses AI. Check for mistakes.
public Optional<Exception> onAllTasksLost() { | ||
final long start = time.milliseconds(); | ||
try { | ||
taskManager.handleLostAll(); | ||
} catch (final Exception exception) { | ||
return Optional.of(exception); | ||
} finally { | ||
tasksLostSensor.record(time.milliseconds() - start); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Similar to onTasksAssigned, this method records latency metrics even when exceptions occur. The finally block will execute regardless of whether the operation completed successfully or failed, which may not provide accurate latency measurements for successful operations.
Copilot uses AI. Check for mistakes.
Background
Add new metrics for TasksLost, TasksAssigned and TasksRevoked in
StreamsRebalanceListener
.Features Implemented
Added three rebalance latency metrics to DefaultStreamsRebalanceListener:
tasks-revoked-latency-avg/max
- Average/max latency when revoking taskstasks-assigned-latency-avg/max
- Average/max latency when assigning taskstasks-lost-latency-avg/max
- Average/max latency when losing tasksMain Changes
RebalanceMetrics.java
tasksRevokedSensor()
,tasksAssignedSensor()
,tasksLostSensor()
StreamsMetricsImpl
and threadId parametersonTasksRevoked()
,onTasksAssigned()
, andonAllTasksLost()
methodsstreamsMetrics
andgetName()
when creatingDefaultStreamsRebalanceListener
DefaultStreamsRebalanceListenerTest
- Added 6 tests to verify metrics recording (both normal and exception cases)RebalanceMetricsTest
- Verifies sensor creation logic