OpenTelemetry metric plugin and dual emitting

[mqliang]

This PR add OpenTelemetry metric plugin.

Problem and Challenge:

The Pinot metric SPI is not OpenTelemetry friendly, when call the metric SPI to emit metric, it has concatenated all attributes/dimensions into metric name. The concatenating happens at every caller place, not inside plugin implementation.

For different metrics, the order of the concatenating are different.

Example of metric naming pattern includes:

• pinot.broker.<rawTableName>.<queryPhaseName>
• pinot.controller.<metricName>.<taskType>
• pinot.controller.<metricName><tableNameWithType>.<taskType>
• pinot.controller.<metricName>.<rawTableName>.<topicName>
• pinot.controller.<resourceName>.<metricName>
• pinot.minion.<tableName>.<taskType>.<metricName>
• pinot.minion.<taskType>.<metricName>
• pinot.minion.<tableName>.<metricName>
• etc

Proposed solution

The idea of this PR is using PinotMatricName to carry a String simplifiedMericName and Map<Stting, String> attributes, Yammer/Dropwizard will just ignore those two extra filed, but OpenTelemetry will use the simplifiedMericName as metric name and attach attributes when updating metric value.

For example, the metric of pinot.controller.Purge.NumMinionSubtasksWaiting will be treated as:

- fullMetricName: pinot.controller.Purge.NumMinionSubtasksWaiting
- simplifiedMetricName: NumMinionSubtasksWaiting
- attributes:
     - PinotMetricName: pinot.controller.Purge.NumMinionSubtasksWaiting
     - Component: Controller
     - TaskType: Purge

This approach has minimal impact to existing Yammer and Dropwizard metrics.

Alternative considered

Another approach is updating the Pinot metric SPI to be dimension friendly and concatenating metic name with dimensions at Yammer/Dropwizard plugin implementation layer. The concern of this approach is that many pinot users has dashboard/alert created on top of existing Yammer/Dropwizard metric. This approach will touch huge number of places and has very high risk of breaking existing Yammer/Dropwizard metric and causing regression.

Local Testing

Step 1: Launch a Opentelemetry-Collector.

➜  ~ docker run \
  -p 127.0.0.1:4317:4317 \
  -p 127.0.0.1:4318:4318 \
  -p 127.0.0.1:55679:55679 \
  otel/opentelemetry-collector-contrib:0.133.0 \
  2>&1 | tee collector-output.txt # Optionally tee output for easier search later

Step 2: Build and Run the Pinot Quick Demo cluster

Tweak DEFAULT_HTTP_METRIC_EXPORTER

First please change DEFAULT_HTTP_METRIC_EXPORTER endpoint as http://127.0.0.1:4318/v1/metrics. The Opentelemetry-Collector that we use to collect/display metrics is listening to localhost:4318 instead of 0.0.0.0:4318, so we have to emit/report/export metrics to 127.0.0.1:4318.

Please note that Docker port forwarding (meaning docker run -p 127.0.0.1:22784:4318 otel/opentelemetry-collector-contrib:0.133.0)will not work. As the /opentelemetry-collector container be configured to listen only on 127.0.0.1 (localhost) instead of 0.0.0.0 (all interfaces). Docker's port forwarding relies on the application binding to an address accessible from outside the container's localhost. This is a bug/feature-gap that opentelemetry-collector need to fix.

Enable OTel metric plugin

Change DEFAULT_METRICS_FACTORY_CLASS_NAME in pinot-spi/src/main/java/org/apache/pinot/spi/utils/CommonConstants.java as org.apache.pinot.plugin.metrics.opentelemetry.CompoundPinotMetricsFactory so that we can tryout Yammer and OTel metrics dual emitting.

There are huge number of testing are based on Yammer metric, if we make OpenTelemetry as default, it will bring down many testing.

Build and launch a Pinot cluster

# Build Pinot
# -Pbin-dist is required to build the binary distribution
# -Pbuild-shaded-jar is required to build the shaded jar, which is necessary for some features like spark connectors
$ ./mvnw clean install -DskipTests -Pbin-dist -Pbuild-shaded-jar

# Run the Quick Demo
$ cd build/
$ bin/quick-start-batch.sh

Step 3: Verifying metric being reported/exported

On the terminal where we launch the docker Opentelemetry-Collector container, we can see Pinot metric has been emitted and successfully reported/exported:

Metric #204
Descriptor:
     -> Name: realtimeBytesDropped
     -> Description:
     -> Unit: SECONDS
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> Component: Str(server)
     -> PinotMetricName: Str(pinot.server.realtimeBytesDropped)
StartTimestamp: 2025-08-26 21:28:05.576318 +0000 UTC
Timestamp: 2025-08-26 21:50:15.579809 +0000 UTC
Value: 0
Metric #205
Descriptor:
     -> Name: uncaughtGetExceptions
     -> Description:
     -> Unit: SECONDS
     -> DataType: Sum
     -> IsMonotonic: true
     -> AggregationTemporality: Cumulative
NumberDataPoints #0
Data point attributes:
     -> Component: Str(broker)
     -> PinotMetricName: Str(pinot.broker.uncaughtGetExceptions)
StartTimestamp: 2025-08-26 21:28:05.576318 +0000 UTC
Timestamp: 2025-08-26 21:50:15.579809 +0000 UTC
Value: 0

For example, we can see that pinot.server.realtimeBytesDropped has been parsed as

Name: realtimeBytesDropped
Attributes
    * Component: Str(server)
    * PinotMetricName: Str(pinot.server.realtimeBytesDropped)

[codecov-commenter]

Codecov Report

❌ Patch coverage is 41.82754% with 452 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.35%. Comparing base (af3dc6a) to head (fe6b33b).

Files with missing lines	Patch %	Lines
...lugin/metrics/opentelemetry/OpenTelemetryUtil.java	40.74%	126 Missing and 2 partials ⚠️
...cs/opentelemetry/OpenTelemetryMetricsRegistry.java	28.47%	104 Missing and 4 partials ⚠️
...ugin/metrics/opentelemetry/OpenTelemetryMeter.java	0.00%	21 Missing ⚠️
.../pinot/minion/taskfactory/TaskFactoryRegistry.java	0.00%	18 Missing ⚠️
.../metrics/compound/CompoundPinotMetricsFactory.java	0.00%	18 Missing ⚠️
.../metrics/compound/CompoundPinotMetricRegistry.java	0.00%	17 Missing ⚠️
...ugin/metrics/opentelemetry/OpenTelemetryTimer.java	0.00%	17 Missing ⚠️
...g/apache/pinot/common/metrics/AbstractMetrics.java	85.10%	12 Missing and 2 partials ⚠️
...ugin/metrics/compound/CompoundPinotMetricName.java	0.00%	12 Missing ⚠️
...ics/opentelemetry/OpenTelemetryMetricsFactory.java	47.82%	10 Missing and 2 partials ⚠️
... and 22 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #16665      +/-   ##
============================================
- Coverage     63.42%   63.35%   -0.07%     
- Complexity     1400     1424      +24     
============================================
  Files          3054     3067      +13     
  Lines        178766   179400     +634     
  Branches      27399    27450      +51     
============================================
+ Hits         113378   113666     +288     
- Misses        56656    57008     +352     
+ Partials       8732     8726       -6     

Flag	Coverage Δ
custom-integration1	?
integration	0.00% <ø> (-100.00%)	⬇️
integration1	?
integration2	0.00% <ø> (ø)
java-11	63.32% <41.82%> (-0.08%)	⬇️
java-21	63.31% <41.82%> (-0.09%)	⬇️
temurin	63.35% <41.82%> (-0.07%)	⬇️
unittests	63.35% <41.82%> (-0.07%)	⬇️
unittests1	56.53% <64.42%> (+0.05%)	⬆️
unittests2	33.42% <39.63%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[noob-se7en]

There must be some Opentelemetry Collector processor which can help achieve the same thing we are achieving here?

[mqliang]

There must be some Opentelemetry Collector processor which can help achieve the same thing we are achieving here?

The major concern is: pinot metric name is NOT Opentelemetry friendly (or more specifically, not multi-dimension friendly), meaning: whenever pinot core call the metric SPI to emit metric, it has concatenated all attributes/dimensions into metric name. The concatenating happens at every caller place, not inside plugin implementation.

So if using Opentelemetry Collector processor to collect the metric from JMX, it's still NOT multi-dimension metric. This make monitoring dashboard un-maintainable: we have to create a dashboard for each table, because table name has been concatenated into metric name.

This PR emit multi-dimension metric, where table name is a dimension (or "attribute" in Opentelemetry's terminology). It will make dashboard more maintainable: you will be bale to create one dashboard, then on your metric platform UI, there will be a dropdown list to let you select the table (because table name is now a dimension, you will be able to search metrics using filters etc).