Skip to content

expand gitlab_runner supported metrics #21261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
+263 −0
Open

expand gitlab_runner supported metrics #21261

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8cc71d8
add up to date metric list
eliebleton-manomano Sep 4, 2025
0a8b83f
add exemple metrics to the spec
eliebleton-manomano Sep 5, 2025
01f7982
regenerate example config file
eliebleton-manomano Sep 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions gitlab_runner/metadata.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,22 @@
metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name,curated_metric
gitlab_runner.acceptable_job_queuing_duration_exceeded_total,count,,request,,Counts how often jobs exceed the configured queuing time threshold,0,gitlab_runner,acceptable job queuing duration exceeded total,
gitlab_runner.api_request_duration_seconds_bucket,gauge,,request,second,Latency histogram of API requests made by GitLab Runner,0,gitlab_runner,api request duration time,
gitlab_runner.api_request_duration_seconds_count,gauge,,request,second,Latency histogram of API requests made by GitLab Runner,0,gitlab_runner,api request duration time count,
gitlab_runner.api_request_duration_seconds_sum,gauge,,request,second,Latency histogram of API requests made by GitLab Runner,0,gitlab_runner,api request duration time sum,
gitlab_runner.api_request_statuses_total,count,,request,,"The total number of api requests, partitioned by runner, system_id, endpoint and status.",0,gitlab_runner,api request statuses total,
gitlab_runner.autoscaling_machine_creation_duration_seconds_bucket,gauge,,request,second,Histogram of machine creation time.,0,gitlab_runner,autoscaling machine creation duration time,
gitlab_runner.autoscaling_machine_creation_duration_seconds_count,gauge,,request,second,Histogram of machine creation time.,0,gitlab_runner,autoscaling machine creation duration time count,
gitlab_runner.autoscaling_machine_creation_duration_seconds_sum,gauge,,request,second,Histogram of machine creation time.,0,gitlab_runner,autoscaling machine creation duration time sum,
gitlab_runner.autoscaling_machine_failed_creation_duration_seconds_bucket,gauge,,request,second,Histogram of machine failed creation timings,0,gitlab_runner,autoscaling machine failed creation duration time,
gitlab_runner.autoscaling_machine_failed_creation_duration_seconds_count,gauge,,request,second,Histogram of machine failed creation timings,0,gitlab_runner,autoscaling machine failed creation duration time count,
gitlab_runner.autoscaling_machine_failed_creation_duration_seconds_sum,gauge,,request,second,Histogram of machine failed creation timings,0,gitlab_runner,autoscaling machine failed creation duration time sum,
gitlab_runner.autoscaling_machine_removal_duration_seconds_bucket,gauge,,request,second,Histogram of machine removal time.,0,gitlab_runner,autoscaling machine removal duration time,
gitlab_runner.autoscaling_machine_removal_duration_seconds_count,gauge,,request,second,Histogram of machine removal time.,0,gitlab_runner,autoscaling machine removal duration time count,
gitlab_runner.autoscaling_machine_removal_duration_seconds_sum,gauge,,request,second,Histogram of machine removal time.,0,gitlab_runner,autoscaling machine removal duration time sum,
gitlab_runner.autoscaling_machine_states,gauge,,,,The current number of machines per state in this provider.,0,gitlab_runner,autoscaling machine states,
gitlab_runner.autoscaling_machine_stopping_duration_seconds_bucket,gauge,,request,second,Histogram of machine stopping time.,0,gitlab_runner,autoscaling machine stopping duration time,
gitlab_runner.autoscaling_machine_stopping_duration_seconds_count,gauge,,request,second,Histogram of machine stopping time.,0,gitlab_runner,autoscaling machine stopping duration time count,
gitlab_runner.autoscaling_machine_stopping_duration_seconds_sum,gauge,,request,second,Histogram of machine stopping time.,0,gitlab_runner,autoscaling machine stopping duration time sum,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update! Adding the metrics to metadata.csv does not ensures the metrics are going to be emitted. These metrics need to be allowlisted (see here). You need to add the metrics that you want to be allowed in the configuration spec.

Once you add them CI will fail telling you to update the config example. Should guide you to get it fixed.

Could you also include a changelog for this? Let me know if you need help with any of these steps.

Adding new metrics in a OpenMetricsCheck is safe since we will only emit the metrics if they are found in the Prometheus endpoint. So there is no need to split per version.

gitlab_runner.ci_docker_machines_provider_machine_creation_duration_seconds_bucket,gauge,,request,second,A histogram of Docker machine creation time. Applies to GitLab Runner < 1.11.0,0,gitlab_runner,docker machine creation time,
gitlab_runner.ci_docker_machines_provider_machine_creation_duration_seconds_count,gauge,,request,second,The count of Docker machine creation time. Applies to GitLab Runner < 1.11.0,0,gitlab_runner,count docker machine creation time,
gitlab_runner.ci_docker_machines_provider_machine_creation_duration_seconds_sum,gauge,,request,second,The sum of Docker machine creation time. Applies to GitLab Runner < 1.11.0,0,gitlab_runner,sum docker machine creation time,
Expand All @@ -10,6 +28,39 @@ gitlab_runner.ci_ssh_docker_machines_provider_machine_creation_duration_seconds_
gitlab_runner.ci_ssh_docker_machines_provider_machine_creation_duration_seconds_count,gauge,,request,second,The count of SSH Docker machine creation time. Applies to GitLab Runner < 1.11.0,0,gitlab_runner,count ssh docker machine creation time,
gitlab_runner.ci_ssh_docker_machines_provider_machine_creation_duration_seconds_sum,gauge,,request,second,The sum of SSH Docker machine creation time. Applies to GitLab Runner < 1.11.0,0,gitlab_runner,sum ssh docker machine creation time,
gitlab_runner.ci_ssh_docker_machines_provider_machine_states,gauge,,request,second,The current number of SSH machines per state in this ssh provider. Applies to GitLab Runner < 1.11.0,0,gitlab_runner,total ssh docker machines per state,
gitlab_runner.concurrent,gauge,,,,The current value of concurrent setting,0,gitlab_runner,concurrent,
gitlab_runner.configuration_loaded_total,count,,request,,Total number of times the configuration file was loaded by Runner process,0,gitlab_runner,configuration loaded total,
gitlab_runner.configuration_loading_error_total,count,,request,,Total number of times the configuration file was not loaded by Runner process due to errors,0,gitlab_runner,configuration loading error total,
gitlab_runner.configuration_saved_total,count,,request,,Total number of times the configuration file was saved by Runner process,0,gitlab_runner,configuration saved total,
gitlab_runner.configuration_saving_error_total,count,,request,,Total number of times the configuration file was not saved by Runner process due to errors,0,gitlab_runner,configuration saving error total,
gitlab_runner.errors_total,count,,request,,The number of caught errors.,0,gitlab_runner,errors total,
gitlab_runner.failed_jobs_total,count,,request,,Total number of failed jobs,0,gitlab_runner,failed jobs total,
gitlab_runner.fleeting_provisioner_instance_creation_time_seconds_bucket,gauge,,request,second,Histogram of instance creation times,0,gitlab_runner,fleeting provisioner instance creation time time,
gitlab_runner.fleeting_provisioner_instance_creation_time_seconds_count,gauge,,request,second,Histogram of instance creation times,0,gitlab_runner,fleeting provisioner instance creation time time count,
gitlab_runner.fleeting_provisioner_instance_creation_time_seconds_sum,gauge,,request,second,Histogram of instance creation times,0,gitlab_runner,fleeting provisioner instance creation time time sum,
gitlab_runner.fleeting_provisioner_instance_deletion_time_seconds_bucket,gauge,,request,second,Histogram of instance deletion times,0,gitlab_runner,fleeting provisioner instance deletion time time,
gitlab_runner.fleeting_provisioner_instance_deletion_time_seconds_count,gauge,,request,second,Histogram of instance deletion times,0,gitlab_runner,fleeting provisioner instance deletion time time count,
gitlab_runner.fleeting_provisioner_instance_deletion_time_seconds_sum,gauge,,request,second,Histogram of instance deletion times,0,gitlab_runner,fleeting provisioner instance deletion time time sum,
gitlab_runner.fleeting_provisioner_instance_is_running_time_seconds_bucket,gauge,,request,second,Histogram of duration for the instance to ender running state,0,gitlab_runner,fleeting provisioner instance is running time time,
gitlab_runner.fleeting_provisioner_instance_is_running_time_seconds_count,gauge,,request,second,Histogram of duration for the instance to ender running state,0,gitlab_runner,fleeting provisioner instance is running time time count,
gitlab_runner.fleeting_provisioner_instance_is_running_time_seconds_sum,gauge,,request,second,Histogram of duration for the instance to ender running state,0,gitlab_runner,fleeting provisioner instance is running time time sum,
gitlab_runner.fleeting_provisioner_instance_life_duration_seconds_bucket,gauge,,request,second,Histogram of instance's life durations,0,gitlab_runner,fleeting provisioner instance life duration time,
gitlab_runner.fleeting_provisioner_instance_life_duration_seconds_count,gauge,,request,second,Histogram of instance's life durations,0,gitlab_runner,fleeting provisioner instance life duration time count,
gitlab_runner.fleeting_provisioner_instance_life_duration_seconds_sum,gauge,,request,second,Histogram of instance's life durations,0,gitlab_runner,fleeting provisioner instance life duration time sum,
gitlab_runner.fleeting_provisioner_instance_operations_total,count,,request,,Counter of total operations performed on instances,0,gitlab_runner,fleeting provisioner instance operations total,
gitlab_runner.fleeting_provisioner_instances,gauge,,,,Current number of instances in different states,0,gitlab_runner,fleeting provisioner instances,
gitlab_runner.fleeting_provisioner_internal_operations_total,count,,request,,Counter of total internal operations handled by provisioner,0,gitlab_runner,fleeting provisioner internal operations total,
gitlab_runner.fleeting_provisioner_max_instances,gauge,,,,Maximum number of instances allowed for fleeting to maintain,0,gitlab_runner,fleeting provisioner max instances,
gitlab_runner.fleeting_provisioner_missed_updates_total,count,,request,,Counter of total times an instance reached the 'missed_update' state,0,gitlab_runner,fleeting provisioner missed updates total,
gitlab_runner.fleeting_taskscaler_desired_instances,gauge,,,,Current number of desired instances (negative means number of instances that should be requested for deletion),0,gitlab_runner,fleeting taskscaler desired instances,
gitlab_runner.fleeting_taskscaler_max_tasks_per_instance,gauge,,,,Maximum number of tasks executed on a single instance,0,gitlab_runner,fleeting taskscaler max tasks per instance,
gitlab_runner.fleeting_taskscaler_max_use_count_per_instance,gauge,,,,Maximum number of times the instance can be used,0,gitlab_runner,fleeting taskscaler max use count per instance,
gitlab_runner.fleeting_taskscaler_scale_operations_total,count,,request,,Total number of tasks scaling operations,0,gitlab_runner,fleeting taskscaler scale operations total,
gitlab_runner.fleeting_taskscaler_task_instance_readiness_time_seconds_bucket,gauge,,request,second,Histogram of task instance readiness times,0,gitlab_runner,fleeting taskscaler task instance readiness time time,
gitlab_runner.fleeting_taskscaler_task_instance_readiness_time_seconds_count,gauge,,request,second,Histogram of task instance readiness times,0,gitlab_runner,fleeting taskscaler task instance readiness time time count,
gitlab_runner.fleeting_taskscaler_task_instance_readiness_time_seconds_sum,gauge,,request,second,Histogram of task instance readiness times,0,gitlab_runner,fleeting taskscaler task instance readiness time time sum,
gitlab_runner.fleeting_taskscaler_task_operations_total,count,,request,,Counter of total operations performed on instances,0,gitlab_runner,fleeting taskscaler task operations total,
gitlab_runner.fleeting_taskscaler_tasks,gauge,,,,Current number of tasks in different states,0,gitlab_runner,fleeting taskscaler tasks,
gitlab_runner.gitlab_runner_autoscaling_machine_creation_duration_seconds,gauge,,request,second,A histogram of Docker machine creation time. Applies to GitLab Runner 1.11.0+,0,gitlab_runner,auto docker machine creation time,
gitlab_runner.gitlab_runner_autoscaling_machine_states,gauge,,request,second,The current number of machines per state in this provider. Applies to GitLab Runner 1.11.0+,0,gitlab_runner,auto total docker machines per state,
gitlab_runner.gitlab_runner_errors_total,count,,request,second,The number of caught errors. Applies to GitLab Runner 1.11.0+,0,gitlab_runner,errors,
Expand All @@ -19,7 +70,10 @@ gitlab_runner.gitlab_runner_version_info,gauge,,request,,A metric with a constan
gitlab_runner.go_gc_duration_seconds,gauge,,request,second,A summary of the GC invocation durations,0,gitlab_runner,gc duration,
gitlab_runner.go_gc_duration_seconds_count,gauge,,request,second,The count of the GC invocation durations,0,gitlab_runner,count gc duration,
gitlab_runner.go_gc_duration_seconds_sum,gauge,,request,second,The sum of the GC invocation durations,0,gitlab_runner,sum gc duration,
gitlab_runner.go_gc_gogc_percent,gauge,,,,"Heap size target percentage configured by the user, otherwise 100. This value is set by the GOGC environment variable, and the runtime/debug.SetGCPercent function. Sourced from /gc/gogc:percent",0,gitlab_runner,go gc gogc percent,
gitlab_runner.go_gc_gomemlimit_bytes,gauge,,byte,,"Go runtime memory limit configured by the user, otherwise math.MaxInt64. This value is set by the GOMEMLIMIT environment variable, and the runtime/debug.SetMemoryLimit function. Sourced from /gc/gomemlimit:bytes",0,gitlab_runner,go gc gomemlimit bytes,
gitlab_runner.go_goroutines,gauge,,request,second,The number of goroutines that currently exist,0,gitlab_runner,goroutines,
gitlab_runner.go_info,gauge,,,,Information about the Go environment.,0,gitlab_runner,go info,
gitlab_runner.go_memstats_alloc_bytes,gauge,,byte,,The number of bytes allocated and still in use,0,gitlab_runner,bytes allocated in use,
gitlab_runner.go_memstats_alloc_bytes_total,count,,byte,,The total number of bytes allocated,0,gitlab_runner,bytes allocated,
gitlab_runner.go_memstats_buck_hash_sys_bytes,gauge,,byte,,The number of bytes used by the profiling bucket hash table,0,gitlab_runner,bytes profiling bucket,
Expand All @@ -29,6 +83,7 @@ gitlab_runner.go_memstats_heap_alloc_bytes,gauge,,byte,,The number of heap bytes
gitlab_runner.go_memstats_heap_idle_bytes,gauge,,byte,,The number of heap bytes waiting to be used,0,gitlab_runner,heap bytes unused,
gitlab_runner.go_memstats_heap_inuse_bytes,gauge,,byte,,The number of heap bytes that are in use,0,gitlab_runner,heap bytes in use,
gitlab_runner.go_memstats_heap_objects,gauge,,request,,The number of allocated objects,0,gitlab_runner,allocated objects,
gitlab_runner.go_memstats_heap_released_bytes,gauge,,byte,,Number of heap bytes released to OS. Equals to /memory/classes/heap/released:bytes.,0,gitlab_runner,go memstats heap released bytes,
gitlab_runner.go_memstats_heap_released_bytes_total,count,,byte,,The total number of heap bytes released to OS,0,gitlab_runner,heap bytes released,
gitlab_runner.go_memstats_heap_sys_bytes,gauge,,byte,,The number of heap bytes obtained from system,0,gitlab_runner,heap bytes obtained,
gitlab_runner.go_memstats_last_gc_time_seconds,gauge,,request,,The number of seconds since 1970 of last garbage collection,0,gitlab_runner,epoch since last gc,
Expand All @@ -43,9 +98,35 @@ gitlab_runner.go_memstats_other_sys_bytes,gauge,,byte,,The number of bytes used
gitlab_runner.go_memstats_stack_inuse_bytes,gauge,,byte,,The number of bytes in use by the stack allocator,0,gitlab_runner,bytes stack allocator,
gitlab_runner.go_memstats_stack_sys_bytes,gauge,,byte,,The number of bytes obtained from system for stack allocator,0,gitlab_runner,bytes stack allocator from sys,
gitlab_runner.go_memstats_sys_bytes,gauge,,byte,,The number of bytes obtained by system. Sum of all system allocations,0,gitlab_runner,sum system allocations,
gitlab_runner.go_sched_gomaxprocs_threads,gauge,,,,"The current runtime.GOMAXPROCS setting, or the number of operating system threads that can execute user-level Go code simultaneously. Sourced from /sched/gomaxprocs:threads",0,gitlab_runner,go sched gomaxprocs threads,
gitlab_runner.go_threads,gauge,,,,Number of OS threads created.,0,gitlab_runner,go threads,
gitlab_runner.job_duration_seconds_bucket,gauge,,request,second,Histogram of job durations,0,gitlab_runner,job duration time,
gitlab_runner.job_duration_seconds_count,gauge,,request,second,Histogram of job durations,0,gitlab_runner,job duration time count,
gitlab_runner.job_duration_seconds_sum,gauge,,request,second,Histogram of job durations,0,gitlab_runner,job duration time sum,
gitlab_runner.job_queue_duration_seconds_bucket,gauge,,request,second,A histogram representing job queue duration.,0,gitlab_runner,job queue duration time,
gitlab_runner.job_queue_duration_seconds_count,gauge,,request,second,A histogram representing job queue duration.,0,gitlab_runner,job queue duration time count,
gitlab_runner.job_queue_duration_seconds_sum,gauge,,request,second,A histogram representing job queue duration.,0,gitlab_runner,job queue duration time sum,
gitlab_runner.jobs,gauge,,,,The current number of running builds.,0,gitlab_runner,jobs,
gitlab_runner.jobs_total,count,,request,,Total number of handled jobs,0,gitlab_runner,jobs total,
gitlab_runner.limit,gauge,,,,The current value of concurrent setting,0,gitlab_runner,limit,
gitlab_runner.process_cpu_seconds_total,count,,request,second,The total user and system CPU time spent in seconds,0,gitlab_runner,user and system cpu time,
gitlab_runner.process_max_fds,gauge,,request,,The maximum number of open file descriptors,0,gitlab_runner,max fds,
gitlab_runner.process_network_receive_bytes_total,count,,byte,,Number of bytes received by the process over the network.,0,gitlab_runner,process network receive bytes total,
gitlab_runner.process_network_transmit_bytes_total,count,,byte,,Number of bytes sent by the process over the network.,0,gitlab_runner,process network transmit bytes total,
gitlab_runner.process_open_fds,gauge,,request,,The number of open file descriptors,0,gitlab_runner,open fds,
gitlab_runner.process_resident_memory_bytes,gauge,,byte,,The resident memory size in bytes,0,gitlab_runner,rss bytes,
gitlab_runner.process_start_time_seconds,gauge,,request,second,The start time of the process since unix epoch in seconds,0,gitlab_runner,epoch time since start,
gitlab_runner.process_virtual_memory_bytes,gauge,,byte,,The virtual memory size in bytes,0,gitlab_runner,virtual memory bytes,
gitlab_runner.process_virtual_memory_max_bytes,gauge,,byte,,Maximum amount of virtual memory available in bytes.,0,gitlab_runner,process virtual memory max bytes,
gitlab_runner.request_concurrency,gauge,,,,The current number of concurrent requests for a new job,0,gitlab_runner,request concurrency,
gitlab_runner.request_concurrency_adaptive_limit,gauge,,,,Computed adaptive request concurrency limit,0,gitlab_runner,request concurrency adaptive limit,
gitlab_runner.request_concurrency_exceeded_total,count,,request,,Count of excess requests above the configured request_concurrency limit,0,gitlab_runner,request concurrency exceeded total,
gitlab_runner.request_concurrency_hard_limit,gauge,,,,Configured request_concurrency limit,0,gitlab_runner,request concurrency hard limit,
gitlab_runner.request_concurrency_used_limit,gauge,,,,Used request concurrency limit,0,gitlab_runner,request concurrency used limit,
gitlab_runner.version_info,gauge,,,,A metric with a constant '1' value labeled by different build stats fields.,0,gitlab_runner,version info,
gitlab_runner.worker_feed_failures_total,count,,request,,Total number of times that runner worker feeding have failed,0,gitlab_runner,worker feed failures total,
gitlab_runner.worker_feeds_total,count,,request,,Total number of times that runner worker is fed to the main loop,0,gitlab_runner,worker feeds total,
gitlab_runner.worker_health_check_failures_total,count,,request,,Total number of runner worker health check failures,0,gitlab_runner,worker health check failures total,
gitlab_runner.worker_processing_failures_total,count,,request,,Total number of failures when processing runner worker,0,gitlab_runner,worker processing failures total,
gitlab_runner.worker_slot_operations_total,count,,request,,Total number of runner workers slot operations (starting and stopping slots),0,gitlab_runner,worker slot operations total,
gitlab_runner.worker_slots_number,gauge,,,,Current number of runner worker slots,0,gitlab_runner,worker slots number,
Loading