#122 Feature/new copilot metrics---update to latest metrics API, repalce the old usage API #125

DevOps-zhuang · 2024-11-29T07:49:56Z

#122
This pull request includes various changes aimed at enhancing the metrics visualization and data validation for GitHub Copilot. The most important changes include updates to the README.md file, refactoring the API calls and data handling, adding a new data conversion utility, and implementing a metrics validator. Additionally, there are updates to the Vue components to improve the display and interaction with the metrics data.

Documentation Updates:

README.md: Updated image sources and restructured the key metrics section for better clarity and presentation.

API and Data Handling Improvements:

src/api/GitHubApi.ts: Refactored the API calls to handle Copilot metrics, including ensuring data structure consistency and converting metrics data for usage. [1] [2] [3] [4]
src/api/MetricsToUsageConverter.ts: Added a new utility to convert Copilot metrics to a structured format suitable for usage analysis.
src/api/MetricsValidator.ts: Implemented a validator class to check for continuous dates, validate code completions, and chat engaged users in the metrics data.

Frontend Enhancements:

src/components/ApiResponse.vue: Enhanced the component to include a button for checking metrics data quality and updated the display logic for metrics and seats data. [1] [2] [3]
src/components/BreakdownComponent.vue: Updated the component to include new breakdown charts and additional data columns for suggested prompts and acceptance rates by count and lines. [1] [2] [3] [4]

…ons bug for acceptance rate (by lines); remove unused variables in SeatsAnalysisViewer

… sources for top 5 accepted prompts

…te labels, improve clarity, and reorder cards for better focus on acceptance rates by count

… enhance error handling and improve code readability

…age paths for key metrics

…ceRatebycount

…nd images.

…/DevOps-zhuang/copilot-metrics-viewer into Feature/AddAcceptanceRatebycount

…the old one is called metrics

…usage route 2) add MetricsToUsageConverter class, to map the new returned metrics schedma to old metrics schema, so the caller don't need to update more. 3) add CopilotUsageChecker to check the data quality of the new fetched schema, and display it in api-response page.

…re, simplify the button layout in the ApiResponse component.

massimogentilini · 2024-12-05T09:32:10Z

Tested on my installation with my data, some feedbacks:

New breakdown: really useful, but in my scenario the colors of the pie is too similar
API analysis, good but from my point of view this error is quite normal because it's normal to do not have data during weekends or festivities. Very minor one indeed, maybe improving the message (do not over analyze them to consider non working days because it's a mess)
Some metrics are invalid! nonContinuousDates: [ "2024-11-10", "2024-11-26", "2024-11-30" ]
In the main page the first two graphs are larger than the other ones, not nice to see.

DevOps-zhuang · 2024-12-06T00:23:36Z

[like] Junqian Zhuang reacted to your message:

…

________________________________ From: Massimo Gentilini ***@***.***> Sent: Thursday, December 5, 2024 9:32:34 AM To: github-copilot-resources/copilot-metrics-viewer ***@***.***> Cc: Junqian Zhuang ***@***.***>; Author ***@***.***> Subject: Re: [github-copilot-resources/copilot-metrics-viewer] #122 Feature/new copilot metrics---update to latest metrics API, repalce the old usage API (PR #125) Tested on my installation with my data, some feedbacks: * New breakdown: really useful, but in my scenario the colors of the pie is too similar image.png (view on web)<https://github.com/user-attachments/assets/8fdf21f9-eb8f-42ee-9b0c-b78579f39806> * API analysis, good but from my point of view this error is quite normal because it's normal to do not have data during weekends or festivities. Very minor one indeed, maybe improving the message (do not over analyze them to consider non working days because it's a mess) Some metrics are invalid! nonContinuousDates: [ "2024-11-10", "2024-11-26", "2024-11-30" ] * In the main page the first two graphs are larger than the other ones, not nice to see. image.png (view on web)<https://github.com/user-attachments/assets/6482a638-5435-4a4c-9e8f-695e2522b0f4> — Reply to this email directly, view it on GitHub<#125 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AM4XDKAMYAANCDVP6HIFJF32EAMTFAVCNFSM6AAAAABSWRGQFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJZG42DSMJWGQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

DevOps-zhuang · 2024-12-06T01:33:21Z

Tested on my installation with my data, some feedbacks:

New breakdown: really useful, but in my scenario the colors of the pie is too similar

API analysis, good but from my point of view this error is quite normal because it's normal to do not have data during weekends or festivities. Very minor one indeed, maybe improving the message (do not over analyze them to consider non working days because it's a mess)
Some metrics are invalid! nonContinuousDates: [ "2024-11-10", "2024-11-26", "2024-11-30" ]

In the main page the first two graphs are larger than the other ones, not nice to see.

thanks, very sueful suggestion and feedback, thanks!
will consider the UI part. and for the API message part, since it is for reference, and it is not easy to consider it is weekend or holiday since holiday is different in differt countries/time difference. I will update the the message accordingly. the main reason is that I found the data is not always right, so I added for visibility and it takes double check. once the metric data is of high quality, will consider remvoe this kind of check :-)

the message will be like as below:
Some metrics are possible invalid! please double check! nonContinuousDates: [ "2024-11-10", "2024-11-26", "2024-11-30" ]

…der positions and container structure.

…al distinction

…alid metrics output

DevOps-zhuang · 2024-12-15T15:50:37Z

Tested on my installation with my data, some feedbacks:

New breakdown: really useful, but in my scenario the colors of the pie is too similar

API analysis, good but from my point of view this error is quite normal because it's normal to do not have data during weekends or festivities. Very minor one indeed, maybe improving the message (do not over analyze them to consider non working days because it's a mess)
Some metrics are invalid! nonContinuousDates: [ "2024-11-10", "2024-11-26", "2024-11-30" ]

In the main page the first two graphs are larger than the other ones, not nice to see.

Thanks for your comments, I just updated based on your suggestion, can you double check it? thanks.

the color of the pie for new breakdown is as below:

Copilot

Copilot reviewed 18 out of 28 changed files in this pull request and generated 1 comment.

Files not reviewed (10)

src/assets/enterprise_seats_response_sample.json: Language not supported
src/assets/organization_seats_response_sample.json: Language not supported
src/components/SeatsAnalysisViewer.vue: Evaluated as low risk
src/components/ApiResponse.vue: Evaluated as low risk
src/components/BreakdownComponent.vue: Evaluated as low risk
src/components/MainComponent.vue: Evaluated as low risk
src/components/MetricsViewer.vue: Evaluated as low risk
src/api/MetricsValidator.ts: Evaluated as low risk
src/model/Breakdown.ts: Evaluated as low risk
src/model/Copilot_Metrics.ts: Evaluated as low risk

Copilot · 2025-01-05T00:34:41Z

src/api/MetricsToUsageConverter.ts

+      const totalChatCopies = metric.copilot_ide_chat?.editors?.reduce((sum, editor) => 
+        sum + editor.models?.reduce((sum, model) => sum + model.total_chat_copy_events, 0), 0) || 0;
+
+      console.log(`Date: ${metric.date}`);


Console logs should be removed or replaced with proper logging mechanisms.

karpikpl · 2025-01-05T00:38:48Z

src/components/BreakdownComponent.vue

-    // Get the top 5 breakdowns by acceptance rate
-    const top5BreakdownsAcceptanceRate = breakdownList.value.slice(0, 5);
+    // for test, it seems there is an issue in data, so need to get the data in console
+    console.log('Breakdown List:', JSON.stringify(breakdownList.value, null, 2));


is this line still needed?

it is not needed now, the issue is fixed. in the past, it didn't refresh

karpikpl · 2025-01-05T01:26:44Z

is this an issue with data when Acceptance is higher than Suggestions?

Same with lines suggested/accepted

It would be better if we used valid sample data instead of ones that has those abnormalities.

When I checked API response I got

Some metrics maybe are invalid, pls double check! 1). invalidCodeCompletions:
{
        "date": "2024-11-18",
        "editor": "JetBrains",
        "language": "java"
},

but it doesn't say what's wrong, which is little confusing.

karpikpl · 2025-01-05T05:24:57Z

README.md

+  <img width="800" alt="image" src="./images/KeyMetrics.png">
+</p>
+
+1. **Acceptance Rate:** This metric represents the ratio of accepted numbers to the total numbers suggested by GitHub Copilot. This rate is an indicator of the relevance and usefulness of Copilot's suggestions.


Suggested change

1. **Acceptance Rate:** This metric represents the ratio of accepted numbers to the total numbers suggested by GitHub Copilot. This rate is an indicator of the relevance and usefulness of Copilot's suggestions.

1. **Acceptance Rate:** This metric represents the ratio of accepted lines and suggestions to the total suggested by GitHub Copilot. This rate is an indicator of the relevance and usefulness of Copilot's suggestions.

However, as any metric, should be used with caution as developers use Copilot in many different ways (research, confirm, verify etc. not always "inject").

karpikpl · 2025-01-05T05:25:19Z

README.md

@@ -15,45 +15,49 @@ https://github.com/github-copilot-resources/copilot-metrics-viewer/assets/332930
 ## Charts

 ## Key Metrics
-Here are the key metrics visualized in these charts:
-1. **Acceptance Rate:** This metric represents the ratio of accepted lines to the total lines suggested by GitHub Copilot. This rate is an indicator of the relevance and usefulness of Copilot's suggestions.
+Here are the key metrics visualized in these charts,will introduce them one by one as below


Suggested change

Here are the key metrics visualized in these charts,will introduce them one by one as below

>[!NOTE]

> Metrics details are described in detail in [GitHub API response schema](https://docs.github.com/en/rest/copilot/copilot-metrics?apiVersion=2022-11-28#get-copilot-metrics-for-an-organization)

Here are the key metrics visualized in these charts:

karpikpl · 2025-01-05T05:39:44Z

src/components/ApiResponse.vue

+      required: true
+    },
+    metrics: {
+      type: Array as () => CopilotMetrics[],


Suggested change

type: Array as () => CopilotMetrics[],

type: Array as () => Metrics[],

karpikpl · 2025-01-05T05:40:28Z

src/components/ApiResponse.vue

  </v-container>
 </template>

 <script lang="ts">
 import { defineComponent } from 'vue';
 import config from '../config';
+import { MetricsValidator } from '../api/MetricsValidator';
+import { CopilotMetrics } from '../model/Copilot_Metrics';


Suggested change

import { CopilotMetrics } from '../model/Copilot_Metrics';

import { CopilotMetrics } from '../model/Copilot_Metrics';

import { Metrics } from '@/model/Metrics';

karpikpl · 2025-01-05T05:48:46Z

@DevOps-zhuang - from what I can tell this also contains #116 and #115 which fixes #113 ?

DevOps-zhuang · 2025-01-05T05:58:18Z

yes,it is a latest version 获取 Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ 发件人: Piotr Karpala ***@***.***> 发送时间: Sunday, January 5, 2025 1:49:08 PM 收件人: github-copilot-resources/copilot-metrics-viewer ***@***.***> 抄送: Junqian Zhuang ***@***.***>; Mention ***@***.***> 主题: Re: [github-copilot-resources/copilot-metrics-viewer] #122 Feature/new copilot metrics---update to latest metrics API, repalce the old usage API (PR #125) @DevOps-zhuang<https://github.com/DevOps-zhuang> - from what I can tell this also contains #116<#116> and #115<#115> which fixes #113<#113> ? ― Reply to this email directly, view it on GitHub<#125 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AM4XDKHMO2VFNV2ZOMEL7LL2JDBVJAVCNFSM6AAAAABSWRGQFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZRGUYTCNRQHE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

karpikpl · 2025-01-06T04:18:44Z

src/components/ApiResponse.vue

+        this.message = 'All metrics are valid!';
+        this.isError = false;
+      } else {
+        this.message = 'Some metrics maybe are invalid, pls double check!\n';


Suggested change

this.message = 'Some metrics maybe are invalid, pls double check!\n';

this.message = 'Some metrics might be inconsistent, please double check the API response.\n';

karpikpl

Thanks for the API update! I've posted some code suggestions

karpikpl · 2025-01-07T03:33:57Z

src/components/BreakdownComponent.vue

+    // for test, it seems there is an issue in data, so need to get the data in console
+    console.log('Breakdown List:', JSON.stringify(breakdownList.value, null, 2));


Suggested change

// for test, it seems there is an issue in data, so need to get the data in console

console.log('Breakdown List:', JSON.stringify(breakdownList.value, null, 2));

karpikpl · 2025-01-11T07:44:11Z

this has been replaced by #138 which included all the code changes + tests.
Thanks @DevOps-zhuang !

DevOps-zhuang · 2025-01-11T07:45:55Z

[heart] Junqian Zhuang reacted to your message:

…

________________________________ From: Piotr Karpala ***@***.***> Sent: Saturday, January 11, 2025 7:44:34 AM To: github-copilot-resources/copilot-metrics-viewer ***@***.***> Cc: Junqian Zhuang ***@***.***>; Mention ***@***.***> Subject: Re: [github-copilot-resources/copilot-metrics-viewer] #122 Feature/new copilot metrics---update to latest metrics API, repalce the old usage API (PR #125) this has been replaced by #138<#138> which included all the code changes + tests. Thanks @DevOps-zhuang<https://github.com/DevOps-zhuang> ! — Reply to this email directly, view it on GitHub<#125 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AM4XDKH47Y3HFRO4PFU3LJL2KDDWFAVCNFSM6AAAAABSWRGQFWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOBVGEZTKMRQGI>. You are receiving this because you were mentioned.Message ID: ***@***.***>

…ilot-metrics Feature/new copilot metrics API - replaces #125

DevOps-zhuang added 12 commits November 19, 2024 12:31

Show the Acceptance Rate (by counts) and Acceptance Rate (by lines)

b2f7431

Refactor MetricsViewer to update acceptance rate labels and calculati…

1742d20

…ons bug for acceptance rate (by lines); remove unused variables in SeatsAnalysisViewer

Refactor BreakdownComponent to update acceptance rate labels and data…

1047c0a

… sources for top 5 accepted prompts

Refactor BreakdownComponent and MetricsViewer to update acceptance ra…

5292628

…te labels, improve clarity, and reorder cards for better focus on acceptance rates by count

Fix a hidden bug for data fetch. it takes me 3 days! MainComponent to…

40131d1

… enhance error handling and improve code readability

Update README.md to correct acceptance rate terminology and update im…

032e0d1

…age paths for key metrics

Merge branch 'github-copilot-resources:main' into Feature/AddAcceptan…

747c1e0

…ceRatebycount

update the readme to add the newly update metric report description a…

d391446

…nd images.

Merge branch 'Feature/AddAcceptanceRatebycount' of https://github.com…

f573fef

…/DevOps-zhuang/copilot-metrics-viewer into Feature/AddAcceptanceRatebycount

add new schema for new metrics, the new one is named copilotMetrics, …

0dd5f1b

…the old one is called metrics

Update the example response file name to reflect the new data structu…

4bb8962

…re, simplify the button layout in the ApiResponse component.

DevOps-zhuang mentioned this pull request Nov 29, 2024

Update API to latest version #122

Closed

rename to keep same file name format

bc6b28b

DevOps-zhuang added 3 commits December 15, 2024 23:30

Refactor MetricsViewer layout to improve readability by adjusting hea…

4adadae

…der positions and container structure.

Enhance pie chart color palette in BreakdownComponent for better visu…

8cee938

…al distinction

Improve error message clarity in ApiResponse component and format inv…

8fdabe9

…alid metrics output

karpikpl requested a review from Copilot January 5, 2025 00:33

Copilot AI reviewed Jan 5, 2025

View reviewed changes

karpikpl reviewed Jan 5, 2025

View reviewed changes

karpikpl linked an issue Jan 5, 2025 that may be closed by this pull request

Update API to latest version #122

Closed

karpikpl reviewed Jan 6, 2025

View reviewed changes

karpikpl reviewed Jan 7, 2025

View reviewed changes

This was referenced Jan 11, 2025

Use of /metrics #137

Closed

Feature/new copilot metrics API - replaces #125 #138

Merged

karpikpl closed this Jan 11, 2025

karpikpl added a commit that referenced this pull request Jan 13, 2025

Merge pull request #138 from github-copilot-resources/feature/new-cop…

a5cfc56

…ilot-metrics Feature/new copilot metrics API - replaces #125

	1. Acceptance Rate: This metric represents the ratio of accepted numbers to the total numbers suggested by GitHub Copilot. This rate is an indicator of the relevance and usefulness of Copilot's suggestions.
	1. Acceptance Rate: This metric represents the ratio of accepted lines and suggestions to the total suggested by GitHub Copilot. This rate is an indicator of the relevance and usefulness of Copilot's suggestions.
	However, as any metric, should be used with caution as developers use Copilot in many different ways (research, confirm, verify etc. not always "inject").

	type: Array as () => CopilotMetrics[],
	type: Array as () => Metrics[],

	import { CopilotMetrics } from '../model/Copilot_Metrics';
	import { CopilotMetrics } from '../model/Copilot_Metrics';
	import { Metrics } from '@/model/Metrics';

	this.message = 'Some metrics maybe are invalid, pls double check!\n';
	this.message = 'Some metrics might be inconsistent, please double check the API response.\n';

		// for test, it seems there is an issue in data, so need to get the data in console
		console.log('Breakdown List:', JSON.stringify(breakdownList.value, null, 2));

#122 Feature/new copilot metrics---update to latest metrics API, repalce the old usage API #125

#122 Feature/new copilot metrics---update to latest metrics API, repalce the old usage API #125

Uh oh!

Conversation

DevOps-zhuang commented Nov 29, 2024

Documentation Updates:

API and Data Handling Improvements:

Frontend Enhancements:

Uh oh!

massimogentilini commented Dec 5, 2024

Uh oh!

DevOps-zhuang commented Dec 6, 2024 via email

Uh oh!

DevOps-zhuang commented Dec 6, 2024

Uh oh!

DevOps-zhuang commented Dec 15, 2024

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 5, 2025

Choose a reason for hiding this comment

Uh oh!

karpikpl Jan 5, 2025

Choose a reason for hiding this comment

Uh oh!

DevOps-zhuang Jan 6, 2025

Choose a reason for hiding this comment

Uh oh!

karpikpl commented Jan 5, 2025

Uh oh!

karpikpl Jan 5, 2025

Choose a reason for hiding this comment

Uh oh!

karpikpl Jan 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karpikpl Jan 5, 2025

Choose a reason for hiding this comment

Uh oh!

karpikpl Jan 5, 2025

Choose a reason for hiding this comment

Uh oh!

karpikpl commented Jan 5, 2025

Uh oh!

DevOps-zhuang commented Jan 5, 2025 via email

Uh oh!

karpikpl Jan 6, 2025

Choose a reason for hiding this comment

Uh oh!

karpikpl left a comment

Choose a reason for hiding this comment

Uh oh!

karpikpl Jan 7, 2025

Choose a reason for hiding this comment

Uh oh!

karpikpl commented Jan 11, 2025

Uh oh!

DevOps-zhuang commented Jan 11, 2025 via email

Uh oh!

Uh oh!

karpikpl Jan 5, 2025 •

edited

Loading