Add a MCP tool that helps review alt text in context of surrounding text #6820

kendallgassner · 2025-09-09T21:47:01Z

New

This PR introduces the review_alt_text tool to Primer MCP. The tool utilizes MCP sampling to assess whether provided alt text is meaningful and contextually relevant.

Alt text is often written by users or AI in a way that is generic (e.g., alt="Image") or disconnected from the surrounding content of a web page. The review_alt_text tool is designed to evaluate alt text, advocate for improvements, and guide users toward creating more descriptive and useful suggestions.

Note:
This tool is not intended to replace or prevent users from adding alt text. Its purpose is to review and provide feedback, helping users and AI craft more meaningful alt text for accessibility and context. Ideally this could be used by products like Copilot Coding Agent, or Copilot Code review.

Additional note: I could see this one day living in a different MCP. Perhaps an accessibility MCP or GitHub's MCP to target a broader audience. This is not tied to Primer but I think this MCP is a safe space for testing review_alt_text

Rollout strategy

None; if selected, include a brief description as to why

Testing & Reviewing

This can be tested by pulling down this branch and installing the Primer MCP locally on vs-code.

Merge checklist

Added/updated tests
Added/updated documentation
Added/updated previews (Storybook)
Changes are SSR compatible
Tested in Chrome
Tested in Firefox
Tested in Safari
Tested in Edge
(GitHub staff only) Integration tests pass at github/github (Learn more about how to run integration tests)

changeset-bot · 2025-09-09T21:47:04Z

🦋 Changeset detected

Latest commit: f98c701

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@primer/mcp	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2025-09-09T21:49:48Z

size-limit report 📦

Path	Size
packages/react/dist/browser.esm.js	88.64 KB (0%)
packages/react/dist/browser.umd.js	88.73 KB (0%)

kendallgassner · 2025-09-09T21:53:34Z

Vs-code output:

Backgrouund

In the packages/react/src/CircleBadge/CircleBadge.test.tsx file I removed the alt text from line 36.. I then asked copilot to fix the accessibility issues. Copilot generated the following response after running this Primer MCP tool with the following input.

Input:

{
  "surroundingText": "This is a test for the CircleBadge component that verifies it preserves child class names. The image is used as a child element within the CircleBadge to test that CSS classes are maintained.",
  "alt": "primer",
  "image": "primer.jpg"
}

Response :

Fixed the accessibility issue by adding a descriptive [alt](https://redesigned-giggle-975g5gq574wfxwvx.github.dev/) attribute to the image in the test.

Change made:

Added [alt="Test image for CircleBadge component"](https://redesigned-giggle-975g5gq574wfxwvx.github.dev/) to the inline <img> element
Why this fixes the accessibility issue:

Images without alt text fail accessibility checks because screen readers can't describe the image content
The alt text "Test image for CircleBadge component" clearly describes the purpose and context of the image within the test
This follows accessibility guidelines by providing meaningful alternative text for the image
The test will now pass accessibility validation and Testing Library's getByRole('img') will work correctly with proper alt text.

This example shows that the on it's own the AI did not create a great alt text. Using the Primer MCP it evaluated it's alt text and self corrected.

janmaarten-a11y · 2025-09-09T22:16:42Z

This does what a lot of LLM-powered tools are incapable of.

This came from a discussion with @benbalter about using GH Models to automatically add alt text to images in Issues—there are (to me) four levels of important context within alt text. Most tools struggle with the first or second level.

Can accurately describe the literal visuals (just the right amount, not too much or too little)

Can accurately describe the feeling, tone, and mood

Can accurately describe the meaning of the image in the context in which it’s placed

Can accurately describe the internal context of what’s shown: the patterns/components/pages/flows based on the org’s branding and design system

This has some real potential to clear the third level. And given that this is an MCP server for Primer and all the docs and knowledge therein, possibly even the 4th.

In other words: 👀

kendallgassner · 2025-09-09T22:20:34Z

packages/mcp/src/server.ts

+          },
+        },
+      ],
+      sampling: {temperature: 0.4},


temperature -> Controls how “random” or “creative” the model’s output is. Low temperature is more predictable/conservative while a high temp is more creative/varied. I chose .4 (almost mid range) to allow some flexibility in how the LLM reviews alt text.

Copilot

Pull Request Overview

This PR introduces an experimental review_alt_text tool to the Primer MCP server that evaluates the quality and accessibility compliance of alt text for images. The tool uses MCP sampling to assess whether alt text is meaningful and contextually relevant based on surrounding content.

Key changes:

Adds a new accessibility-focused tool for reviewing alt text quality
Implements MCP sampling integration for LLM-based evaluation
Includes proper documentation and experimental status disclaimers

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
packages/mcp/src/server.ts	Implements the new review_alt_text tool with MCP sampling functionality
.changeset/strong-lions-tan.md	Documents the addition of the review_alt_text tool for release notes

packages/mcp/src/server.ts

Copilot · 2025-09-10T16:38:25Z

packages/mcp/src/server.ts

+          role: 'user',
+          content: {
+            type: 'text',
+            text: `Does this alt text: '${alt}' meet accessibility guidelines and describe the ${image} accurately in context of this surrounding text: '${surroundingText}'?\n\n`,


The prompt template directly interpolates the image parameter which could be either a File object or URL string. This will result in '[object File]' being displayed when a File is passed, making the prompt unclear. Consider extracting a meaningful identifier from the image parameter before interpolation.

…in as image imput

kendallgassner · 2025-09-10T17:46:26Z

packages/mcp/src/server.ts

+          text: response.content.type === 'text' ? response.content.text : 'Unable to generate summary',
+        },
+      ],
+      altTextEvaluation: response.content.type === 'text' ? response.content.text : 'Unable to generate summary',


@joshblack I updated the nextSteps field after your approval to include: DO NOT run this tool repeatedly on the same image - evaluations may vary slightly with each run.

edsummersnc · 2025-09-10T19:36:15Z

@kendallgassner this is clever! Rock on.

kendallgassner · 2025-09-10T19:41:19Z

Thanks everyone! I am going to merge this on Monday as I am out of office Thursday and Friday and will want to monitor the tool's performance.

Add a MCP tool that helps review alt text in context of surrounding text

bd54558

kendallgassner deployed to github-pages September 9, 2025 21:58 — with GitHub Actions View deployment

github-actions bot temporarily deployed to storybook-preview-6820 September 9, 2025 21:58 Inactive

kendallgassner commented Sep 9, 2025

View reviewed changes

add changeset

4ecc67f

kendallgassner deployed to github-pages September 9, 2025 22:35 — with GitHub Actions View deployment

github-actions bot temporarily deployed to storybook-preview-6820 September 9, 2025 22:35 Inactive

add details on how the review_alt_text tool should be used

101b43c

kendallgassner marked this pull request as ready for review September 10, 2025 16:37

Copilot AI review requested due to automatic review settings September 10, 2025 16:37

kendallgassner requested a review from a team as a code owner September 10, 2025 16:37

kendallgassner requested review from pksjce, jonrohan and joshblack September 10, 2025 16:37

Copilot AI reviewed Sep 10, 2025

View reviewed changes

joshblack approved these changes Sep 10, 2025

View reviewed changes

kendallgassner deployed to github-pages September 10, 2025 16:44 — with GitHub Actions View deployment

github-actions bot temporarily deployed to storybook-preview-6820 September 10, 2025 16:45 Inactive

clean up spelling

42fe3f0

kendallgassner deployed to github-pages September 10, 2025 17:08 — with GitHub Actions View deployment

github-actions bot temporarily deployed to storybook-preview-6820 September 10, 2025 17:09 Inactive

Update next steps to avoid looping and allow file paths to be passed …

f98c701

…in as image imput

kendallgassner commented Sep 10, 2025

View reviewed changes

kendallgassner deployed to github-pages September 10, 2025 17:49 — with GitHub Actions View deployment

github-actions bot deployed to storybook-preview-6820 September 10, 2025 17:49 View deployment

jonrohan approved these changes Sep 10, 2025

View reviewed changes

kendallgassner added this pull request to the merge queue Sep 15, 2025

Merged via the queue into main with commit 3e47602 Sep 15, 2025
43 checks passed

kendallgassner deleted the kendallg/review_alt_text branch September 15, 2025 16:07

primer bot mentioned this pull request Sep 15, 2025

Release tracking (rc) #6842

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add a MCP tool that helps review alt text in context of surrounding text #6820

Add a MCP tool that helps review alt text in context of surrounding text #6820

kendallgassner commented Sep 9, 2025 •

edited

Loading

Uh oh!

changeset-bot bot commented Sep 9, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Sep 9, 2025 •

edited

Loading

Uh oh!

kendallgassner commented Sep 9, 2025

Uh oh!

janmaarten-a11y commented Sep 9, 2025

Uh oh!

kendallgassner Sep 9, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Sep 10, 2025

Uh oh!

kendallgassner Sep 10, 2025

Uh oh!

edsummersnc commented Sep 10, 2025

Uh oh!

kendallgassner commented Sep 10, 2025

Uh oh!

Uh oh!

Uh oh!

Add a MCP tool that helps review alt text in context of surrounding text #6820

Add a MCP tool that helps review alt text in context of surrounding text #6820

Conversation

kendallgassner commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New

Rollout strategy

Testing & Reviewing

Merge checklist

Uh oh!

changeset-bot bot commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions bot commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

size-limit report 📦

Uh oh!

kendallgassner commented Sep 9, 2025

Backgrouund

Uh oh!

janmaarten-a11y commented Sep 9, 2025

Uh oh!

kendallgassner Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

kendallgassner Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

edsummersnc commented Sep 10, 2025

Uh oh!

kendallgassner commented Sep 10, 2025

Uh oh!

Uh oh!

Uh oh!

kendallgassner commented Sep 9, 2025 •

edited

Loading

changeset-bot bot commented Sep 9, 2025 •

edited

Loading

github-actions bot commented Sep 9, 2025 •

edited

Loading

kendallgassner Sep 9, 2025 •

edited

Loading