Skip to content

feat(ai): support URL to base64 conversion in tool toModelOutput #8257

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
+152 −3
Open

feat(ai): support URL to base64 conversion in tool toModelOutput #8257

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
db368cf
feat(ai): support URL to base64 conversion in tool result outputs
xinyao27 Aug 23, 2025
43af0c8
Merge branch 'main' into main
xinyao27 Aug 26, 2025
5775c0b
Merge branch 'main' into main
gr2m Sep 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions packages/ai/src/prompt/convert-to-language-model-prompt.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1426,5 +1426,70 @@ describe('convertToLanguageModelMessage', () => {
}
`);
});

it('should convert URL in tool result content to base64', async () => {
const result = await convertToLanguageModelPrompt({
prompt: {
messages: [
{
role: 'tool',
content: [
{
type: 'tool-result',
toolName: 'screenshot',
toolCallId: 'call-123',
output: {
type: 'content',
value: [
{ type: 'text', text: 'Screenshot captured' },
{
type: 'media',
data: 'https://example.com/screenshot.png',
mediaType: 'image/png',
},
],
},
},
],
},
],
},
supportedUrls: {},
downloadImplementation: async ({ url }) => {
expect(url).toEqual(new URL('https://example.com/screenshot.png'));
return {
data: new Uint8Array([137, 80, 78, 71]), // PNG magic bytes
mediaType: 'image/png',
};
},
});

expect(result).toEqual([
{
role: 'tool',
content: [
{
type: 'tool-result',
toolCallId: 'call-123',
toolName: 'screenshot',
output: {
type: 'content',
value: [
{ type: 'text', text: 'Screenshot captured' },
{
type: 'media',
data: 'iVBORw==', // base64 of [137, 80, 78, 71]
mediaType: 'image/png',
},
],
},
providerOptions: undefined,
},
],
providerOptions: undefined,
},
]);
});

});
});
90 changes: 87 additions & 3 deletions packages/ai/src/prompt/convert-to-language-model-prompt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,10 @@ import {
LanguageModelV2Message,
LanguageModelV2Prompt,
LanguageModelV2TextPart,
LanguageModelV2ToolResultOutput,
} from '@ai-sdk/provider';
import {
convertToBase64,
DataContent,
FilePart,
ImagePart,
Expand Down Expand Up @@ -156,7 +158,10 @@ export function convertToLanguageModelMessage({
type: 'tool-result' as const,
toolCallId: part.toolCallId,
toolName: part.toolName,
output: part.output,
output: convertOutputToLanguageModelOutput(
part.output,
downloadedAssets,
),
providerOptions,
};
}
Expand All @@ -173,7 +178,10 @@ export function convertToLanguageModelMessage({
type: 'tool-result' as const,
toolCallId: part.toolCallId,
toolName: part.toolName,
output: part.output,
output: convertOutputToLanguageModelOutput(
part.output,
downloadedAssets,
),
providerOptions: part.providerOptions,
})),
providerOptions: message.providerOptions,
Expand Down Expand Up @@ -237,8 +245,49 @@ async function downloadAssets(
}),
}));

const toolUrls = messages
.filter(message => message.role === 'tool')
.map(message => message.content)
.flat()
.filter(item => item.type === 'tool-result')
.flatMap(item => {
if (item.output.type === 'content') {
const results = item.output.value;
return results
.map(result => {
if (result.type === 'media' && result.data && result.mediaType) {
const url =
typeof result.data === 'string' ? new URL(result.data) : null;
if (url instanceof URL) {
return { ...result, data: url };
}
}
return null;
})
.filter(url => url !== null);
}
return null;
})
.filter(
item =>
item &&
!isUrlSupported({
url: item.data.toString(),
mediaType: item.mediaType,
supportedUrls,
}),
)
.map(item => item!.data);

const allUrls = [...urls, ...toolUrls];

// download in parallel:
const downloadedFiles = await download(plannedDownloads);
const downloadedFiles = await Promise.all(
allUrls.map(async url => ({
url,
data: await downloadImplementation({ url }),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The variable downloadImplementation is undefined - it should be download to reference the function parameter.

View Details
📝 Patch Details 
diff --git a/packages/ai/src/prompt/convert-to-language-model-prompt.test.ts b/packages/ai/src/prompt/convert-to-language-model-prompt.test.ts
index d34044049..21707edb9 100644
--- a/packages/ai/src/prompt/convert-to-language-model-prompt.test.ts
+++ b/packages/ai/src/prompt/convert-to-language-model-prompt.test.ts
@@ -1455,12 +1455,13 @@ describe('convertToLanguageModelMessage', () => {
           ],
         },
         supportedUrls: {},
-        downloadImplementation: async ({ url }) => {
-          expect(url).toEqual(new URL('https://example.com/screenshot.png'));
-          return {
+        download: async (downloads) => {
+          expect(downloads).toHaveLength(1);
+          expect(downloads[0].url).toEqual(new URL('https://example.com/screenshot.png'));
+          return [{
             data: new Uint8Array([137, 80, 78, 71]), // PNG magic bytes
             mediaType: 'image/png',
-          };
+          }];
         },
       });
 
diff --git a/packages/ai/src/prompt/convert-to-language-model-prompt.ts b/packages/ai/src/prompt/convert-to-language-model-prompt.ts
index 13ddda99a..1f7b3a7e1 100644
--- a/packages/ai/src/prompt/convert-to-language-model-prompt.ts
+++ b/packages/ai/src/prompt/convert-to-language-model-prompt.ts
@@ -279,30 +279,35 @@ async function downloadAssets(
     )
     .map(item => item!.data);
 
+  // Extract URLs from plannedDownloads that need to be downloaded
+  const urls = plannedDownloads
+    .filter(download => !download.isUrlSupportedByModel)
+    .map(download => download.url);
+
   const allUrls = [...urls, ...toolUrls];
 
-  // download in parallel:
-  const downloadedFiles = await Promise.all(
-    allUrls.map(async url => ({
-      url,
-      data: await downloadImplementation({ url }),
-    })),
-  );
+  // download in parallel using the batch download function:
+  const downloadRequests = [
+    ...plannedDownloads.filter(download => !download.isUrlSupportedByModel),
+    ...toolUrls.map(url => ({ url, isUrlSupportedByModel: false }))
+  ];
+  
+  const downloadResults = await download(downloadRequests);
 
   return Object.fromEntries(
-    downloadedFiles
-      .filter(
-        (
-          downloadedFile,
-        ): downloadedFile is {
-          mediaType: string | undefined;
-          data: Uint8Array;
-        } => downloadedFile?.data != null,
-      )
-      .map(({ data, mediaType }, index) => [
-        plannedDownloads[index].url.toString(),
-        { data, mediaType },
-      ]),
+    downloadResults
+      .map((result, index) => {
+        if (result?.data != null) {
+          return [
+            downloadRequests[index].url.toString(),
+            { data: result.data, mediaType: result.mediaType },
+          ];
+        }
+        return null;
+      })
+      .filter((entry): entry is [string, { data: Uint8Array; mediaType: string | undefined }] => 
+        entry !== null
+      ),
   );
 }

Analysis

Undefined Variables in Download Assets Function

Issue Summary

The downloadAssets function in packages/ai/src/prompt/convert-to-language-model-prompt.ts contains multiple undefined variables that cause TypeScript compilation errors and would result in runtime failures.

Specific Problems

1. Undefined Variable: downloadImplementation (Line 288)

The code calls downloadImplementation({ url }) but this variable does not exist in scope. The function parameter is named download.

// Current (incorrect) code:
data: await downloadImplementation({ url }),

// Should be:
data: await download(plannedDownloads.filter(...)),

2. Undefined Variable: urls (Line 282)

The code references [...urls, ...toolUrls] but urls is never defined. This should be extracting URLs from the plannedDownloads array.

// Current (incorrect) code:
const allUrls = [...urls, ...toolUrls];

// Should extract URLs from plannedDownloads:
const urls = plannedDownloads
  .filter(download => !download.isUrlSupportedByModel)
  .map(download => download.url);

3. Function Signature Mismatch

The DownloadFunction type expects an array of download requests and returns an array of results, but the current code calls it with individual URL objects. This architectural mismatch shows the code expects batch downloads but implements individual calls.

Compilation Impact

These errors prevent the TypeScript build from completing:

packages/ai/src/prompt/convert-to-language-model-prompt.ts(282,23): error TS2552: Cannot find name 'urls'. Did you mean 'URL'?
packages/ai/src/prompt/convert-to-language-model-prompt.ts(288,19): error TS2552: Cannot find name 'downloadImplementation'. Did you mean 'DOMImplementation'?

Runtime Impact

If the TypeScript errors were somehow bypassed, this would cause:

  • ReferenceError: downloadImplementation is not defined when URLs need downloading
  • ReferenceError: urls is not defined when processing URL collections
  • Application crashes when users upload files or images with URLs

Root Cause

This appears to be incomplete refactoring where:

  1. A download parameter was renamed but call sites weren't updated
  2. URL extraction logic was removed but the usage wasn't updated
  3. The function signature was changed from individual to batch processing without updating the implementation

})),
);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The array index mapping is incorrect - plannedDownloads[index] references the wrong array when both regular URLs and tool URLs are processed.

View Details
📝 Patch Details 
diff --git a/packages/ai/src/prompt/convert-to-language-model-prompt.test.ts b/packages/ai/src/prompt/convert-to-language-model-prompt.test.ts
index d34044049..00c87dcf3 100644
--- a/packages/ai/src/prompt/convert-to-language-model-prompt.test.ts
+++ b/packages/ai/src/prompt/convert-to-language-model-prompt.test.ts
@@ -1455,15 +1455,17 @@ describe('convertToLanguageModelMessage', () => {
           ],
         },
         supportedUrls: {},
-        downloadImplementation: async ({ url }) => {
-          expect(url).toEqual(new URL('https://example.com/screenshot.png'));
-          return {
-            data: new Uint8Array([137, 80, 78, 71]), // PNG magic bytes
-            mediaType: 'image/png',
+        download: async (downloads) => {
+          return downloads.map(({ url }) => {
+            expect(url).toEqual(new URL('https://example.com/screenshot.png'));
+            return {
+              data: new Uint8Array([137, 80, 78, 71]), // PNG magic bytes
+              mediaType: 'image/png',
           };
+          });
+
         },
       });
-
       expect(result).toEqual([
         {
           role: 'tool',
diff --git a/packages/ai/src/prompt/convert-to-language-model-prompt.ts b/packages/ai/src/prompt/convert-to-language-model-prompt.ts
index 13ddda99a..bb5650a72 100644
--- a/packages/ai/src/prompt/convert-to-language-model-prompt.ts
+++ b/packages/ai/src/prompt/convert-to-language-model-prompt.ts
@@ -279,30 +279,27 @@ async function downloadAssets(
     )
     .map(item => item!.data);
 
-  const allUrls = [...urls, ...toolUrls];
+  const urls = plannedDownloads.map(pd => pd.url);
+  const allDownloads = [
+    ...plannedDownloads.map(pd => ({ url: pd.url, isUrlSupportedByModel: pd.isUrlSupportedByModel })),
+    ...toolUrls.map(url => ({ url, isUrlSupportedByModel: false }))
+  ];
 
   // download in parallel:
-  const downloadedFiles = await Promise.all(
-    allUrls.map(async url => ({
-      url,
-      data: await downloadImplementation({ url }),
-    })),
-  );
+  const downloadedFiles = await download(allDownloads);
 
   return Object.fromEntries(
     downloadedFiles
-      .filter(
-        (
-          downloadedFile,
-        ): downloadedFile is {
-          mediaType: string | undefined;
-          data: Uint8Array;
-        } => downloadedFile?.data != null,
-      )
-      .map(({ data, mediaType }, index) => [
-        plannedDownloads[index].url.toString(),
-        { data, mediaType },
-      ]),
+      .map((downloadedFile, index) => {
+        if (downloadedFile?.data != null) {
+          return [
+            allDownloads[index].url.toString(),
+            { data: downloadedFile.data, mediaType: downloadedFile.mediaType },
+          ];
+        }
+        return null;
+      })
+      .filter(entry => entry !== null) as [string, { data: Uint8Array; mediaType: string | undefined }][],
   );
 }

Analysis

Array Index Mapping Bug in URL Download Function

Technical Issue Description

The downloadAssets function in packages/ai/src/prompt/convert-to-language-model-prompt.ts contains multiple critical bugs that prevent compilation and would cause runtime errors:

Primary Issues Identified

  1. Undefined Variable urls (Line 282): The code references urls in const allUrls = [...urls, ...toolUrls]; but this variable is never defined. The TypeScript compiler fails with error TS2552: Cannot find name 'urls'.

  2. Undefined Function downloadImplementation (Line 288): The code calls downloadImplementation({ url }) but this function doesn't exist. The correct parameter is download passed to the function.

  3. Array Index Mismatch (Line 303): As reported, the mapping logic uses plannedDownloads[index].url.toString() but downloadedFiles corresponds to allUrls, not plannedDownloads. When tool URLs are present, this creates an index mismatch that would cause runtime errors.

  4. Type Predicate Error: The filter function expects objects with url property but receives objects with different structure.

Code Analysis

The downloadAssets function:

  • Extracts URLs from user messages into plannedDownloads (objects with url property)
  • Extracts URLs from tool messages into toolUrls (just URL objects)
  • Attempts to combine them in allUrls = [...urls, ...toolUrls] but urls is undefined
  • Downloads files parallel to allUrls array
  • Maps results back using plannedDownloads[index] which is incorrect

Impact

  • Compilation Failure: TypeScript compilation fails with multiple errors
  • Runtime Errors: If somehow compiled, would throw TypeError when accessing undefined array elements
  • Data Corruption: Incorrect URL mapping would return wrong file associations

Root Cause

The code appears to be in an incomplete or partially refactored state where:

  1. urls should be plannedDownloads.map(p => p.url) or similar extraction
  2. downloadImplementation should be the download parameter
  3. The final mapping should use allUrls[index] instead of plannedDownloads[index]

This represents a fundamental logical error in array correspondence that would prevent the feature from working correctly when both user message URLs and tool message URLs are present.


return Object.fromEntries(
downloadedFiles
Expand Down Expand Up @@ -347,3 +396,38 @@ function convertPartToLanguageModelPart(
}
}
}

function convertOutputToLanguageModelOutput(
output: LanguageModelV2ToolResultOutput,
downloadedAssets: Record<
string,
{ mediaType: string | undefined; data: Uint8Array }
>,
): LanguageModelV2ToolResultOutput {
if (output.type === 'content') {
return {
type: 'content' as const,
value: output.value.map(result => {
if (result.type === 'media' && result.data && result.mediaType) {
const { data: convertedData, mediaType: convertedMediaType } =
convertToLanguageModelV2DataContent(result.data);
if (convertedData instanceof URL) {
const downloadedFile = downloadedAssets[convertedData.toString()];
if (downloadedFile) {
return {
type: 'media' as const,
data: convertToBase64(downloadedFile.data),
mediaType:
downloadedFile.mediaType ??
convertedMediaType ??
result.mediaType,
};
}
}
}
return result;
Comment on lines +411 to +428
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to avoid nesting, can you please implement an early return pattern?

}),
};
}
return output;
}