Skip to content

Conversation

mwln
Copy link

@mwln mwln commented Aug 26, 2025

Background

The validateUIMessages utility function does not cover the potential input
and rawInput states for output-error scenarios properly. In the case that a
tool fails due to a schematic error in input, the validation of that
ToolUIPart of { state: "output-error" } fails its schematic check against
the tool's inputSchema, creating an invalid messages array despite the array
being technically valid based on it's type. see ToolUIPart

The main reason this happens is due to:

  1. the schemas defined in the validation aren't directly covering all possible scenarios
    that the type defines
  2. the output-error scenario still forces a schematic check on the input, even if the
    input does not exist. In these scenarios, the part returned actually uses the rawInput
    property to portray that failed input.

The tests for this function do not cover this particular state, which allows the
test-suite to pass gracefully despite this potential state occurring in real-world
scenarios.

Manual Verification

  • test the scenario where unhandled, yet valid, input scenarios fail the test suite
  • ensure the schema is only checked when it makes sense for output-error's.

Tasks

  • add additional test case for scenario
  • A patch changeset for relevant packages has been added
  • Formatting issues have been fixed (run pnpm prettier-fix in the project root)
  • I have reviewed this pull request (self-review)

@mwln mwln changed the title improve ui message validation scenarios and documentation feat(ai): improve ui message validation scenarios and documentation Aug 26, 2025
@mwln mwln force-pushed the improve-validate-ui-messages branch from 7a0af02 to 3dd3aa5 Compare August 26, 2025 18:39
Copy link
Collaborator

@lgrammel lgrammel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert all the nullish changes and the docs changes

add missing schema properties and set input to optional in output error
add test case for no schema check on output-error scenario

remove output-error if check
@mwln mwln force-pushed the improve-validate-ui-messages branch from 05489db to 3b454dc Compare August 27, 2025 14:07
@mwln mwln changed the title feat(ai): improve ui message validation scenarios and documentation feat(ai): improve ui message validation in output-error scenarios Aug 27, 2025
@mwln mwln changed the title feat(ai): improve ui message validation in output-error scenarios feat(ai): gracefully handle ui message validation in output-error scenarios Aug 27, 2025
@mwln mwln requested a review from lgrammel August 27, 2025 14:12
@mwln
Copy link
Author

mwln commented Aug 27, 2025

revert all the nullish changes and the docs changes

@lgrammel fair enough.

Just a note though - since these schema's are only used in this function, and the validation function is only exported as a nice-to-have, there's not much reason to block the support of a wider range of falsy/nullish scenarios on set properties.

I'd argue the utility of the function is just to validate the bare minimum requirements for a UIMessage to display it's necessary information, which is in most cases just the type and then the messages core information given its state.

The omitted/undefined/nullish/falsy scenarios for properties kind of becomes unimportant at that point, which then questions the utility of this function if its going to throw errors unless all the aforementioned property scenarios are coerced to undefined.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants