response_format error after updating to => 9.5.0

[EdCharbeneau]

Description

I wasn't getting any errors prior to 9.5.0 and greater.

After updating my project to the latest version, I started getting errors.

Here is the call that is failing.

https://github.com/EdCharbeneau/NaturalLanguageGrid2/blob/6c75f6445c2e46c4cc1210e909c45427ec5256e3/TelerikAIExtensions/NaturalLanguage/NaturalLanguageGridService.cs#L84

If this isn't a bug, some advice would be appreciated. Thanks!

UPDATE

It appears that the schema for response_format by GetResponseAsync may be incomplete.
#6656 (comment)

Parameter: response_format

Invalid schema for response_format 'NaturalLanguageGridState_1': In context=('properties', 'gridState', 'properties', 'groupDescriptors', 'type', '0', 'items', 'properties', 'aggregateFunctions', 'type', '0'), array schema missing items.
   at Azure.AI.OpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)
   at Azure.AI.OpenAI.Chat.AzureChatClient.CompleteChatAsync(BinaryContent content, RequestOptions options)
   at OpenAI.Chat.ChatClient.CompleteChatAsync(IEnumerable`1 messages, ChatCompletionOptions options, CancellationToken cancellationToken)
   at Microsoft.Extensions.AI.OpenAIChatClient.GetResponseAsync(IEnumerable`1 messages, ChatOptions options, CancellationToken cancellationToken)
   at Microsoft.Extensions.AI.ChatClientStructuredOutputExtensions.GetResponseAsync[T](IChatClient chatClient, IEnumerable`1 messages, JsonSerializerOptions serializerOptions, ChatOptions options, Nullable`1 useJsonSchemaResponseFormat, CancellationToken cancellationToken)
   at TelerikAIExtensions.NaturalLanguage.NaturalLanguageGridService.<>c__DisplayClass8_0`1.<<ProcessGridRequest>g__TryProcessingGridState|3>d.MoveNext() in C:\Users\echarben\source\repos\EdCharbeneau\NaturalLanguageGrid2\TelerikAIExtensions\NaturalLanguage\NaturalLanguageGridService.cs:line 84
--- End of stack trace from previous location ---
   at TelerikAIExtensions.NaturalLanguage.NaturalLanguageGridService.ProcessGridRequest[T](String query, GridState`1 state) in C:\Users\echarben\source\repos\EdCharbeneau\NaturalLanguageGrid2\TelerikAIExtensions\NaturalLanguage\NaturalLanguageGridService.cs:line 43
   at TelerikAIExtensions.Tests.Tests.CanSort.CanSortColumns() in C:\Users\echarben\source\repos\EdCharbeneau\NaturalLanguageGrid2\NaturalLanguageGrid.Tests\Tests\CanSort.razor:line 24
--- End of stack trace from previous location ---

Reproduction Steps

Update from 9.1.0 to 9.5.0

Expected behavior

Structured output is returned and serialized

Actual behavior

Error occurs

Regression?

No response

Known Workarounds

No response

Configuration

https://github.com/EdCharbeneau/NaturalLanguageGrid2/blob/vNext/TelerikAIExtensions/NaturalLanguage/NaturalLanguageGridService.cs#L84

Other information

No response

[stephentoub]

@EdCharbeneau, do you still hit this on newer versions, e.g. 9.7.1 instead of 9.5?

[EdCharbeneau]

@EdCharbeneau, do you still hit this on newer versions, e.g. 9.7.1 instead of 9.5?

Yes, 9.5 and greater. I tried 9.6 and 9.7.1

[stephentoub]

@eiriktsarpalis, can you take a look?

[stephentoub]

Btw, until Eirik gets a chance to take a look, @EdCharbeneau, the main change here that happened for the stable 9.5 release was that the GetResponseAsync<T> method changed its default for the useJsonSchemaResponseFormat parameter, defaulting to true instead of false. If you set it back to false, I expect you'll get back to the state you were previously in. That's not actually using the service's/model's structured output capabilities, and instead is just putting the schema into the prompt with a message asking the LLM to try to output according to that schema. Something about the generated schema when actually used with structured output is making Azure OpenAI unhappy, and I don't currently know if that's an issue with the service or an issue with the generated schema.

[eiriktsarpalis]

Would it be possible to share the schema that is being generated in the failing call? It should be possible to extract it by setting a breakpoint at this location:

extensions/src/Libraries/Microsoft.Extensions.AI/ChatCompletion/ChatClientStructuredOutputExtensions.cs

Line 193 in 8f0f5a7

ChatMessage? promptAugmentation = null;

[EdCharbeneau]

Would it be possible to share the schema that is being generated in the failing call? It should be possible to extract it by setting a breakpoint at this location:

extensions/src/Libraries/Microsoft.Extensions.AI/ChatCompletion/ChatClientStructuredOutputExtensions.cs

Line 193 in 8f0f5a7

ChatMessage? promptAugmentation = null;

This is it.

https://github.com/EdCharbeneau/NaturalLanguageGrid2/blob/6c75f6445c2e46c4cc1210e909c45427ec5256e3/TelerikAIExtensions/NaturalLanguage/NaturalLanguageGridService.jsonSchema.cs#L1-L122

It's possible I'm hitting a limitation of the structured output feature: supported-schemas-and-limitations

[eiriktsarpalis]

Which limitation specifically?

[EdCharbeneau]

Which limitation specifically?

That's what I'm trying to figure out :)

I've done some debugging and logged the full request. It's quite large and easier to view in a JSON viewer. I'm currently cross checking it with the limitations, I'll report back.

{
    "messages": [
        {
            "role": "system",
            "content": "# You are helping to edit a JSON object that represents a Data Grid User Interface component\u0027s meta data.\r\n# This JSON object conforms to the following schema: {\r\n  \u0022$schema\u0022: \u0022http://json-schema.org/draft-07/schema#\u0022,\r\n  \u0022type\u0022: \u0022object\u0022,\r\n  \u0022properties\u0022: {\r\n    \u0022gridState\u0022: {\r\n      \u0022type\u0022: \u0022object\u0022,\r\n      \u0022properties\u0022: {\r\n        \u0022GroupDescriptors\u0022: {\r\n          \u0022type\u0022: \u0022array\u0022,\r\n          \u0022items\u0022: {\r\n            \u0022type\u0022: \u0022object\u0022,\r\n            \u0022properties\u0022: {\r\n              \u0022DisplayContent\u0022: { \u0022type\u0022: \u0022string\u0022 },\r\n              \u0022AggregateFunctions\u0022: { \u0022type\u0022: \u0022array\u0022, \u0022items\u0022: { \u0022type\u0022: \u0022string\u0022 } },\r\n              \u0022Member\u0022: { \u0022type\u0022: \u0022string\u0022 },\r\n              \u0022SortDirection\u0022: { \u0022type\u0022: \u0022integer\u0022 }\r\n            },\r\n            \u0022required\u0022: [\u0022DisplayContent\u0022, \u0022AggregateFunctions\u0022, \u0022Member\u0022, \u0022SortDirection\u0022],\r\n            \u0022additionalProperties\u0022: false\r\n          }\r\n        },\r\n        \u0022CollapsedGroups\u0022: {\r\n          \u0022type\u0022: \u0022array\u0022,\r\n          \u0022items\u0022: { \u0022type\u0022: \u0022string\u0022 }\r\n        },\r\n        \u0022ColumnStates\u0022: {\r\n          \u0022type\u0022: \u0022array\u0022,\r\n          \u0022items\u0022: {\r\n            \u0022type\u0022: \u0022object\u0022,\r\n            \u0022properties\u0022: {\r\n              \u0022Index\u0022: { \u0022type\u0022: \u0022integer\u0022 },\r\n              \u0022Width\u0022: { \u0022type\u0022: [\u0022number\u0022, \u0022null\u0022] },\r\n              \u0022Visible\u0022: { \u0022type\u0022: [\u0022boolean\u0022, \u0022null\u0022] },\r\n              \u0022Locked\u0022: { \u0022type\u0022: \u0022boolean\u0022 },\r\n              \u0022Id\u0022: { \u0022type\u0022: [\u0022string\u0022, \u0022null\u0022] },\r\n              \u0022Field\u0022: { \u0022type\u0022: \u0022string\u0022 }\r\n            },\r\n            \u0022required\u0022: [\u0022Index\u0022, \u0022Width\u0022, \u0022Visible\u0022, \u0022Locked\u0022, \u0022Id\u0022, \u0022Field\u0022],\r\n            \u0022additionalProperties\u0022: false\r\n          }\r\n        },\r\n        \u0022ExpandedItems\u0022: {\r\n          \u0022type\u0022: \u0022array\u0022,\r\n          \u0022items\u0022: { \u0022type\u0022: \u0022string\u0022 }\r\n        },\r\n        \u0022FilterDescriptors\u0022: {\r\n          \u0022type\u0022: \u0022array\u0022,\r\n          \u0022items\u0022: {\r\n            \u0022type\u0022: \u0022object\u0022,\r\n            \u0022properties\u0022: {\r\n              \u0022LogicalOperator\u0022: { \u0022type\u0022: \u0022integer\u0022 },\r\n              \u0022FilterDescriptors\u0022: {\r\n                \u0022type\u0022: \u0022array\u0022,\r\n                \u0022items\u0022: {\r\n                  \u0022type\u0022: \u0022object\u0022,\r\n                  \u0022properties\u0022: {\r\n                    \u0022Member\u0022: { \u0022type\u0022: \u0022string\u0022 },\r\n                    \u0022Operator\u0022: { \u0022type\u0022: \u0022integer\u0022 },\r\n                    \u0022Value\u0022: { \u0022type\u0022: [\u0022number\u0022, \u0022string\u0022, \u0022null\u0022, \u0022boolean\u0022] }\r\n                  },\r\n                  \u0022required\u0022: [\u0022Member\u0022, \u0022Operator\u0022, \u0022Value\u0022],\r\n                  \u0022additionalProperties\u0022: false\r\n                }\r\n              }\r\n            },\r\n            \u0022required\u0022: [\u0022LogicalOperator\u0022, \u0022FilterDescriptors\u0022],\r\n            \u0022additionalProperties\u0022: false\r\n          }\r\n        },\r\n        \u0022SortDescriptors\u0022: {\r\n          \u0022type\u0022: \u0022array\u0022,\r\n          \u0022items\u0022: {\r\n            \u0022type\u0022: \u0022object\u0022,\r\n            \u0022properties\u0022: {\r\n              \u0022Member\u0022: { \u0022type\u0022: \u0022string\u0022 },\r\n              \u0022SortDirection\u0022: { \u0022type\u0022: \u0022integer\u0022 }\r\n            },\r\n            \u0022required\u0022: [\u0022Member\u0022, \u0022SortDirection\u0022],\r\n            \u0022additionalProperties\u0022: false\r\n          }\r\n        },\r\n        \u0022SearchFilter\u0022: { \u0022type\u0022: [\u0022string\u0022, \u0022null\u0022] },\r\n        \u0022Page\u0022: { \u0022type\u0022: \u0022integer\u0022 },\r\n        \u0022Skip\u0022: { \u0022type\u0022: \u0022integer\u0022 },\r\n        \u0022SelectedItems\u0022: {\r\n          \u0022type\u0022: \u0022array\u0022,\r\n          \u0022items\u0022: { \u0022type\u0022: \u0022string\u0022 }\r\n        },\r\n        \u0022OriginalEditItem\u0022: { \u0022type\u0022: [\u0022string\u0022, \u0022null\u0022] },\r\n        \u0022EditItem\u0022: { \u0022type\u0022: [\u0022string\u0022, \u0022null\u0022] },\r\n        \u0022EditField\u0022: { \u0022type\u0022: [\u0022string\u0022, \u0022null\u0022] },\r\n        \u0022InsertedItem\u0022: { \u0022type\u0022: [\u0022string\u0022, \u0022null\u0022] },\r\n        \u0022TableWidth\u0022: { \u0022type\u0022: [\u0022number\u0022, \u0022null\u0022] }\r\n      },\r\n      \u0022required\u0022: [\r\n        \u0022GroupDescriptors\u0022, \u0022CollapsedGroups\u0022, \u0022ColumnStates\u0022, \u0022ExpandedItems\u0022, \u0022FilterDescriptors\u0022, \u0022SortDescriptors\u0022, \u0022Page\u0022, \u0022Skip\u0022, \u0022SelectedItems\u0022, \u0022OriginalEditItem\u0022, \u0022EditItem\u0022, \u0022EditField\u0022, \u0022InsertedItem\u0022, \u0022TableWidth\u0022\r\n      ],\r\n      \u0022additionalProperties\u0022: false\r\n    },\r\n    \u0022columnOrdering\u0022: {\r\n      \u0022type\u0022: \u0022object\u0022,\r\n      \u0022properties\u0022: {\r\n        \u0022columns\u0022: {\r\n          \u0022type\u0022: \u0022array\u0022,\r\n          \u0022items\u0022: { \u0022type\u0022: \u0022string\u0022 }\r\n        }\r\n      },\r\n      \u0022required\u0022: [\u0022columns\u0022]\r\n    }\r\n  },\r\n  \u0022required\u0022: [\u0022gridState\u0022, \u0022columnOrdering\u0022]\r\n}\r\n# Users can request assistance with configuring:\r\n1. Filtering\r\n2. Sorting\r\n3. Grouping\r\n4. Paging\r\n5. Hide and Show columns\r\n6. Column Order | Ordering | Move\r\n\r\n# Columns in the \u0022columnStates\u0022 array\r\n\r\n- Columns are found in the \u0022columnStates\u0022 array.\r\n- The Column name is the \u0022field\u0022 property value.\r\n\r\n# In the JSON schema, the \u0022sortDescriptors\u0022 array contains sorting element \u0022SortDescriptor\u0022\r\n- \u0022sortDirection\u0022 is set to 0 for ascending\r\n- \u0022sortDirection\u0022 is set to 1 for decending\r\n\r\n# Steps for Ordering, Reordering, Moving\r\n\r\n- DO NOT MODIFY THE \u0022field\u0022 in \u0022columnStates\u0022 even if you think it is incorrect.\r\n- Using the ColumnOrdering array, or create a new ColumnOrdering:\r\n\r\n\u003Cexample\u003E\r\n\r\n**Input:**\r\n\u0060\u0060\u0060\r\n\u0022ColumnOrdering\u0022:\r\n[\r\n    \u0022Column1\u0022,\r\n    \u0022Column2\u0022,\r\n    \u0022Column3\u0022,\r\n    \u0022Column4\u0022,\r\n    \u0022Column5\u0022\r\n]\r\n\u0060\u0060\u0060\r\n**Query**\r\n\r\nMove Column3 to the start.\r\n\r\n**Output:**\r\n\u0060\u0060\u0060\r\n\u0022ColumnOrdering\u0022:\r\n[\r\n    \u0022Column3\u0022,\r\n    \u0022Column1\u0022,\r\n    \u0022Column2\u0022,\r\n    \u0022Column4\u0022,\r\n    \u0022Column5\u0022\r\n]\r\n\u0060\u0060\u0060\r\n\r\n\u003C/example\u003E\r\n\r\n\r\n\r\n# LogicalOperator\r\n\r\n- Where GridState contains the element \u0022LogicalOperator\u0022, the element\u0027s value indicates a boolean logical operation where 0 is \u0022AND\u0022, or 1 is \u0022OR\u0022.\r\n\r\n# Where GridState contains the element \u0022FilterDescriptors\u0022, the element contains an array of values that indicates a grid filtering operation. The Member property is the Grid Column. The Value property indicates the filter\u0027s value.\u0022\r\n# Where a \u0022filterDescriptors\u0022 contains an \u0022operator\u0022 element. The operator numeric value represents the corresponding filter operations:\r\n** Filter Operators Key:Value **\r\n    0: Less Than\r\n    1: Less Than Or Equals\r\n    2: Equals\r\n    3: Not Equals\r\n    4: Greater Than Or Equals\r\n    5: Greater Than\r\n    6: Starts With\r\n    7: Ends With\r\n    8: Contains\r\n    10: Not Contains\r\n    11: Is Null\r\n    12: Is Not Null\r\n    13: Is Empty\r\n    14: Is Not Empty\r\n    15: Has No Value\r\n    16: Has Value\r\n\r\n# Hide and Show columns\r\n\r\n- Columns are hidden or shown by the \u0022ColumnStates\u0022 array.\r\n- If a column is hidden, the \u0022isVisible\u0022 value is set to \u0060false\u0060\r\n- In all other states the \u0022isVisible\u0022 value is \u0060null\u0060\r\n- When hiding and showing columns, only change the \u0022isVisible\u0022 value for the column names specified by the user, all other \u0022isVisible\u0022 values should remain unchanged."
        },
        {
            "role": "user",
            "content": "**User Request**: Update the given the Current GridState with my request. show page 5\r\n**Current GridState as JSON**: {\u0022gridState\u0022:{\u0022groupDescriptors\u0022:[],\u0022collapsedGroups\u0022:[],\u0022columnStates\u0022:[{\u0022index\u0022:0,\u0022width\u0022:null,\u0022visible\u0022:null,\u0022locked\u0022:false,\u0022id\u0022:null,\u0022field\u0022:\u0022CustomerId\u0022},{\u0022index\u0022:1,\u0022width\u0022:null,\u0022visible\u0022:null,\u0022locked\u0022:false,\u0022id\u0022:null,\u0022field\u0022:\u0022CompanyName\u0022},{\u0022index\u0022:2,\u0022width\u0022:null,\u0022visible\u0022:null,\u0022locked\u0022:false,\u0022id\u0022:null,\u0022field\u0022:\u0022Address\u0022},{\u0022index\u0022:3,\u0022width\u0022:null,\u0022visible\u0022:null,\u0022locked\u0022:false,\u0022id\u0022:null,\u0022field\u0022:\u0022City\u0022},{\u0022index\u0022:4,\u0022width\u0022:null,\u0022visible\u0022:null,\u0022locked\u0022:false,\u0022id\u0022:null,\u0022field\u0022:\u0022Country\u0022},{\u0022index\u0022:5,\u0022width\u0022:null,\u0022visible\u0022:null,\u0022locked\u0022:false,\u0022id\u0022:null,\u0022field\u0022:\u0022Number\u0022}],\u0022expandedItems\u0022:[],\u0022filterDescriptors\u0022:[],\u0022sortDescriptors\u0022:[],\u0022searchFilter\u0022:null,\u0022page\u0022:1,\u0022skip\u0022:0,\u0022selectedItems\u0022:[],\u0022originalEditItem\u0022:null,\u0022editItem\u0022:null,\u0022editField\u0022:null,\u0022insertedItem\u0022:null,\u0022tableWidth\u0022:null},\u0022columnOrdering\u0022:{\u0022columns\u0022:[\u0022CustomerId\u0022,\u0022CompanyName\u0022,\u0022Address\u0022,\u0022City\u0022,\u0022Country\u0022,\u0022Number\u0022]}}\r\n**Current Column Order from left to right**: \u0022ColumnOrdering\u0022:ColumnOrdering { Columns = System.String[] }"
        }
    ],
    "model": "gpt-4o-mini",
    "response_format": {
        "type": "json_schema",
        "json_schema": {
            "name": "NaturalLanguageGridState_1",
            "schema": {
                "$schema": "https://json-schema.org/draft/2020-12/schema",
                "type": "object",
                "properties": {
                    "gridState": {
                        "type": "object",
                        "properties": {
                            "groupDescriptors": {
                                "type": [
                                    "array",
                                    "null"
                                ],
                                "items": {
                                    "type": "object",
                                    "properties": {
                                        "displayContent": {},
                                        "aggregateFunctions": {
                                            "type": [
                                                "array",
                                                "null"
                                            ]
                                        },
                                        "member": {
                                            "type": [
                                                "string",
                                                "null"
                                            ]
                                        },
                                        "sortDirection": {
                                            "type": "string",
                                            "enum": [
                                                "Ascending",
                                                "Descending"
                                            ]
                                        }
                                    },
                                    "additionalProperties": false,
                                    "required": [
                                        "displayContent",
                                        "aggregateFunctions",
                                        "member",
                                        "sortDirection"
                                    ]
                                }
                            },
                            "collapsedGroups": {
                                "type": [
                                    "array",
                                    "null"
                                ],
                                "items": {
                                    "type": "integer"
                                }
                            },
                            "columnStates": {
                                "type": [
                                    "array",
                                    "null"
                                ],
                                "items": {
                                    "type": "object",
                                    "properties": {
                                        "index": {
                                            "type": "integer"
                                        },
                                        "width": {
                                            "type": [
                                                "string",
                                                "null"
                                            ]
                                        },
                                        "visible": {
                                            "type": [
                                                "boolean",
                                                "null"
                                            ]
                                        },
                                        "locked": {
                                            "type": "boolean"
                                        },
                                        "id": {
                                            "type": [
                                                "string",
                                                "null"
                                            ]
                                        },
                                        "field": {
                                            "type": [
                                                "string",
                                                "null"
                                            ]
                                        }
                                    },
                                    "additionalProperties": false,
                                    "required": [
                                        "index",
                                        "width",
                                        "visible",
                                        "locked",
                                        "id",
                                        "field"
                                    ]
                                }
                            },
                            "expandedItems": {
                                "type": [
                                    "array",
                                    "null"
                                ],
                                "items": {
                                    "type": "object",
                                    "properties": {
                                        "customerId": {
                                            "type": "string",
                                            "description": "maxLength: 5"
                                        },
                                        "companyName": {
                                            "type": "string",
                                            "description": "maxLength: 40"
                                        },
                                        "contactName": {
                                            "type": "string",
                                            "description": "maxLength: 30"
                                        },
                                        "contactTitle": {
                                            "type": "string",
                                            "description": "maxLength: 30"
                                        },
                                        "address": {
                                            "type": "string",
                                            "description": "maxLength: 60"
                                        },
                                        "city": {
                                            "type": "string",
                                            "description": "maxLength: 15"
                                        },
                                        "region": {
                                            "type": "string",
                                            "description": "maxLength: 15"
                                        },
                                        "postalCode": {
                                            "type": "string",
                                            "description": "maxLength: 10"
                                        },
                                        "country": {
                                            "type": "string",
                                            "description": "maxLength: 15"
                                        },
                                        "phone": {
                                            "type": "string",
                                            "description": "maxLength: 24"
                                        },
                                        "fax": {
                                            "type": "string",
                                            "description": "maxLength: 24"
                                        },
                                        "bool": {
                                            "type": [
                                                "boolean",
                                                "null"
                                            ]
                                        },
                                        "number": {
                                            "type": [
                                                "integer",
                                                "null"
                                            ]
                                        }
                                    },
                                    "additionalProperties": false,
                                    "required": [
                                        "customerId",
                                        "companyName",
                                        "contactName",
                                        "contactTitle",
                                        "address",
                                        "city",
                                        "region",
                                        "postalCode",
                                        "country",
                                        "phone",
                                        "fax",
                                        "bool",
                                        "number"
                                    ]
                                }
                            },
                            "filterDescriptors": {
                                "type": [
                                    "array",
                                    "null"
                                ]
                            },
                            "sortDescriptors": {
                                "type": [
                                    "array",
                                    "null"
                                ],
                                "items": {
                                    "type": "object",
                                    "properties": {
                                        "member": {
                                            "type": [
                                                "string",
                                                "null"
                                            ]
                                        },
                                        "sortDirection": {
                                            "type": "string",
                                            "enum": [
                                                "Ascending",
                                                "Descending"
                                            ]
                                        }
                                    },
                                    "additionalProperties": false,
                                    "required": [
                                        "member",
                                        "sortDirection"
                                    ]
                                }
                            },
                            "searchFilter": {},
                            "page": {
                                "type": [
                                    "integer",
                                    "null"
                                ]
                            },
                            "skip": {
                                "type": [
                                    "integer",
                                    "null"
                                ]
                            },
                            "selectedItems": {
                                "type": [
                                    "array",
                                    "null"
                                ],
                                "items": {
                                    "$ref": "#/properties/gridState/properties/expandedItems/items"
                                }
                            },
                            "originalEditItem": {
                                "type": [
                                    "object",
                                    "null"
                                ],
                                "properties": {
                                    "customerId": {
                                        "type": "string",
                                        "description": "maxLength: 5"
                                    },
                                    "companyName": {
                                        "type": "string",
                                        "description": "maxLength: 40"
                                    },
                                    "contactName": {
                                        "type": "string",
                                        "description": "maxLength: 30"
                                    },
                                    "contactTitle": {
                                        "type": "string",
                                        "description": "maxLength: 30"
                                    },
                                    "address": {
                                        "type": "string",
                                        "description": "maxLength: 60"
                                    },
                                    "city": {
                                        "type": "string",
                                        "description": "maxLength: 15"
                                    },
                                    "region": {
                                        "type": "string",
                                        "description": "maxLength: 15"
                                    },
                                    "postalCode": {
                                        "type": "string",
                                        "description": "maxLength: 10"
                                    },
                                    "country": {
                                        "type": "string",
                                        "description": "maxLength: 15"
                                    },
                                    "phone": {
                                        "type": "string",
                                        "description": "maxLength: 24"
                                    },
                                    "fax": {
                                        "type": "string",
                                        "description": "maxLength: 24"
                                    },
                                    "bool": {
                                        "type": [
                                            "boolean",
                                            "null"
                                        ]
                                    },
                                    "number": {
                                        "type": [
                                            "integer",
                                            "null"
                                        ]
                                    }
                                },
                                "additionalProperties": false,
                                "required": [
                                    "customerId",
                                    "companyName",
                                    "contactName",
                                    "contactTitle",
                                    "address",
                                    "city",
                                    "region",
                                    "postalCode",
                                    "country",
                                    "phone",
                                    "fax",
                                    "bool",
                                    "number"
                                ]
                            },
                            "editItem": {
                                "type": [
                                    "object",
                                    "null"
                                ],
                                "properties": {
                                    "customerId": {
                                        "type": "string",
                                        "description": "maxLength: 5"
                                    },
                                    "companyName": {
                                        "type": "string",
                                        "description": "maxLength: 40"
                                    },
                                    "contactName": {
                                        "type": "string",
                                        "description": "maxLength: 30"
                                    },
                                    "contactTitle": {
                                        "type": "string",
                                        "description": "maxLength: 30"
                                    },
                                    "address": {
                                        "type": "string",
                                        "description": "maxLength: 60"
                                    },
                                    "city": {
                                        "type": "string",
                                        "description": "maxLength: 15"
                                    },
                                    "region": {
                                        "type": "string",
                                        "description": "maxLength: 15"
                                    },
                                    "postalCode": {
                                        "type": "string",
                                        "description": "maxLength: 10"
                                    },
                                    "country": {
                                        "type": "string",
                                        "description": "maxLength: 15"
                                    },
                                    "phone": {
                                        "type": "string",
                                        "description": "maxLength: 24"
                                    },
                                    "fax": {
                                        "type": "string",
                                        "description": "maxLength: 24"
                                    },
                                    "bool": {
                                        "type": [
                                            "boolean",
                                            "null"
                                        ]
                                    },
                                    "number": {
                                        "type": [
                                            "integer",
                                            "null"
                                        ]
                                    }
                                },
                                "additionalProperties": false,
                                "required": [
                                    "customerId",
                                    "companyName",
                                    "contactName",
                                    "contactTitle",
                                    "address",
                                    "city",
                                    "region",
                                    "postalCode",
                                    "country",
                                    "phone",
                                    "fax",
                                    "bool",
                                    "number"
                                ]
                            },
                            "editField": {
                                "type": [
                                    "string",
                                    "null"
                                ]
                            },
                            "insertedItem": {
                                "type": [
                                    "object",
                                    "null"
                                ],
                                "properties": {
                                    "customerId": {
                                        "type": "string",
                                        "description": "maxLength: 5"
                                    },
                                    "companyName": {
                                        "type": "string",
                                        "description": "maxLength: 40"
                                    },
                                    "contactName": {
                                        "type": "string",
                                        "description": "maxLength: 30"
                                    },
                                    "contactTitle": {
                                        "type": "string",
                                        "description": "maxLength: 30"
                                    },
                                    "address": {
                                        "type": "string",
                                        "description": "maxLength: 60"
                                    },
                                    "city": {
                                        "type": "string",
                                        "description": "maxLength: 15"
                                    },
                                    "region": {
                                        "type": "string",
                                        "description": "maxLength: 15"
                                    },
                                    "postalCode": {
                                        "type": "string",
                                        "description": "maxLength: 10"
                                    },
                                    "country": {
                                        "type": "string",
                                        "description": "maxLength: 15"
                                    },
                                    "phone": {
                                        "type": "string",
                                        "description": "maxLength: 24"
                                    },
                                    "fax": {
                                        "type": "string",
                                        "description": "maxLength: 24"
                                    },
                                    "bool": {
                                        "type": [
                                            "boolean",
                                            "null"
                                        ]
                                    },
                                    "number": {
                                        "type": [
                                            "integer",
                                            "null"
                                        ]
                                    }
                                },
                                "additionalProperties": false,
                                "required": [
                                    "customerId",
                                    "companyName",
                                    "contactName",
                                    "contactTitle",
                                    "address",
                                    "city",
                                    "region",
                                    "postalCode",
                                    "country",
                                    "phone",
                                    "fax",
                                    "bool",
                                    "number"
                                ]
                            },
                            "tableWidth": {
                                "type": [
                                    "string",
                                    "null"
                                ]
                            }
                        },
                        "additionalProperties": false,
                        "required": [
                            "groupDescriptors",
                            "collapsedGroups",
                            "columnStates",
                            "expandedItems",
                            "filterDescriptors",
                            "sortDescriptors",
                            "searchFilter",
                            "page",
                            "skip",
                            "selectedItems",
                            "originalEditItem",
                            "editItem",
                            "editField",
                            "insertedItem",
                            "tableWidth"
                        ]
                    },
                    "columnOrdering": {
                        "type": "object",
                        "properties": {
                            "columns": {
                                "type": "array",
                                "items": {
                                    "type": "string"
                                }
                            }
                        },
                        "required": [
                            "columns"
                        ],
                        "additionalProperties": false
                    }
                },
                "required": [
                    "gridState",
                    "columnOrdering"
                ],
                "additionalProperties": false
            }
        }
    }
}

[EdCharbeneau]

It appears that when the schema is generated some of the types are not identified. Specifically array items.

Example

response_format is:

    "aggregateFunctions": {
        "type": [ "array", "null" ]
   },

Should be:

    "aggregateFunctions": {
        "type": ["array", "null"],
        "items": { "type": "string" }
    }

Now the error starts to make sense:

In context=('properties', 'gridState', 'properties', 'groupDescriptors', 'type', '0', 'items', 'properties', 'aggregateFunctions', 'type', '0'), array schema missing items.

Background:

This is a POC and I'm working with the path of least resistance by using existing classes that are available. The GridState object is quite complex and may not be best practice to send in its entirety. Some DTOs might be in order here. If it is determined that this issue is not a bug or helpful to the goals Extensions.AI, then please close at your discretion.

Current fix:

Per @stephentoub 's reply.
Setting useJsonSchemaResponseFormat to false resolves the issue.

[stephentoub]

Thanks. Any chance you could narrow down what Type results in the lack of items? e.g. if I run this:

using Microsoft.Extensions.AI;

Console.WriteLine(AIJsonUtilities.CreateJsonSchema(typeof(Wrapper<string>)));

class Wrapper<T>
{
    public T[] Values { get; set; }
}

I get output that includes items:

{"type":"object","properties":{"values":{"type":"array","items":{"type":"string"}}}}

It'd be helpful to understand / have a minimal repro for what Type is being passed that's resulting in us outputting an array type without items.

[stephentoub]

I can make that happen by having the array element type be object:

using Microsoft.Extensions.AI;

Console.WriteLine(AIJsonUtilities.CreateJsonSchema(typeof(object[])));

Based on https://github.com/dotnet/runtime/blob/09aa370dea075acf0b917b63bd41ebf4b92c1269/src/libraries/System.Text.Json/src/System/Text/Json/Schema/JsonSchema.cs#L35 and
https://github.com/dotnet/runtime/blob/09aa370dea075acf0b917b63bd41ebf4b92c1269/src/libraries/System.Text.Json/src/System/Text/Json/Schema/JsonSchemaExporter.cs#L274
it seems like this is by design, @eiriktsarpalis? Should we be emitting a type even in this true case? Or is there a transform we should be doing either in STJ, AIJsonUtilities, or in the OpenAI implementations to address that?

[EdCharbeneau]

        "type": [ "array", "null" ]
   },

AggregateFunctions's type is Collection

[JsonConverter(typeof(AggregateFunctionConverter))]
public abstract class AggregateFunction
{  ... props }

So basically object[] like your example?

There's a lot of custom JSON mapping happening within this object. I don't own the object's source code in this instance.