Default useNativeJsonSchema to true

[stephentoub]

Per #6064 (comment), flip useNativeJsonSchema to default to true.

In addition or alternatively, we should consider exposing a capabilities mechanism that IChatClients can implement. This could be:

• Augmenting ChatClientMetadata with a capabilities enum or struct, where the client can make claims about supported features, like function calling, structured output, etc.
• Adding a marker interface for the capability, such that it can be queried via GetService

useNativeJsonSchema could then default to false unless this flag is observed.

[SteveSandersonMS]

"Capabilities" API design thoughts:

• We would want this to be extensible. There might be capabilities defined by 3rd party code.
• Ideally it wouldn't just be a set of boolean flags. It's possible to imagine future capabilities like "max context window length". As such it's not really different from ChatClientMetadata/EmbeddingGeneratorMetadata in general.
  • Except in one way it might be different: some capabilities will be model-specific. For example, depending on which OpenAI model you're using, it might or might not support native structured output.

Our existing metadata concept is a bit limited when it comes to model-specific things. For example, EmbeddingGeneratorMetadata has both:

• ModelId, which may be null, and even if it isn't, may be overridden on a per-call basis
• Dimensions, which may be model-specific, and for some models, a range of possible dimension values may be supported. Having a single int? is barely useful since you don't really know which model will be used, and even if you do, the dimensionality may vary.

The naming doesn't indicate it, but these values are both defaults that might be overridden on a per-call basis. And there simply isn't a way to query for the default number of dimensions for a particular other model.

Possible alternative

Imagine we split "provider metadata" from "model metadata". That is, we continue to have ChatClientMetadata and EmbeddingGeneratorMetadata, with properties like:

• ProviderName
• ProviderUri
• DefaultModelId (replaces ModelId)

Then we would also give them a method:

public virtual ChatModelMetadata GetModelMetadata(string? modelId = null) // If modelId is null, uses DefaultModelId

The new types ChatModelMetadata and EmbeddingModelMetadata could initially be very minimal:

public class ChatModelMetadata
{
    public bool? SupportsNativeJsonSchema { get; } // Null if the provider doesn't know
}

public class EmbeddingModelMetadata
{
    public int? Dimensions { get; } // Can be null if the provider doesn't know or if it can vary
}

Of course these could be extended arbitrarily, either:

• By third-parties right now subclassing them
  • App devs can always check if (modelMetadata is OllamaChatModelMetadata) { ... }
• Or by us in the future adding properties like ContextWindowLength, SupportsReasoning, etc.

All the properties on these would be nullable unless we can be sure that all providers will always know the details for all models. Because the set of models can change on an hourly basis. So it would be up to the provider to decide what to do, e.g., to return null if it doesn't recognize the model name or to use some assumption like "all future models can be expected to support structured output" or whatever. But that's up to the provider to decide as there's no rule MEAI can impose in general.

Drawbacks / Limitations

The GetModelMetadata method would allocate on a per-call basis unless the provider implements some caching. We'd expect these metadata types never to be mutable.

This doesn't allow for the model-specific metadata to be determined asynchronously, e.g., by querying some HTTP endpoint on the provider. We could have GetModelMetadataAsync but is that making things too hard for the consumer?

It's not very friendly to middleware intercepting GetModelMetadata calls to modify the results. The only way to do that would be to intercept GetService<ChatClientMetadata>() and then call through into innerClient to get the metadata, and return some custom metadata object that forwards property getters to the original object, and intercepts GetModelMetadata calls with one that calls the original and returns some new wrapper object that overrides properties. The whole thing isn't really workable. However, unless we have specific scenarios in mind, I'm inclined to think this is not a desired scenario at all: the metadata is meant to be about the underlying provider and its models, and not about any middleware that wraps it. So for example if you had some "context window reduction middleware", you would not want to override ContextWindowLength to be infinite but would still want the metadata to reflect the true underlying LLM.

[SteveSandersonMS]

@stephentoub What do you think about the above? Maybe it's more comprehensive a solution than we need right now if we could just add a bool flag to ChatClientMetadata. However if we do later need a more comprehensive solution, and I expect we will, it would be best to have it now.

[stephentoub]

Overall, I agree we should do this or something along these lines.

https://learn.microsoft.com/en-us/azure/ai-foundry/model-inference/concepts/models#model-families might provide inspiration about the kinds of things we'd want to expose (now or later), or the info Google shows in their model cards, e.g. https://ai.google.dev/gemini-api/docs/models#gemini-2.0-flash.

It does, though, beg the question I think of what information the client implementation would actually know. Presumably a client like OpenAIClient isn't going to hardcode information about every model, especially since the available models evolve so rapidly. I'd expect instead that the client would need to fetch model information from the service, and if so, it having to do that might need to be factored into the API shape, e.g. if GetModelMetadata should actually be GetModelMetadataAsync. An implementation could of course implement it synchronously if it had a hardcoded answer, e.g. if it knew no models supported structured output or if all of them did.

Various services do provide endpoints today for getting model information, but generally limited in the information they provide, e.g.
https://learn.microsoft.com/en-us/azure/machine-learning/reference-model-inference-info?view=azureml-api-2
https://platform.openai.com/docs/api-reference/models/object
https://docs.aws.amazon.com/bedrock/latest/APIReference/API_ListFoundationModels.html#API_ListFoundationModels_ResponseSyntax
https://ai.google.dev/api/models#Model
https://docs.anthropic.com/en/api/models

There's also then the question of abstracting over the ability to get a list of available models. Maybe if we wanted to do that eventually, though, we'd choose to do it via some kind of mix-in / separate interface that could be queried for. While in theory it'd be nice to say "use any model that meets the following criteria", in practice I've not seen situations where that would be viable.

[SteveSandersonMS]

Another design point: should ChatClientMetadata/EmbeddingGeneratorMetadata become abstract so that GetModelMetadataAsync can itself be abstract? This would force anyone who wants to return provider metadata to also subclass the metadata type and be forced to implement GetModelMetadataAsync.

This is a more pure design, but puts more burden on anyone getting started with implementing IChatClient etc, and has more of a back-compat cost for existing implementations.

Right now I'm leaning towards having GetModelMetadataAsync be virtual and defaulting to returning empty model-specific metadata, but very open to changing that.

[stephentoub]

Right now I'm leaning towards having GetModelMetadataAsync be virtual and defaulting to returning empty model-specific metadata, but very open to changing that.

This would be my preference.

[SteveSandersonMS]

A drawback to having GetModelMetadataAsync is that it implies you're allowed to do expensive work each time it's called. This means that having telemetry show the number of embedding dimensions by default means we're potentially introducing extra expensive work even if the app developer doesn't care about it.

OpenTelemetryEmbeddingGenerator can't legitimately cache this across instances since it doesn't know if, for example, the number of dimensions varies based on some separate but mutable config. So we would become reliant on IChatClient implementations handling their own caching for this, for example using a ConcurrentDictionary<string, Task<EmbeddingModelMetadata>>. It's certainly possible for them to do that, and not hugely difficult, but is a risk we'd be taking because if they fail to do it, simply collecting telemetry would introduce nontrivial extra costs.

Alternatives:

• Have a synchronous GetModelMetadata, which would more strongly communicate the need not to do expensive work such as async I/O
• Or, in ChatClientMetadata, deal with caching in the base class. Subclasses would only be able to override GetModelMetadataCoreAsync(string modelId) and its docs would be clear that (1) you're only allowed to return results as a function of modelId and not any other contextual information, since it will be cached, and (2) cache entries never expire, so you have to just know that whatever you return will be valid for the lifetime of the application instance, and (3) the cache size is unbounded, so you have to not be generating distinct model IDs for every call otherwise it's a memory leak.

Right now I'm still thinking we'd have GetModelMetadataAsync and would leave it up to IChatClient implementations to use suitable caching with their own cache keys and expiry policy.

[stephentoub]

Right now I'm still thinking we'd have GetModelMetadataAsync and would leave it up to IChatClient implementations to use suitable caching with their own cache keys and expiry policy.

Agreed.