ObjectDisposedException during HTTP/2 mTLS handshake in Kubernetes production environment #119022
Description
Description
Scenario: Health probe service making mTLS gRPC calls to a TTS service in Kubernetes production environment.
Code Pattern:
// Create gRPC channel with mTLS client certificate
var clientCert = X509Certificate2.CreateFromPemFile(certPath, keyPath);
var httpHandler = new HttpClientHandler
{
ClientCertificates = { clientCert }
};
using var channel = GrpcChannel.ForAddress(serviceUrl, new GrpcChannelOptions
{
HttpHandler = httpHandler
});
var client = new Synthesizer.SynthesizerClient(channel);
var reply = client.Synthesize(request, headers);
await foreach (var message in reply.ResponseStream.ReadAllAsync())
{
// Process response
}
Reproduction Steps
Issue reproduces consistently in Kubernetes production environment but not in local development, suggesting environmental factors (network latency, SSL negotiation timing) contribute to the disposal race condition.
- Configure gRPC client with mTLS (client certificates)
- Connect to HTTPS endpoint requiring client certificates
- Make repeated gRPC calls
- Observe ObjectDisposedException in SslStream
Expected behavior
Successful HTTP/2 gRPC connection establishment and streaming response processing without disposal errors.
Actual behavior
Error: ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.Security.SslStream'
Full Stack Trace:
Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake. ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.Security.SslStream'.", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.")
---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.
---> System.ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.Security.SslStream'.
at System.Net.Security.SslStream.<ThrowIfExceptional>g__ThrowExceptional|126_0(ExceptionDispatchInfo e)
at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.SetupAsync(CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.SetupAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConstructHttp2ConnectionAsync(Stream stream, HttpRequestMessage request, IPEndPoint remoteEndPoint, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConstructHttp2ConnectionAsync(Stream stream, HttpRequestMessage request, IPEndPoint remoteEndPoint, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
--- End of inner exception stack trace ---
at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+MoveNext()
at Grpc.Core.AsyncStreamReaderExtensions.ReadAllAsyncCore[T](IAsyncStreamReader`1 streamReader, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
Regression?
No response
Known Workarounds
- Using HTTPS without client certs (port 8079) - works
- Only mTLS (port 8081) causes issues
Configuration
Language: C# / .NET 8.0
gRPC Packages:
Grpc.AspNetCore: 2.71.0
Grpc.Net.Client: 2.71.0 (implicit via Grpc.AspNetCore)
Grpc.Tools: 2.72.0
Google.Protobuf: 3.32.0
.NET SDK: 8.0.413
Runtime: Microsoft.AspNetCore.App 8.0.19
Target Framework: net8.0
Other information
Key Observations:
Issue occurs consistently in Kubernetes production environment
Timing: ~16 seconds after application startup, during first gRPC call attempt
mTLS specific: Issue appears when using client certificates for mutual TLS
HTTP/2 related: Error occurs during HTTP/2 handshake setup phase
Workarounds Attempted:
Fresh HttpClientHandler creation per call
Connection pooling disabled (PooledConnectionLifetime = TimeSpan.Zero)
OCSP/CRL check disabling
Multiple timeout configurations
DisposeHttpClient = true in GrpcChannelOptions
Custom SSL certificate validation callbacks
Environment Context:
Kubernetes service mesh: Complex network infrastructure
SSL handshake timing: Often takes 15+ seconds in this environment
Certificate management: PEM files loaded from mounted volumes