Getting Google.Protobuf.InvalidProtocolBufferException when running Aspire

[KenBrannigan]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

When I enable Data Access Protection on my ASP.NET Core MVC site running under Aspire, I get a Google.Protobuf.InvalidProtocolBufferException.

Expected Behavior

No exceptions to be thrown when running Aspire.

Steps To Reproduce

I have created a repo for you so you can see the exception:

https://github.com/KenBrannigan/AspireDefaultAzureCredentialIssue

After running the app wait a few seconds and the exception will display.

Exceptions (if any)

info: Aspire.Hosting.DistributedApplication[0]
Distributed application started. Press Ctrl+C to shut down.
fail: Aspire.Hosting.Dashboard.Grpc.AspNetCore.Server.ServerCallHandler[6]
Error when executing service method 'Export'.
Google.Protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either that the input has been truncated or that an embedded message misreported its own length.
at Google.Protobuf.SegmentedBufferHelper.RefillFromReadOnlySequence(ReadOnlySpan1& buffer, ParserInternalState& state, Boolean mustSucceed) at Google.Protobuf.SegmentedBufferHelper.RefillBuffer(ReadOnlySpan1& buffer, ParserInternalState& state, Boolean mustSucceed)
at Google.Protobuf.ParsingPrimitives.ReadRawByte(ReadOnlySpan1& buffer, ParserInternalState& state) at Google.Protobuf.ParsingPrimitives.ParseRawVarint32SlowPath(ReadOnlySpan1& buffer, ParserInternalState& state)
at Google.Protobuf.ParsingPrimitives.ParseRawVarint32(ReadOnlySpan1& buffer, ParserInternalState& state) at OpenTelemetry.Proto.Trace.V1.Status.pb::Google.Protobuf.IBufferMessage.InternalMergeFrom(ParseContext& input) in /_/artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/trace/v1/Trace.cs:line 2612 at Google.Protobuf.ParsingPrimitivesMessages.ReadRawMessage(ParseContext& ctx, IMessage message) at Google.Protobuf.ParsingPrimitivesMessages.ReadMessage(ParseContext& ctx, IMessage message) at OpenTelemetry.Proto.Trace.V1.Span.pb::Google.Protobuf.IBufferMessage.InternalMergeFrom(ParseContext& input) in /_/artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/trace/v1/Trace.cs:line 1642 at Google.Protobuf.ParsingPrimitivesMessages.ReadRawMessage(ParseContext& ctx, IMessage message) at Google.Protobuf.ParsingPrimitivesMessages.ReadMessage(ParseContext& ctx, IMessage message) at Google.Protobuf.FieldCodec.<>c__DisplayClass32_01.b__0(ParseContext& ctx)
at Google.Protobuf.Collections.RepeatedField1.AddEntriesFrom(ParseContext& ctx, FieldCodec1 codec)
at OpenTelemetry.Proto.Trace.V1.ScopeSpans.pb::Google.Protobuf.IBufferMessage.InternalMergeFrom(ParseContext& input) in //artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/trace/v1/Trace.cs:line 844 at Google.Protobuf.ParsingPrimitivesMessages.ReadRawMessage(ParseContext& ctx, IMessage message)
at Google.Protobuf.ParsingPrimitivesMessages.ReadMessage(ParseContext& ctx, IMessage message)
at Google.Protobuf.FieldCodec.<>c__DisplayClass32_01.<ForMessage>b__0(ParseContext& ctx) at Google.Protobuf.Collections.RepeatedField1.AddEntriesFrom(ParseContext& ctx, FieldCodec1 codec) at OpenTelemetry.Proto.Trace.V1.ResourceSpans.pb::Google.Protobuf.IBufferMessage.InternalMergeFrom(ParseContext& input) in /_/artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/trace/v1/Trace.cs:line 560 at Google.Protobuf.ParsingPrimitivesMessages.ReadRawMessage(ParseContext& ctx, IMessage message) at Google.Protobuf.ParsingPrimitivesMessages.ReadMessage(ParseContext& ctx, IMessage message) at Google.Protobuf.FieldCodec.<>c__DisplayClass32_01.b__0(ParseContext& ctx)
at Google.Protobuf.Collections.RepeatedField1.AddEntriesFrom(ParseContext& ctx, FieldCodec1 codec)
at OpenTelemetry.Proto.Collector.Trace.V1.ExportTraceServiceRequest.pb::Google.Protobuf.IBufferMessage.InternalMergeFrom(ParseContext& input) in //artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/collector/trace/v1/TraceService.cs:line 241
at Google.Protobuf.ParsingPrimitivesMessages.ReadRawMessage(ParseContext& ctx, IMessage message)
at Google.Protobuf.MessageExtensions.MergeFrom(IMessage message, ReadOnlySequence1 data, Boolean discardUnknownFields, ExtensionRegistry registry) at Google.Protobuf.MessageParser1.ParseFrom(ReadOnlySequence1 data) at OpenTelemetry.Proto.Collector.Trace.V1.TraceService.__Helper_DeserializeMessage[T](DeserializationContext context, MessageParser1 parser) in //artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/collector/trace/v1/TraceServiceGrpc.cs:line 62
at OpenTelemetry.Proto.Collector.Trace.V1.TraceService.<>c.<.cctor>b__13_0(DeserializationContext context) in //artifacts/obj/Aspire.Dashboard/win-x64/Release/net8.0/win-x64/opentelemetry/proto/collector/trace/v1/TraceServiceGrpc.cs:line 69
at Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadSingleMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func2 deserializer) at Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext)
at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase3.<HandleCallAsync>g__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method2 method, Task handleCall)

.NET Version info

.NET SDK:
Version: 9.0.102
Commit: cb83cd4923
Workload version: 9.0.100-manifests.4a54b1a6
MSBuild version: 17.12.18+ed8c6aec5

Runtime Environment:
OS Name: Windows
OS Version: 10.0.22631
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\9.0.102\

.NET workloads installed:
[aspire]
Installation Source: VS 17.12.35707.178
Manifest Version: 8.2.2/8.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.2.2\WorkloadManifest.json
Install Type: Msi

[maccatalyst]
Installation Source: VS 17.12.35707.178
Manifest Version: 18.1.9163/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.maccatalyst\18.1.9163\WorkloadManifest.json
Install Type: Msi

[maui-windows]
Installation Source: VS 17.12.35707.178
Manifest Version: 9.0.14/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.maui\9.0.14\WorkloadManifest.json
Install Type: Msi

[ios]
Installation Source: VS 17.12.35707.178
Manifest Version: 18.1.9163/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.ios\18.1.9163\WorkloadManifest.json
Install Type: Msi

[android]
Installation Source: VS 17.12.35707.178
Manifest Version: 35.0.7/9.0.100
Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.android\35.0.7\WorkloadManifest.json
Install Type: Msi

Configured to use loose manifests when installing new manifests.

Host:
Version: 9.0.1
Architecture: x64
Commit: c8acea2262

.NET SDKs installed:
9.0.102 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 8.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.1 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
Not set

global.json file:
Not found

Anything else?

I noticed when I change the data access protection code in line 34 of Program.cs (WebApplication1) from using DefaultAzureCredential to use VisualStudioCredential I do not get the error. If I change it to ManagedIdentityCredential I do get the exception. Since I am running locally I do not have a managed identity and it seems that the error that gets thrown by that call failure is causing the telemetry to fail.

I also noticed that even if I use DefaultAzureCredential but I do not use Azure.Monitor.OpenTelemetry.AspNetCore nuget package in the service defaults project to send my telemetry to application insights I do not get the exception. It seems the issue is sending the failed managed identity call from DefaultAzureCredential through the Azure.Monitor.OpenTelemetry.AspNetCore nuget package to application insights

[maddymontaquila]

Have you tried running the project without Aspire, or will it not be able to repro without the rest of your apphost resources?

[maddymontaquila]

cc @samsp-msft potentially an otel issue??

[KenBrannigan]

Have you tried running the project without Aspire, or will it not be able to repro without the rest of your apphost resources?

I just tried running WebApplication1 using connection strings for Application Insights, KeyVault, and Blob Storage all pointed to Azure resources, and did not have the issue. It only happens when running through Aspire.