NodeFailedToLaunchException thrown from Codespaces container

[AArnott]

When using a devcontainer hosted by the Codespaces Preview, and when the .NET Core SDK is installed at the machine location, dotnet build or variants fail because msbuild can't fork its own process.

It works fine when I host the docker container locally (VS Code should have produced the same docker container locally).
It also works fine if I run ./init.ps1 -InstallLocality repo to install the .NET Core SDK to a user-level location and run dotnet build again.

Steps to reproduce

1. Visit https://github.com/AArnott/Library.Template/tree/msbuild5355
2. Click Open with Codespaces (you must be included in the limited preview)
   

Environment data

dotnet build /version
Microsoft (R) Build Engine version 16.5.0+d4cbfca49 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

16.5.0.12403

OS info:
Docker image based on mcr.microsoft.com/dotnet/core/sdk:3.1-bionic

 dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.1.201
 Commit:    b1768b4ae7

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/3.1.201/

Host (useful for support):
  Version: 3.1.4
  Commit:  0c2e69caa6

.NET Core SDKs installed:
  3.1.201 [/usr/share/dotnet/sdk]
  3.1.202 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.4 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.18 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.4 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download


❯ dotnet test

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.39
MSBUILD : error MSB1025: An internal failure occurred while running MSBuild.
Microsoft.Build.BackEnd.NodeFailedToLaunchException: No such file or directory
 ---> System.ComponentModel.Win32Exception (2): No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.LaunchNode(String msbuildLocation, String commandLineArgs)
   --- End of inner exception stack trace ---
   at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.LaunchNode(String msbuildLocation, String commandLineArgs)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.GetNode(String msbuildLocation, String commandLineArgs, Int32 nodeId, INodePacketFactory factory, Int64 hostHandshake, Int64 clientHandshake, NodeContextTerminateDelegate terminateNode)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProc.CreateNode(Int32 nodeId, INodePacketFactory factory, NodeConfiguration configuration)
   at Microsoft.Build.BackEnd.NodeManager.AttemptCreateNode(INodeProvider nodeProvider, NodeConfiguration nodeConfiguration)
   at Microsoft.Build.BackEnd.NodeManager.CreateNode(NodeConfiguration configuration, NodeAffinity nodeAffinity)
   at Microsoft.Build.Execution.BuildManager.PerformSchedulingActions(IEnumerable`1 responses)
   at Microsoft.Build.Execution.BuildManager.HandleNewRequest(Int32 node, BuildRequestBlocker blocker)
   at Microsoft.Build.Execution.BuildManager.ProcessPacket(Int32 node, INodePacket packet)
   at Microsoft.Build.Execution.BuildManager.<>c__DisplayClass67_0.<Microsoft.Build.BackEnd.INodePacketHandler.PacketReceived>b__0()
   at Microsoft.Build.Execution.BuildManager.ProcessWorkQueue(Action action)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Build.Execution.BuildManager.EndBuild()
   at Microsoft.Build.CommandLine.MSBuildApp.BuildProject(String projectFile, String[] targets, String toolsVersion, Dictionary`2 globalProperties, Dictionary`2 restoreProperties, ILogger[] loggers, LoggerVerbosity verbosity, DistributedLoggerRecord[] distributedLoggerRecords, Int32 cpuCount, Boolean enableNodeReuse, TextWriter preprocessWriter, Boolean detailedSummary, ISet`1 warningsAsErrors, ISet`1 warningsAsMessages, Boolean enableRestore, ProfilerLogger profilerLogger, Boolean enableProfiler, Boolean interactive, Boolean isolateProjects, Boolean graphBuild, String[] inputResultsCaches, String outputResultsCache)

MSBUILD : error MSB1025: An internal failure occurred while running MSBuild.
Microsoft.Build.BackEnd.NodeFailedToLaunchException: No such file or directory
 ---> System.ComponentModel.Win32Exception (2): No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.LaunchNode(String msbuildLocation, String commandLineArgs)
   --- End of inner exception stack trace ---
   at Microsoft.Build.CommandLine.MSBuildApp.BuildProject(String projectFile, String[] targets, String toolsVersion, Dictionary`2 globalProperties, Dictionary`2 restoreProperties, ILogger[] loggers, LoggerVerbosity verbosity, DistributedLoggerRecord[] distributedLoggerRecords, Int32 cpuCount, Boolean enableNodeReuse, TextWriter preprocessWriter, Boolean detailedSummary, ISet`1 warningsAsErrors, ISet`1 warningsAsMessages, Boolean enableRestore, ProfilerLogger profilerLogger, Boolean enableProfiler, Boolean interactive, Boolean isolateProjects, Boolean graphBuild, String[] inputResultsCaches, String outputResultsCache)
   at Microsoft.Build.CommandLine.MSBuildApp.Execute(String[] commandLine)
Unhandled exception. Microsoft.Build.BackEnd.NodeFailedToLaunchException: No such file or directory
 ---> System.ComponentModel.Win32Exception (2): No such file or directory
   at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at System.Diagnostics.Process.Start(ProcessStartInfo startInfo)
   at Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.LaunchNode(String msbuildLocation, String commandLineArgs)
   --- End of inner exception stack trace ---
   at Microsoft.Build.CommandLine.MSBuildApp.BuildProject(String projectFile, String[] targets, String toolsVersion, Dictionary`2 globalProperties, Dictionary`2 restoreProperties, ILogger[] loggers, LoggerVerbosity verbosity, DistributedLoggerRecord[] distributedLoggerRecords, Int32 cpuCount, Boolean enableNodeReuse, TextWriter preprocessWriter, Boolean detailedSummary, ISet`1 warningsAsErrors, ISet`1 warningsAsMessages, Boolean enableRestore, ProfilerLogger profilerLogger, Boolean enableProfiler, Boolean interactive, Boolean isolateProjects, Boolean graphBuild, String[] inputResultsCaches, String outputResultsCache)
   at Microsoft.Build.CommandLine.MSBuildApp.Execute(String[] commandLine)
   at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)

[AArnott]

A very similar error: dotnet/core#3309

[danmoseley]

In dotnet/core#3309 the issue was a Linux kernel bug that meant we did not get the file name of the main module - we got \. This callstack seems to have gotten past that point so I assume you don't have that bug. This is most likely because MSBuild is asking to launch a nonexistent file. It would be nice if we added the filename to the exception.

[rainersigwald]

@AArnott are you still seeing this? I don't repro--though my container now seems to have

.NET Core SDKs installed:
  3.1.201 [/usr/share/dotnet/sdk]
  3.1.300 [/usr/share/dotnet/sdk]

But it works even with a (modified) global.json pinning to 3.1.201.

[AArnott]

yes, I'm still seeing this. I see the same output from dotnet --list-sdks as you when it repros.
I'm happy to screenshare with you to get it to repro on your machine.

[danmoseley]

It would be helpful to have an env var that would trace the parameters that went to Microsoft.Build.BackEnd.NodeProviderOutOfProcBase.LaunchNode(String msbuildLocation, String commandLineArgs)

[marcpopMSFT]

@AArnott Is this still reproducing for you?

[AArnott]

@marcpopMSFT I can't test it, because before it creates a codespace per the repro steps, I hit this error:

[AArnott]

I just verified that it still repros. Please fix.

[rainersigwald]

@AArnott I still don't repro with a fresh Codespace:

PS /root/workspace/Library.Template> dotnet test      
Test run for /root/workspace/Library.Template/bin/Library.Tests/Debug/net472/Library.Tests.dll(.NETFramework,Version=v4.7.2)
Microsoft (R) Test Execution Command Line Tool Version 16.5.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 6.0310 Seconds
Test run for /root/workspace/Library.Template/bin/Library.Tests/Debug/netcoreapp2.1/Library.Tests.dll(.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 16.5.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 4.4405 Seconds
Test run for /root/workspace/Library.Template/bin/Library.Tests/Debug/netcoreapp3.1/Library.Tests.dll(.NETCoreApp,Version=v3.1)
Microsoft (R) Test Execution Command Line Tool Version 16.5.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...

A total of 1 test files matched the specified pattern.

Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 2.9370 Seconds

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:01:00.28
PS /root/workspace/Library.Template> dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   3.1.201
 Commit:    b1768b4ae7

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/3.1.201/

Host (useful for support):
  Version: 3.1.8
  Commit:  9c1330dedd

.NET Core SDKs installed:
  3.1.201 [/usr/share/dotnet/sdk]
  3.1.402 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.22 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download
PS /root/workspace/Library.Template> dotnet build /version
Microsoft (R) Build Engine version 16.5.0+d4cbfca49 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

16.5.0.12403

It sounds like yours was also a fresh codespace so the differing behavior is mysterious to me.

Want to grab some time on my calendar to screen share to your instance to poke at it? Also, have you heard of anyone else with such a problem?

[AArnott]

I just tried it again. I see this in the corner after the codespace is initialized:

Should I file a separate bug on that?

But the dotnet test issue did not repro this time using the link in the issue description. But then I went to the tip of master of the same repo and created a container and this time it did repo. Given it has repro'd and not on both of these commits, I think this is a non-deterministic failure.

@rainersigwald I'll grab some time on your calendar as you offered. Thank you.