Cohost Razor and Roslyn to better share object types and object instances

[phil-allen-msft]

Individual endpoint status: #10364

In no particular order:

1. Moving off IDynamciFileInfo
   • What to do about SpanMapping?
     • ANSWER: Remove, use #line directives
       • Razor/Roslyn Cohosting: Use #line directives instead of ISpanMappingService roslyn#72136
     • Tweak compiler to fix using directives: Don't emit #line default and #line hidden between consecutive using directives #9991
       • See: No Quick Fix to add using statemens in Razor files #9946 / Add Using code action doesn't work in a world where Roslyn and Razor lack span mapping service #8671
   • What to do about excerpt service?
     • ANSWER: Remove, take the hit. Discuss if someone complains
   • What to do about ClientName:
     • ANSWER: Dynamic registration, and custom API to filter workspace diags
   • What to do about SupportsDiagnostics
     • Nothing? We're pull diags now
   • What to do about CanApplyChanges
     • Nothing? SourceGeneratedDocumentState does the same thing
2. Razor calling Roslyn via C# API
   • Getting a SourceGeneratedDocument
     • Need a way to calc file path?
   • Razor calls "helper" methods in Roslyn LSP handlers that take Document types instead of TextDocumentIdentifier types
     • These methods are exposed through Razor EA. eg. Expose semantic tokens functionality to Razor EA roslyn#72495 and Cohosting: Add semantic tokens support, via Roslyn OOP #10097
     • Need to share Roslyn protocol types (for return types and possibly some parameters): Expose folding range functionality to Razor roslyn#73609
     • Generally Roslyn protocol types should move towards being opinionated about what Roslyn produces, and become idiomatic .NET library, rather than idiomatic JSON objects (eg, no SumType, custom converters etc.)
       • Initially there might be pain due to dual insertions because of changes to protocol types, but hopefully things will settle with this approach
3. Pull data from SG
   • Razor services to OOP
     • OOP needs MEF: Teach OOP how to MEF #9953
   • Razor calls C# API in OOP
     • Not all C# API will work in OOP. Changes on Roslyn side might be needed
     • Razor can call into Roslyn MEF composition in OOP though. See Expose folding range functionality to Razor roslyn#73609
   • Razor returns from OOP with either a result, or instructions for delegating to Html: Uri Presentation Part 2, aka Cohost Html on-demand generation #10359
     • Sometimes, once delegation is done, we have call back to OOP
     • TODO: Callbacks are hard to implement, but might have other benefits? Perf?
4. Miscellaneous files, and custom or non-Razor SDK usage and how that could work
   • Scenario is projects that don’t reference the source generator, and don’t have .cshtml and .razor files as AdditionalDocuments.
   • Can we have a service that allows customization/custom creation of a project in Misc Files workspace – Razor could add additional document and source generator reference
   • Could Project System lie to Roslyn and add an additional file, instead/as well as dynamic
     • Doesn't help in the C# extension without devkit, where that code doesn't exist
     • Doing it in Roslyn would solve for DevKit and non
     • We could however do it through https://github.com/dotnet/project-system/blob/main/src/Microsoft.VisualStudio.ProjectSystem.Managed/ProjectSystem/DesignTimeTargets/Microsoft.CSharp.DesignTime.targets (or equiv in VS Code)
       • We can add .razor and .cshtml as additional documents, and we can reference the source generator, and Roslyn and everything else will be none the wiser
       • Need to ensure nothing conflicts with the real Web and Razor sdks, and what they do
       • Should also create a <DefaultRazorFileIdeSupport>false</DefaultRazorFileIdeSupport> get-out-of-jail-free property that people can specify to disable the behaviour if it causes problems
         • At the end of the day, people who have "weird" set ups can solve their issues with MS Build goo (add additional documents, reference source gen), but the upgrade experience is bad. Is it better if we do that for them, in a way that only affects design time, and give them an opt-out if it causes issues?
   • Could Roslyn just add all dynamic files as additional documents?
     • Needs to be sensitive to duplicates
     • Gets difficult to thread things through, as projects change SDKs and people modify things.
     • Using misc files is easier as every individual file is isolated. Is that an acceptable tooling experience though?
   • We can hook into miscellaneous file creation easily enough:
     • Currently every individual misc file in Roslyn gets its own project, logic is here which is called from here
     • These methods won't be called for .razor files now, because Roslyn doesn't own the buffer, so we'd have to call them through EA.
     • Possibly just as easy for us to create our own misc files project(s), with the right references etc., and expose a way to add them to the Roslyn misc workspace. As long as the LSP server can find them again, the rest shouldn't really matter
     • Will need separate hooks for VS Code
     • Won't help if a file is an AdditionalDocument, but the source generator isn't referenced
5. Dynamic registration
   • ANSWER
     • Remove RazorInProcLanguageServer once we're using C# API
     • Dynamic registration for Razor files, in AlwaysActivateInProcLanguageServerClient: Expose a way for Razor to register capabilities into the AlwaysActivateInProcLanguageClient roslyn#72609 and Move cohosting to dynamic registration #10165
     • Remove Cohost language server client entirely: Clean up Razor EA roslyn#73040
   • We can’t statically and dynamically register for the same capabilities, so need an API to allow Razor to know not to register for certain things, and maybe Roslyn to delegate to Razor on those endpoints (eg pull diagnostics) or does that happen naturally with Marco’s work?
     • Not sure if this is necessary, but happy to move C# to dynamic registration if so
   • CustomMessageTarget
     • Roslyn doesn't specify one currently
       • Can plug this in somehow
       • Can remove the need for it since Cohosting is all on the VS side? Though that won't work for VS Code
       • Can remove the need for it in C# by calling through C# API instead of LSP
   • Need to be smarter about when we light up too: Don't load Razor when not necessary #10279
6. Legacy Editor
   • Still needs Razor Project System?
     • ANSWER: Yes
   • Still needs IDynamicFileInfo?
     • ANSWER: Yes, for background compilation so local tag helpers are found
     • To discuss: Suppress generator in Legacy only, but not LSP?
       • Also needed when Cohost feature flag is off
7. Faults and telemetry etc.
   • Need to make sure faults from cohost end points can be routed to the Razor team easily
     • Currently being reported by the Roslyn LSP server
     • Should Razor report them too?
     • Can we have a mechanism where Roslyn knows to categorize as razor (LSP request language name?) or override the fault event name itself?
       • We have a mechanism like this in place for logging, for VS Code, so could use that perhaps
8. Roslyn options
   • They don't sync to OOP automatically sadly, so code in OOP always gets defaults
   • Roslyn uses callbacks for getting options values on demand. Razor services don't have callbacks, and if they did, they couldn't get the Roslyn options
   • To Discuss: Can we create a generic system so our EA can fill in options somehow? Will do as they come up. Razor should pass options to Roslyn anyway.
   • Expect Razor files do not respect C# code style new line rules #11276 to regress without some action here Fixed
9. VS Code follow up
   • Log with a custom message to get logs in the Razor output window Allow the Roslyn client to log messages from Razor vscode-csharp#8201
   • Check instructions for the "Report an Issue" command, and TODO include feature flag values when reporting.
   • Settings: ClientSettingsReader is just one big TODO right now
   • In DevKit, Roslyn registered doc sync endpoints for Razor files. In cohosting we need to do this always. Right now these things happen in different repos, and possibly both at the same time, which could be bad.
   • Telemetry needs to be hooked up properly in the VS Code extension
   • Initialization feels slow. Need to make sure we're not doing anything non-cohosting, and consider "warming up" the Razor MEF composition (and/or add caching?)
   • Restore documents don't refresh themselves, so are stuck in a bad state until an edit/scroll/resize
   • Need to communicate back to the HtmlDocumentSynchronizer when a Razor document is closed, we can remove any synchronization tasks