Add ParserOptions and CodeGenerationOptions properties to RazorCodeDocument and rationalize options configuration (#11526)

> [!IMPORTANT]
> This contains RazorSDK breaking changes! I've prepared a fix for these
changes that can applied when the SDK build fails:
DustinCampbell/sdk@4e955e1

> [!WARNING]
> This is a big change with a lot of commits. To facilitate code
reviews, I've tried to keep the commit history informative. Most commits
have explanatory messages and product and pure test changes are in
separate commits.

The majority of the changes are in the compiler, so this will require
two compiler reviews from @chsienki, @333fred, @jjonescz, or @jaredpar.

CI Build:
https://dev.azure.com/dnceng/internal/_build/results?buildId=2647163&view=results
Test Insertion:
https://dev.azure.com/devdiv/DevDiv/_git/VS/pullrequest/612761

## Summary

- Refactor and simplify `RazorParserOptions` API and its builder
pattern.
- Refactor and simplify `RazorCodeGenerationOptions` API and its builder
pattern.
- Introduce `RazorProjectEngineBuilder` extension methods to configure
parser and code generation options.
- Consolidate or remove most `IConfigureRazorParserOptionsFeature` and
`IConfigureRazorCodeGenerationOptionsFeature` implementations.
- Clean up directive configuration to avoid using `ICollection<T>` and
`IReadOnlyDictionary<TKey, TValue>`.
- Add `ParserOptions` and `CodeGenerationOptions` properties to
`RazorCodeDocument` (and remove them from `RazorCodeDocument.Items`)
- Ensure that `RazorCodeDocument` is always created with valid parser
and code generation options.
- Remove the legacy `IRazorParserOptionsFeature` and
`IRazorCodeGenerationOptionsFeature` interfaces and their
implementations. These served as fallbacks when a `RazorCodeDocument`
didn't have parser or code gen options. However, that's no longer the
case, so the fallbacks can be removed.
- Remove the `IRazorParserOptionsFactoryProjectFeature` and
`IRazorCodeGenerationOptionsFactoryProjectFeature` interfaces and their
implementations. These were only used by `RazorProjectEngine` when
creating a `RazorCodeDocument` and their functionality has been rolled
into `RazorProjectEngine.`
- Provide several helper methods in the compiler and test infrastructure
to make it easy to create a `RazorCodeDocument` from a
`RazorProjectEngine`.
- Unify the compiler test infrastructure for `RazorProjectEngine`-based
tests. The same test infrastructure is shared across the Language and
"Extensions" tests
- Introduce a `RazorCodeDocumentProcessor` helper class for executing
particular compiler phases and passes with a `RazorCodeDocument`.
- Graduate many copied-and-pasted test helpers to shared extension
methods.
- Update 100s of tests to always create `RazorCodeDocuments` from a
`RazorProjectEngine`. This ensures that the options are always
configured correctly.
- Rationalize the `RazorCodeDocument.Create(...)` API. Realistically,
this should rarely be used. It's always best to acquire a
`RazorCodeDocument` from a `RazorProjectEngine`.
- Remove `#nullable disable` where it's easy to do. 😄

> [!TIP]
> If you wish to only review product changes, you can skip the following
commits. Note that 1, 2, and 15 are strays, but 3-14 are grouped
together in the commit history.
>
> 1. Remove TestRazorCodeDocument.Create(Source, Imports) test helper
(4955122)
> 2. Fix DefaultRazorParsingPhaseTest
(45e62cb)
> 3. RazorProjectEngineTestBase: Remove CreateEngine()
(ab97efc)
> 4. ModelDirectiveTest: Update to create code document from project
engine
(5e1cd9d)
> 5. ModelDirectiveTest: Fix incorrect test assertions
(c6382d8)
> 6. Refactor to share more test infrastructure
(22ec9bd)
> 7. PageDirectiveTest: Move to new test infrastructure
(4d5954c)
> 8. Clean up Version1_X tests
(ff5c526)
> 9. Clean up Version2_X tests
(601d9be)
> 10. Finish cleaning up MS.ANC.Mvc.Razor.Extensions.Test
(fa98b37)
> 11. Refactor and improve the testing API of RazorProjectEngineTestBase
(209fa7e)
> 12. Rename IntermediateNode test helpers and unneeded remove null
assertions
(55a3731)
> 13. Remove RazorCodeDocument.Create(...) calls from compiler tests
(8f2a0f3)
> 14. Remove RazorCodeDocument.Create(...) calls from tooling code
(9e4db21)
> 15. Clean up and document RazorProjectEngineTestBase
(2243678)

Author: DustinCampbell

src/Compiler/Microsoft.AspNetCore.Mvc.Razor.Extensions.Version1_X/test/InjectDirectiveTest.cs

@@ -1,41 +1,51 @@
 // Licensed to the .NET Foundation under one or more agreements.
 // The .NET Foundation licenses this file to you under the MIT license.
 
-#nullable disable
-
 using Microsoft.AspNetCore.Razor.Language;
 using Microsoft.AspNetCore.Razor.Language.Intermediate;
 using Xunit;
 
 namespace Microsoft.AspNetCore.Mvc.Razor.Extensions.Version1_X;
 
-public class InjectDirectiveTest
+public class InjectDirectiveTest : RazorProjectEngineTestBase
 {
+    protected override RazorLanguageVersion Version => RazorLanguageVersion.Version_1_1;
+
+    protected override void ConfigureProjectEngine(RazorProjectEngineBuilder builder)
+    {
+        // Notice we're not registering the InjectDirective.Pass here so we can run it on demand.
+        builder.AddDirective(InjectDirective.Directive);
+        builder.AddDirective(ModelDirective.Directive);
+
+        builder.Features.Add(new RazorPageDocumentClassifierPass());
+        builder.Features.Add(new MvcViewDocumentClassifierPass());
+    }
+
+    protected override void ConfigureCodeDocumentProcessor(RazorCodeDocumentProcessor processor)
+    {
+        processor.ExecutePhasesThrough<IRazorDocumentClassifierPhase>();
+    }
+
     [Fact]
     public void InjectDirectivePass_Execute_DefinesProperty()
     {
         // Arrange
-        var codeDocument = CreateDocument(@"
+        var codeDocument = ProjectEngine.CreateCodeDocument(@"
 @inject PropertyType PropertyName
 ");
 
-        var engine = CreateEngine();
-        var pass = new InjectDirective.Pass()
-        {
-            Engine = engine,
-        };
-
-        var irDocument = CreateIRDocument(engine, codeDocument);
+        var processor = CreateCodeDocumentProcessor(codeDocument);
 
         // Act
-        pass.Execute(codeDocument, irDocument);
+        processor.ExecutePass<InjectDirective.Pass>();
 
         // Assert
-        var @class = FindClassNode(irDocument);
-        Assert.NotNull(@class);
-        Assert.Equal(2, @class.Children.Count);
+        var documentNode = processor.GetDocumentNode();
+        var classNode = documentNode.GetClassNode();
 
-        var node = Assert.IsType<InjectIntermediateNode>(@class.Children[1]);
+        Assert.Equal(2, classNode.Children.Count);
+
+        var node = Assert.IsType<InjectIntermediateNode>(classNode.Children[1]);
         Assert.Equal("PropertyType", node.TypeName);
         Assert.Equal("PropertyName", node.MemberName);
     }
@@ -44,28 +54,23 @@ @inject PropertyType PropertyName
     public void InjectDirectivePass_Execute_DedupesPropertiesByName()
     {
         // Arrange
-        var codeDocument = CreateDocument(@"
+        var codeDocument = ProjectEngine.CreateCodeDocument(@"
 @inject PropertyType PropertyName
 @inject PropertyType2 PropertyName
 ");
 
-        var engine = CreateEngine();
-        var pass = new InjectDirective.Pass()
-        {
-            Engine = engine,
-        };
-
-        var irDocument = CreateIRDocument(engine, codeDocument);
+        var processor = CreateCodeDocumentProcessor(codeDocument);
 
         // Act
-        pass.Execute(codeDocument, irDocument);
+        processor.ExecutePass<InjectDirective.Pass>();
 
         // Assert
-        var @class = FindClassNode(irDocument);
-        Assert.NotNull(@class);
-        Assert.Equal(2, @class.Children.Count);
+        var documentNode = processor.GetDocumentNode();
+        var classNode = documentNode.GetClassNode();
+
+        Assert.Equal(2, classNode.Children.Count);
 
-        var node = Assert.IsType<InjectIntermediateNode>(@class.Children[1]);
+        var node = Assert.IsType<InjectIntermediateNode>(classNode.Children[1]);
         Assert.Equal("PropertyType2", node.TypeName);
         Assert.Equal("PropertyName", node.MemberName);
     }
@@ -74,27 +79,22 @@ @inject PropertyType2 PropertyName
     public void InjectDirectivePass_Execute_ExpandsTModel_WithDynamic()
     {
         // Arrange
-        var codeDocument = CreateDocument(@"
+        var codeDocument = ProjectEngine.CreateCodeDocument(@"
 @inject PropertyType<TModel> PropertyName
 ");
 
-        var engine = CreateEngine();
-        var pass = new InjectDirective.Pass()
-        {
-            Engine = engine,
-        };
-
-        var irDocument = CreateIRDocument(engine, codeDocument);
+        var processor = CreateCodeDocumentProcessor(codeDocument);
 
         // Act
-        pass.Execute(codeDocument, irDocument);
+        processor.ExecutePass<InjectDirective.Pass>();
 
         // Assert
-        var @class = FindClassNode(irDocument);
-        Assert.NotNull(@class);
-        Assert.Equal(2, @class.Children.Count);
+        var documentNode = processor.GetDocumentNode();
+        var classNode = documentNode.GetClassNode();
+
+        Assert.Equal(2, classNode.Children.Count);
 
-        var node = Assert.IsType<InjectIntermediateNode>(@class.Children[1]);
+        var node = Assert.IsType<InjectIntermediateNode>(classNode.Children[1]);
         Assert.Equal("PropertyType<dynamic>", node.TypeName);
         Assert.Equal("PropertyName", node.MemberName);
     }
@@ -103,28 +103,23 @@ @inject PropertyType<TModel> PropertyName
     public void InjectDirectivePass_Execute_ExpandsTModel_WithModelTypeFirst()
     {
         // Arrange
-        var codeDocument = CreateDocument(@"
+        var codeDocument = ProjectEngine.CreateCodeDocument(@"
 @model ModelType
 @inject PropertyType<TModel> PropertyName
 ");
 
-        var engine = CreateEngine();
-        var pass = new InjectDirective.Pass()
-        {
-            Engine = engine,
-        };
-
-        var irDocument = CreateIRDocument(engine, codeDocument);
+        var processor = CreateCodeDocumentProcessor(codeDocument);
 
         // Act
-        pass.Execute(codeDocument, irDocument);
+        processor.ExecutePass<InjectDirective.Pass>();
 
         // Assert
-        var @class = FindClassNode(irDocument);
-        Assert.NotNull(@class);
-        Assert.Equal(2, @class.Children.Count);
+        var documentNode = processor.GetDocumentNode();
+        var classNode = documentNode.GetClassNode();
+
+        Assert.Equal(2, classNode.Children.Count);
 
-        var node = Assert.IsType<InjectIntermediateNode>(@class.Children[1]);
+        var node = Assert.IsType<InjectIntermediateNode>(classNode.Children[1]);
         Assert.Equal("PropertyType<ModelType>", node.TypeName);
         Assert.Equal("PropertyName", node.MemberName);
     }
@@ -133,81 +128,24 @@ @inject PropertyType<TModel> PropertyName
     public void InjectDirectivePass_Execute_ExpandsTModel_WithModelType()
     {
         // Arrange
-        var codeDocument = CreateDocument(@"
+        var codeDocument = ProjectEngine.CreateCodeDocument(@"
 @inject PropertyType<TModel> PropertyName
 @model ModelType
 ");
 
-        var engine = CreateEngine();
-        var pass = new InjectDirective.Pass()
-        {
-            Engine = engine,
-        };
-
-        var irDocument = CreateIRDocument(engine, codeDocument);
+        var processor = CreateCodeDocumentProcessor(codeDocument);
 
         // Act
-        pass.Execute(codeDocument, irDocument);
+        processor.ExecutePass<InjectDirective.Pass>();
 
         // Assert
-        var @class = FindClassNode(irDocument);
-        Assert.NotNull(@class);
-        Assert.Equal(2, @class.Children.Count);
+        var documentNode = processor.GetDocumentNode();
+        var classNode = documentNode.GetClassNode();
+
+        Assert.Equal(2, classNode.Children.Count);
 
-        var node = Assert.IsType<InjectIntermediateNode>(@class.Children[1]);
+        var node = Assert.IsType<InjectIntermediateNode>(classNode.Children[1]);
         Assert.Equal("PropertyType<ModelType>", node.TypeName);
         Assert.Equal("PropertyName", node.MemberName);
     }
-
-    private RazorCodeDocument CreateDocument(string content)
-    {
-        var source = RazorSourceDocument.Create(content, "test.cshtml");
-        return RazorCodeDocument.Create(source);
-    }
-
-    private ClassDeclarationIntermediateNode FindClassNode(IntermediateNode node)
-    {
-        var visitor = new ClassNodeVisitor();
-        visitor.Visit(node);
-        return visitor.Node;
-    }
-
-    private RazorEngine CreateEngine()
-    {
-        var configuration = new RazorConfiguration(RazorLanguageVersion.Version_1_1, "test", Extensions: []);
-        return RazorProjectEngine.Create(configuration, RazorProjectFileSystem.Empty, b =>
-        {
-            // Notice we're not registering the InjectDirective.Pass here so we can run it on demand.
-            b.AddDirective(InjectDirective.Directive);
-            b.AddDirective(ModelDirective.Directive);
-
-            b.Features.Add(new RazorPageDocumentClassifierPass());
-            b.Features.Add(new MvcViewDocumentClassifierPass());
-        }).Engine;
-    }
-
-    private DocumentIntermediateNode CreateIRDocument(RazorEngine engine, RazorCodeDocument codeDocument)
-    {
-        foreach (var phase in engine.Phases)
-        {
-            phase.Execute(codeDocument);
-
-            if (phase is IRazorDocumentClassifierPhase)
-            {
-                break;
-            }
-        }
-
-        return codeDocument.GetDocumentIntermediateNode();
-    }
-
-    private class ClassNodeVisitor : IntermediateNodeWalker
-    {
-        public ClassDeclarationIntermediateNode Node { get; set; }
-
-        public override void VisitClassDeclaration(ClassDeclarationIntermediateNode node)
-        {
-            Node = node;
-        }
-    }
 }