Dynamic PGO Microbenchmark Regressions

[AndyAyersMS]

This issue tracks investigation into microbenchmarks that have reported regressions with Dynamic PGO enabled. It is a continuation of #84264 which tracked regressions from PGO before it was enabled.

The report below is collated from the following autofiling reports.

• [Perf] Windows/arm64: 67 Regressions on 5/19/2023 1:23:34 PM perf-autofiling-issues#17979
• [Perf] Linux/arm64: 62 Regressions on 5/19/2023 1:23:34 PM perf-autofiling-issues#17982
• [Perf] Windows/x64: 100 Regressions on 5/19/2023 3:32:16 PM perf-autofiling-issues#17994
• [Perf] Windows/arm64: 61 Regressions on 5/19/2023 1:23:34 PM perf-autofiling-issues#18096
• [Perf] Windows/arm64: 7 Regressions on 5/19/2023 9:46:56 PM perf-autofiling-issues#18097
• [Perf] Linux/arm64: 81 Regressions on 5/19/2023 1:23:34 PM perf-autofiling-issues#18103
• [Perf] Windows/arm64: 4 Regressions on 5/19/2023 1:23:34 PM perf-autofiling-issues#18109
• [Perf] Windows/arm64: 9 Regressions on 5/20/2023 3:57:43 PM perf-autofiling-issues#18111
• [Perf] Linux/x64: 100 Regressions on 5/19/2023 3:32:16 PM perf-autofiling-issues#18139
• [Perf] Windows/x64: 122 Regressions on 5/19/2023 3:32:16 PM perf-autofiling-issues#18151
• [Perf] Windows/x64: 55 Regressions on 5/19/2023 3:32:16 PM perf-autofiling-issues#18582

The table is auto generated by a tool written by @EgorBo but may be edited by hand as regression analysis produces results. The "Score" is the geomean regression across all architectures; benchmarks that did not regress (or get reported) on some architectures are assumed to have produced the same results with and without PGO. "Recent Score" is the current performance (as of 2023-0606) versus the non-PGO result; "Orig Score" is based on the results of auto filing. They will differ if benchmark performance has improved or regressed since the auto filing ran (see for example the results for System.Text.Json.Tests.Perf_Get.GetByte, which has improved already).

Only the 36 entries with recent scores >= 1.3 are included; this leaves off approximately 220 more rows with scores between 1.3 or lower. Our plan is to prioritize investigation of these benchmarks initially, as they have the largest aggregate regressions. If time permits, we will regenerate this chart to pick up the impact of any fixes and see how much of the remainder we can tackle.

Each arch/os result is a hyperlink to the performance data graph for that benchmark. ~Note we currently have no autofiling data for win-x64-intel. If/when that shows up we will regenerate the table.~~

[edit: had to regenerate the table once already, as the scoring logic was off]
[edit: have x64 win intel data now, new table. Not current results have shifted so table is somewhat different...]

cc @dotnet/jit-contrib

Notes	Recent Score	Orig Score	arm64-lin-ampere	arm64-win-surface	arm64-win-ampere	x64-lin-intel	x64-win-intel	x64-win-amd	Benchmark
noise	3.38	1.37						3.37 1.36	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: None)
noise	3.36	1.37						3.36 1.37	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: NonBacktracking)
notes	2.71	3.39		2.71 3.39					System.Memory.Span(Int32).EndsWith(Size: 4)
likely same as above	2.62	3.03	2.55 2.27	2.59 3.04					System.Memory.Span(Int32).SequenceEqual(Size: 4)
likely same as above	1.87	1.76		1.87 1.76					System.Memory.Span(Int32).SequenceCompareToDifferent(Size: 512)
(lack of) if conversion	1.82	1.80				1.67 1.63	1.93 1.92	1.86 1.85	System.Tests.Perf_Random.NextSingle
budget	1.75	1.88	1.33 1.47	1.35 1.49		1.90 1.99	2.29 2.43	2.10 2.19	System.Text.Json.Tests.Perf_Get.GetInt16
BDN	1.73	2.81	3.55 3.54	1.89 2.00		1.28 4.73	1.32 2.01	1.39 2.68	System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64EncodeInPlace(NumberOfBytes: 200000000)
notes	1.64	1.63	1.84 1.82	1.65 1.64					System.Tests.Perf_UInt32.TryParseHex(value: "0")
budget	1.61	1.70	1.27 1.44	1.28 1.46	1.24 1.18	2.09 2.17	2.25 2.33	1.86 1.94	System.Text.Json.Tests.Perf_Get.GetSByte
bimodal	1.61	1.59						1.60 1.58	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock Holmes", Options: Compiled)
cast expansion	1.60	1.64				1.82 1.87	1.41 1.43		System.Buffers.Tests.ReadOnlySequenceTests(Char).FirstSingleSegment
cast expansion	1.58	1.62				1.58 1.62			System.Buffers.Tests.ReadOnlySequenceTests(Byte).FirstSpanTenSegments
cast expansion	1.52	1.65				1.48 1.81	1.56 1.50		System.Buffers.Tests.ReadOnlySequenceTests(Byte).FirstSingleSegment
cast expansion	1.50	1.73				1.88 2.13	1.20 1.41		System.Buffers.Tests.ReadOnlySequenceTests(Char).FirstTenSegments
likely same as span cases above	1.48	1.28			1.48 1.28				System.Memory.Span(Int32).Reverse(Size: 4)
cast expansion	1.47	1.44				1.47 1.44			System.Buffers.Tests.ReadOnlySequenceTests(Byte).FirstSpanSingleSegment
notes	1.47	1.42						1.46 1.42	Benchstone.BenchF.InvMt.Test
unclear	1.46	1.15					1.46 1.15		MicroBenchmarks.Serializers.Json_FromStream(MyEventsListerViewModel).DataContractJsonSerializer_
fixed itself	1.45	1.09		1.45 1.09					System.Tests.Perf_Uri.EscapeDataString(input: "{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{
unclear	1.44	1.44				1.44 1.44			Burgers.Test1
unclear	1.43	1.27				1.43 1.27			System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfNumbers)
unclear, linux arm64 only	1.41	1.58	1.41 1.58						System.Text.Tests.Perf_StringBuilder.Append_Char_Capacity(length: 100000)
unclear, linux arm64 only	1.39	1.62	1.39 1.62						BenchmarksGame.RegexRedux_5.RunBench(options: Compiled)
bimodal	1.39	1.39	1.39 1.39						System.MathBenchmarks.Single.Min
bimodal	1.39	1.39	1.39 1.39						System.MathBenchmarks.Single.Max
unclear, linux arm64 only	1.39	1.32	1.39 1.32						System.IO.Pipes.Tests.Perf_NamedPipeStream.ReadWriteAsync(size: 1000000, Options: Asynchronous)
noise	1.38	1.29					1.38 1.29		System.IO.MemoryMappedFiles.Tests.Perf_MemoryMappedFile.CreateFromFile_Read(capacity: 10000000)
bimodal	1.37	1.37						1.37 1.37	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "zqj", Options: Compiled)
notes	1.37	1.36	1.26 1.29	1.42 1.43	1.24 1.28	1.60 1.48			System.Collections.Sort(IntStruct).Array(Size: 512)
budget	1.36	1.93	1.15 1.56	1.15 1.58	1.27 1.66	1.42 2.14	1.80 2.67	1.49 2.24	System.Text.Json.Tests.Perf_Get.GetByte
noise	1.35	1.31						1.36 1.33	System.Memory.Span(Char).IndexOfAnyTwoValues(Size: 512)
arm64 only; ldar vs dmb	1.35	1.36	1.35 1.34	1.38 1.40					System.Collections.CtorFromCollection(Int32).ConcurrentBag(Size: 512)
fixed by physical promotion	1.35	1.36		1.35 1.36					Devirtualization.EqualityComparer.ValueTupleCompareWrapped
budget	1.34	1.42	1.42 1.26	1.28 1.38	1.35 1.44	1.35 1.42	1.35 1.55	1.31 1.46	System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToStream(Mode: SourceGen)
notes	1.34	1.45	1.18 1.29	1.40 1.44	1.13 1.41	1.71 1.71			System.Collections.Sort(IntStruct).List(Size: 512)
notes	1.33	1.33						1.33 1.33	System.Tests.Perf_HashCode.Combine_1
inlining different; exposed local	1.33	1.32					1.34 1.33	1.32 1.32	System.Memory.ReadOnlySequence.Slice_Repeat(Segment: Multiple)
notes	1.33	1.18				1.33 1.18			System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfStrings)
budget	1.32	1.37	1.24 1.39	1.20 1.28	1.37 1.15	1.39 1.46	1.45 1.57	1.27 1.39	System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToWriter(Mode: SourceGen)
budget	1.32	1.39	1.37 1.28	1.22 1.42	1.34 1.31	1.32 1.38	1.30 1.50	1.34 1.38	System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToUtf8Bytes(Mode: SourceGen)
budget	1.31	1.88	1.15 1.59	1.18 1.62	1.03 1.37	1.49 2.22	1.66 2.49	1.49 2.24	System.Text.Json.Tests.Perf_Get.GetUInt16
budget	1.31	1.33	1.38 1.25	1.20 1.23	1.23 1.26	1.35 1.46	1.40 1.40	1.41 1.43	System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToString(Mode: SourceGen)
jcc errata	1.31	1.39					1.31 1.39		Span.Sorting.QuickSortSpan(Size: 512)
lack of cold inline exposes local	1.31	1.29					1.31 1.31	1.31 1.27	System.Memory.ReadOnlySequence.Slice_Start_And_Length(Segment: Multiple)
budget	1.31	1.39	1.32 1.19	1.20 1.50		1.31 1.37	1.40 1.50	1.31 1.34	System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeObjectProperty(Mode: SourceGen)
lack of ldapr	1.30	1.30	1.29 1.30	1.30 1.30					System.Collections.CtorFromCollection(String).ConcurrentBag(Size: 512)

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

---

This issue tracks investigation into microbenchmarks that have reported regressions with Dynamic PGO enabled. It is a continuation of #84264 which tracked regressions from PGO before it was enabled.

The report below is collated from the following autofiling reports.

• [Perf] Windows/arm64: 67 Regressions on 5/19/2023 1:23:34 PM perf-autofiling-issues#17979
• [Perf] Linux/arm64: 62 Regressions on 5/19/2023 1:23:34 PM perf-autofiling-issues#17982
• [Perf] Windows/x64: 100 Regressions on 5/19/2023 3:32:16 PM perf-autofiling-issues#17994
• [Perf] Windows/arm64: 61 Regressions on 5/19/2023 1:23:34 PM perf-autofiling-issues#18096
• [Perf] Windows/arm64: 7 Regressions on 5/19/2023 9:46:56 PM perf-autofiling-issues#18097
• [Perf] Linux/arm64: 81 Regressions on 5/19/2023 1:23:34 PM perf-autofiling-issues#18103
• [Perf] Windows/arm64: 4 Regressions on 5/19/2023 1:23:34 PM perf-autofiling-issues#18109
• [Perf] Windows/arm64: 9 Regressions on 5/20/2023 3:57:43 PM perf-autofiling-issues#18111
• [Perf] Linux/x64: 100 Regressions on 5/19/2023 3:32:16 PM perf-autofiling-issues#18139
• [Perf] Windows/x64: 122 Regressions on 5/19/2023 3:32:16 PM perf-autofiling-issues#18151

The table is auto generated by a tool written by @EgorBo but may be edited by hand as regression analysis produces results. The "Score" is the geomean regression across all architectures; benchmarks that did not regress (or get reported) on some architectures are assumed to have produced the same results with and without PGO. "Recent Score" is the current performance (as of 2023-0606) versus the non-PGO result; "Orig Score" is based on the results of auto filing. They will differ if benchmark performance has improved or regressed since the auto filing ran (see for example row 4's results for System.Text.Json.Tests.Perf_Get.GetUInt16, which has improved already).

Only the 54 entries with recent scores > 1.4 are included; this leaves off approximately 220 more rows with scores between 1.4 and 1.0. Our plan is to prioritize investigation of these benchmarks initially, as they have the largest aggregate regressions. If time permits we will regenerate this chart to pick up the impact of any fixes and see how much of the remainder we can tackle.

Each arch/os result is a hyperlink to the performance data graph for that benchmark. Note we currently have no autofiling data for win-x64-intel. If/when that shows up we will regenerate the table.

cc @dotnet/jit-contrib

Notes	Recent Score	Orig Score	win-x64-intel	win-x64-amd	lin-x64-intel	win-arm64-surface	win-arm64-ampere	lin-arm64-ampere	Benchmark
	3.66	3.66		2.68 2.68	4.60 4.73	2.00 2.00		3.54 3.54	System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64EncodeInPlace(NumberOfBytes: 200000000)
	2.80	2.97		1.86 1.94	2.09 2.17	1.28 1.46	1.22 1.18	1.27 1.44	System.Text.Json.Tests.Perf_Get.GetSByte
	2.79	2.83		1.90 1.85	3.84 3.83	1.30 1.27		2.45 2.70	System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64DecodeInPlace(NumberOfBytes: 200000000)
	2.65	3.41		1.48 2.24	1.49 2.22	1.17 1.62	1.19 1.37	1.17 1.59	System.Text.Json.Tests.Perf_Get.GetUInt16
	2.53	2.60		1.31 1.38	1.37 1.38	1.21 1.42	1.30 1.31	1.35 1.28	System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToUtf8Bytes(Mode: SourceGen)
	2.52	3.32		1.49 2.24	1.42 2.14	1.14 1.58	1.39 1.66	1.14 1.56	System.Text.Json.Tests.Perf_Get.GetByte
	2.50	2.56		1.33 1.39	1.37 1.46	1.21 1.28	1.16 1.15	1.36 1.39	System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToWriter(Mode: SourceGen)
	2.50	2.64		1.36 1.46	1.38 1.42	1.16 1.38	1.15 1.44	1.39 1.26	System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToStream(Mode: SourceGen)
	2.49	2.53		1.43 1.43	1.30 1.46	1.22 1.23	1.17 1.26	1.38 1.25	System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeToString(Mode: SourceGen)
	2.35	2.47		2.10 2.19	1.92 1.99	1.35 1.49		1.33 1.47	System.Text.Json.Tests.Perf_Get.GetInt16
	2.22	2.22		1.23 1.23		1.45 1.45	1.27 1.33	1.53 1.44	System.Tests.Perf_UInt64.TryParseHex(value: "0")
	2.10	2.10			1.48 1.48	1.44 1.43	1.35 1.28	1.25 1.29	System.Collections.Sort(IntStruct).Array(Size: 512)
	2.04	2.10		1.31 1.34	1.34 1.37	1.22 1.50		1.35 1.19	System.Text.Json.Serialization.Tests.WriteJson(ImmutableDictionary(String, String)).SerializeObjectProperty(Mode: SourceGen)
	2.04	2.18			1.63 1.71	1.42 1.44	1.03 1.41	1.19 1.29	System.Collections.Sort(IntStruct).List(Size: 512)
	1.97	1.99			1.33 1.39	1.26 1.22	1.12 1.19	1.23 1.24	System.Memory.Span(Int32).IndexOfValue(Size: 512)
	1.95	2.60		1.15 3.31	1.05 1.29	1.08 1.56		1.08 1.08	System.Text.Json.Tests.Perf_Get.GetInt32
	1.92	1.88				3.36 3.04		2.94 2.27	System.Memory.Span(Int32).SequenceEqual(Size: 4)
	1.92	2.04		1.19 1.43	1.15 1.23	1.20 1.30		1.17 1.27	System.Text.Json.Serialization.Tests.WriteJson(ImmutableSortedDictionary(String, String)).SerializeToWriter(Mode: SourceGen)
	1.87	1.97		0.99 1.11	1.14 1.50	1.31 1.22		1.09 1.10	System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: Url,&lorem ipsum=dolor sit amet,16)
	1.87	1.86		1.47 1.46	1.16 1.13	1.18 1.18		1.20 1.19	System.Collections.ContainsTrue(Int32).List(Size: 512)
	1.83	1.84		1.27 1.31	1.21 1.21	1.17 1.17		1.17 1.17	System.Collections.ContainsTrue(Int32).ICollection(Size: 512)
	1.80	1.89		1.05 1.38	1.30 1.30	1.20 1.21		1.19 1.19	System.Collections.ContainsTrue(Int32).Array(Size: 512)
	1.73	1.82		0.89 1.16		1.13 1.14	1.34 1.45	1.16 1.14	System.Memory.Span(Int32).LastIndexOfValue(Size: 512)
	1.72	2.66		0.84 3.31	0.93 1.48	0.84 1.56		0.85 1.07	System.Text.Json.Tests.Perf_Get.GetUInt32
	1.72	1.73		1.07 1.07	1.28 1.30	1.07 1.06		1.07 1.08	System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: Url,&lorem ipsum=dolor sit amet,16)
	1.71	1.76		1.08 1.18	1.11 1.15	1.10 1.14		1.10 1.10	System.Collections.IterateForEach(Int32).ImmutableHashSet(Size: 512)
	1.71	1.83		0.94 1.13	0.95 1.05	0.99 1.11		1.01 1.10	System.Text.Json.Tests.Perf_Get.GetInt64
	1.67	1.66			1.29 1.23	1.10 1.10		1.12 1.11	System.Memory.Span(Byte).IndexOfValue(Size: 512)
	1.66	1.66			1.34 1.33	1.08 1.06		1.08 1.08	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "(?:(?:25[0-5]
	1.64	1.65		1.15 1.19		1.08 1.10		1.11 1.13	System.Memory.Span(Byte).LastIndexOfValue(Size: 512)
	1.63	1.64			1.10 1.12	1.28 1.29		1.41 1.41	System.Tests.Perf_UInt64.TryParseHex(value: "3039")
	1.60	1.59			1.40 1.34	1.16 1.17		1.15 1.16	Benchstone.BenchI.BubbleSort2.Test
	1.60	1.59			1.33 1.30	1.16 1.16		1.18 1.16	System.Collections.ContainsTrue(Int32).ImmutableArray(Size: 512)
	1.58	1.58		1.85 1.85	1.62 1.63				System.Tests.Perf_Random.NextSingle
	1.57	1.59			1.25 1.35	1.15 1.14		1.13 1.13	System.Collections.ContainsTrue(Int32).Queue(Size: 512)
	1.56	1.67		0.99 1.19		0.98 1.15		1.00 1.13	System.Tests.Perf_Uri.EscapeDataString(input: "a{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{üa{ü
	1.56	1.63		1.17 1.31		1.48 1.46		1.28 1.40	System.Memory.Span(Char).LastIndexOfValue(Size: 512)
	1.55	1.55		1.47 1.45		1.21 1.21		1.21 1.22	System.Collections.ContainsTrue(Int32).Span(Size: 512)
	1.54	1.54		1.20 1.20		1.11 1.09		1.08 1.08	System.Memory.Span(Int32).IndexOfAnyFourValues(Size: 33)
	1.54	1.52		1.16 1.17		1.37 1.28		1.34 1.32	System.Tests.Perf_UInt32.TryParseHex(value: "3039")
	1.52	1.51		1.15 1.20		1.49 1.39			System.Memory.Span(Byte).SequenceCompareToDifferent(Size: 512)
	1.51	1.55		1.08 1.24		1.13 1.13		1.05 1.10	System.Text.Perf_Utf8Encoding.GetBytes(Input: Chinese)
	1.50	1.61				1.26 1.47		1.22 1.72	System.Tests.Perf_Guid.GuidToString
	1.48	1.49				1.20 1.19		1.18 1.24	System.Numerics.Tests.Perf_BigInteger.Subtract(arguments: 16,16 bits)
	1.47	1.49			1.30 1.35	1.03 1.08			System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: JavaScript,&Hello+(World)!,16)
	1.46	1.62		0.78 1.09		0.88 1.08		0.87 1.07	System.Text.Json.Tests.Perf_Get.GetString
	1.45	1.46				1.54 1.64		1.88 1.82	System.Tests.Perf_UInt32.TryParseHex(value: "0")
	1.45	1.48		1.27 1.41		1.06 1.06		1.09 1.09	System.Memory.Span(Int32).IndexOfAnyFiveValues(Size: 33)
	1.45	1.45			1.14 1.11			1.06 1.10	BenchmarksGame.BinaryTrees_5.RunBench
	1.45	1.45				3.41 3.39			System.Memory.Span(Int32).EndsWith(Size: 4)
	1.43	1.43				1.05 1.05		1.06 1.06	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock
	1.42	1.45				0.99 1.08		1.06 1.10	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Leipzig.Count(Pattern: "[a-z]shing", Options: Compiled)
	1.40	1.41		1.22 1.22	1.24 1.32				LinqBenchmarks.Count00ForX
	1.40	1.40			1.41 1.38	1.09 1.13			System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: Url,&lorem ipsum=dolor sit amet,512)

Author:	AndyAyersMS
Assignees:	AndyAyersMS
Labels:	area-CodeGen-coreclr
Milestone:	-

[AndyAyersMS]

System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64EncodeInPlace(NumberOfBytes: 200000000)

System.Buffers.Text.Tests.Base64EncodeDecodeInPlaceTests.Base64DecodeInPlace(NumberOfBytes: 200000000)

----

These benchmarks were analyzed before PGO was enabled: #84264 (comment)

BDN's strategy doesn't run the benchmark enough, because each iteration is long running, and so (since the key benchmark methods are R2R'd) the test ends up measuring tier1-instr code.

[AndyAyersMS]

System.Text.Json.Tests.Perf_Get.GetInt16

System.Text.Json.Tests.Perf_Get.GetSByte

System.Text.Json.Tests.Perf_Get.GetByte

---

Likely the analysis from #84264 (comment) is still relevant and explains the related tests regressions as well: we run out of inlining budget, in part because the benchmark method is small and there are quite a few large aggressive inline methods, and so we're unable to do some key inlines.

Tracking issue for this is #85531.

Some of these improved with #86551.

[AndyAyersMS]

System.Tests.Perf_UInt32.TryParseHex(value: "0")

---

This regresses across the board, so not sure why we don't have more autofiling for it.

I can't repro on win-x64

Method	Job	Toolchain	value	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
TryParseHex	Job-EEXSZV	\base\corerun.exe	0	7.860 ns	0.1366 ns	0.1573 ns	7.814 ns	7.640 ns	8.281 ns	1.00	0.00	-	NA
TryParseHex	Job-EZIDSX	\diff\corerun.exe	0	7.834 ns	0.1461 ns	0.1683 ns	7.783 ns	7.634 ns	8.344 ns	1.00	0.02	-	NA

But I can (perhaps) on win-arm64 (volterra)

Method	Job	Toolchain	value	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
TryParseHex	Job-DFURZJ	\base\corerun.exe	0	3.265 ns	0.0267 ns	0.0237 ns	3.257 ns	3.235 ns	3.316 ns	1.00	0.00	-	NA
TryParseHex	Job-URZCKJ	\diff\corerun.exe	0	3.876 ns	0.0899 ns	0.0751 ns	3.849 ns	3.794 ns	4.034 ns	1.19	0.02	-	NA

base

00.81%   4.1E+05     ?        Unknown
48.82%   2.458E+07   Tier-1   [491fed18-8a14-46f6-b30c-6320dc770919]Runnable_0.WorkloadActionUnroll(int64)
23.52%   1.184E+07   Tier-1   [System.Private.CoreLib]Number.TryParseBinaryIntegerHexOrBinaryNumberStyle(value class System.ReadOnlySpan`1<!!0>,value class System.Globalization.NumberStyles,!!1&)
12.69%   6.39E+06    Tier-1   [MicroBenchmarks]Perf_UInt32.TryParseHex(class System.String)
08.92%   4.49E+06    Tier-1   [System.Private.CoreLib]NumberFormatInfo.<GetInstance>g__GetProviderNonNull|58_0(class System.IFormatProvider)
04.37%   2.2E+06     Tier-1   [System.Private.CoreLib]CastHelpers.IsInstanceOfClass(void*,class System.Object)

diff

50.62%   2.666E+07   Tier-1   [d735f3a6-719a-4885-9631-16a7aeff132c]Runnable_0.WorkloadActionUnroll(int64)
23.94%   1.261E+07   Tier-1   [System.Private.CoreLib]Number.TryParseBinaryIntegerHexOrBinaryNumberStyle(value class System.ReadOnlySpan`1<!!0>,value class System.Globalization.NumberStyles,!!1&)
10.73%   5.65E+06    Tier-1   [MicroBenchmarks]Perf_UInt32.TryParseHex(class System.String)
08.62%   4.54E+06    Tier-1   [System.Private.CoreLib]NumberFormatInfo.<GetInstance>g__GetProviderNonNull|58_0(class System.IFormatProvider)
04.08%   2.15E+06    Tier-1   [System.Private.CoreLib]CastHelpers.IsInstanceOfClass(void*,class System.Object)

Note the very high overhead. WorkoadActionUnroll should have the AggressiveOptimization attribute so its codegen does not vary.

PGO codegen for TryParseBinaryIntegerHexOrBinaryNumberStyle looks good, all the hot code is adjacent and the method is straight line code. So not clear why it is ~7% or so slower.

Path lengths are similar but PGO does one extra STP/LDP and a few more register arg moves. So perhaps that's the explanation?

[AndyAyersMS]

System.Memory.Span<Int32>.EndsWith(Size: 4)

System.Memory.Span.SequenceEqual(Size: 4)

---

These two appear to be arm64 specific.

---

Method	Job	Toolchain	Size	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
EndsWith	Job-INARIQ	\base-rel\corerun.exe	4	2.712 ns	0.0209 ns	0.0163 ns	2.712 ns	2.680 ns	2.734 ns	1.00	0.00	-	NA
EndsWith	Job-YXKXJS	\diff-rel\corerun.exe	4	3.357 ns	0.2569 ns	0.2959 ns	3.175 ns	3.079 ns	3.928 ns	1.23	0.10	-	NA

base
01.52%   5.7E+05     ?        Unknown
35.26%   1.319E+07   Tier-1   [18850c97-dca6-4aad-8393-0715e0b95a4a]Runnable_0.WorkloadActionUnroll(int64)
34.51%   1.291E+07   Tier-1   [MicroBenchmarks]System.Memory.Span`1[System.Int32].EndsWith()
27.72%   1.037E+07   Tier-1   [System.Private.CoreLib]SpanHelpers.SequenceEqual(unsigned int8&,unsigned int8&,unsigned int)

diff
01.29%   5.1E+05     ?        Unknown
35.20%   1.392E+07   Tier-1   [MicroBenchmarks]System.Memory.Span`1[System.Int32].EndsWith()
31.40%   1.242E+07   Tier-1   [72707307-0596-4e84-8515-1a5c07b80d3a]Runnable_0.WorkloadActionUnroll(int64)
31.10%   1.23E+07    Tier-1   [System.Private.CoreLib]SpanHelpers.SequenceEqual(unsigned int8&,unsigned int8&,unsigned int)

So issue appears to be in SequenceEqual ?

Seems like PGO and non-PGO codegen is the same. Method is R2R'd and with PGO we do a tier1 instr, but do not add any probes. Explanation: this method is an intrinsic and not on the whitelist.

Fixing that (hack) gives:

Method	Job	Toolchain	Size	Mean	Error	StdDev	Median	Min	Max	Ratio	Allocated	Alloc Ratio
EndsWith	Job-FBJVVQ	\base-rel\corerun.exe	4	2.732 ns	0.0330 ns	0.0275 ns	2.729 ns	2.692 ns	2.793 ns	1.00	-	NA
EndsWith	Job-UKVWAD	\diff-rel\corerun.exe	4	3.146 ns	0.0472 ns	0.0419 ns	3.153 ns	3.084 ns	3.208 ns	1.15	-	NA
EndsWith	Job-BJUBBV	\hack-rel\corerun.exe	4	2.660 ns	0.0364 ns	0.0322 ns	2.659 ns	2.610 ns	2.731 ns	0.97	-	NA

and more broady

Method	Job	Toolchain	Size	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
EndsWith	Job-SDVJLL	\base-rel\corerun.exe	4	2.685 ns	0.0479 ns	0.0448 ns	2.679 ns	2.627 ns	2.766 ns	1.00	0.00	-	NA
EndsWith	Job-FTSDAY	\diff-rel\corerun.exe	4	3.151 ns	0.0178 ns	0.0158 ns	3.151 ns	3.126 ns	3.181 ns	1.17	0.02	-	NA
EndsWith	Job-EGIDWC	\hack-rel\corerun.exe	4	3.125 ns	0.0394 ns	0.0368 ns	3.125 ns	3.080 ns	3.180 ns	1.16	0.02	-	NA
EndsWith	Job-SDVJLL	\base-rel\corerun.exe	33	4.739 ns	0.0558 ns	0.0495 ns	4.728 ns	4.686 ns	4.840 ns	1.00	0.00	-	NA
EndsWith	Job-FTSDAY	\diff-rel\corerun.exe	33	5.322 ns	0.0576 ns	0.0539 ns	5.299 ns	5.257 ns	5.446 ns	1.12	0.01	-	NA
EndsWith	Job-EGIDWC	\hack-rel\corerun.exe	33	5.369 ns	0.0265 ns	0.0235 ns	5.369 ns	5.333 ns	5.419 ns	1.13	0.01	-	NA
EndsWith	Job-SDVJLL	\base-rel\corerun.exe	512	32.020 ns	0.0209 ns	0.0175 ns	32.023 ns	31.999 ns	32.059 ns	1.00	0.00	-	NA
EndsWith	Job-FTSDAY	\diff-rel\corerun.exe	512	32.422 ns	0.0351 ns	0.0328 ns	32.417 ns	32.383 ns	32.495 ns	1.01	0.00	-	NA
EndsWith	Job-EGIDWC	\hack-rel\corerun.exe	512	31.819 ns	0.0370 ns	0.0346 ns	31.815 ns	31.776 ns	31.893 ns	0.99	0.00	-	NA

But does not explain why there is a PGO regression. And as you can see the Size=4 results are not very stable.

Similar diffs for EndsWith (better layout, slightly higher prolog/epilog costs).

[AndyAyersMS]

System.Memory.Span<Int32>.SequenceCompareToDifferent(Size: 512)

---

Also seems to be arm64 specific.