Unexpected behavior with .NET's (9.0 specific) Regex engine when matching emoji.

[cheblin]

Hello,

I've noticed some unexpected behavior with .NET's (9.0 specific) Regex engine when matching emoji. I'm trying to find all characters that belong to the "Symbol, Other" (So) Unicode category.

When I run the following code, the result is 0, but I would expect it to be 1, since the hotdog emoji (🌭) is a symbol.

Console.WriteLine(new Regex(@"\p{So}").Matches("🌭").Count); 
// Output: 0

Could you explain why this happens and what the correct way to match such characters would be?

[JuergenAuer]

Hi @cheblin

your code requires a char-input.

But currently, Char.MaxValue = #FFFF.

https://learn.microsoft.com/en-us/dotnet/api/system.char.maxvalue?view=net-9.0

The Hot Dog has the Code point 127789, Hex #x1F32D. So the current NET implements only characters max #FFFF, so you must use two surrogate pairs to deal with chars not in the basic multilingual plane (chars greater #ffff).

See

https://learn.microsoft.com/en-us/dotnet/fundamentals/runtime-libraries/system-char

To determine the Unicode category of a character, call the GetUnicodeCategory method. For example, the following example calls the GetUnicodeCategory to display the Unicode category of each character in a string. The example works correctly only if there are no surrogate pairs in the String instance.

With Char, you use a surrogate pair, so that can't work.

Use the Rune struct.

https://learn.microsoft.com/en-us/dotnet/api/system.text.rune?view=net-9.0

[walid-khalafi]

The reason \p{So} doesn’t match 🌭 is that most emoji are outside the Basic Multilingual Plane and are represented as surrogate pairs in UTF‑16. The \p{So} category in .NET’s regex engine applies to individual UTF‑16 code units, not full Unicode scalar values, so neither half of the surrogate pair matches So on its own.
To match the whole emoji as a single unit, you can either:

• Use \X (grapheme cluster) to match complete Unicode characters, then filter by category if needed.
• Or enable the regex engine’s full Unicode mode and use a pattern that matches the scalar value, e.g.:

var regex = new Regex(@"\p{So}", RegexOptions.Compiled | RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);
Console.WriteLine(regex.Matches("🌭").Count); // 1

This ensures the engine evaluates the actual Unicode code point rather than each UTF‑16 code unit separately.
Let me know if you’d like me to share a minimal repro that demonstrates both approaches.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.