[doc] Add info about Eggex bytes and code points

- doc/ref: Add re-chars section for Eggex
- doc/eggex: more details

Author: Andy C

doc/eggex.md

@@ -182,6 +182,9 @@ We've reserved syntactic space for PCRE and Python variants:
 - lazy/non-greedy: `x{L +}`, `x{L 3,4}`
 - possessive: `x{P +}`, `x{P 3,4}`
 
+(Oils doesn't have these features, because Oils translates Eggex to POSIX ERE
+syntax.)
+
 ### Negation Consistently Uses !
 
 You can negate named char classes:
@@ -258,13 +261,12 @@ You can also add type conversion functions:
 
 Example:
 
-    [ a-f 'A'-'F' \yFF \u{03bc} \n \\ \' \" \0 ]
-
-Terms:
-
-- Ranges: `a-f` or `'A' - 'F'`
-- Literals: `\n`, `\y01`, `\u{3bc}`, etc.
-- Sets specified as strings: `'abc'`
+    [ a b c ]                     # individual characters / code points
+    [ '?' '*' '+' ]               # quoted characters
+    [ 'a' 'bc' '?*+' ]            # reduce the number of quotes
+    [ a-f 'A'-'F' x y ]           # ranges of code points
+    [ \n \\ \' \" ]               # backslash escapes
+    [ \yFF \u{03bc} ]             # a byte and a code point
 
 Only letters, numbers, and the underscore may be unquoted:
 
@@ -409,6 +411,17 @@ Yes:
 
 ## POSIX ERE Limitations
 
+In theory, Eggex can be translated to many different synatxes, like Perl or
+Python.
+
+In practice, Oils translates Eggex to POSIX Extended Regular Expressions (ERE)
+syntax.  This syntax is understood by `libc`, e.g. GNU libc or musl libc.
+
+But not all of Eggex can be translated to ERE syntax.  And the translation is
+straightforward and "dumb", rather than smart.
+
+Here are some limitations.
+
 ### Repetition of Strings Requires Grouping
 
 Repetitions like `* + ?` apply only to the last character, so literal strings
@@ -425,41 +438,15 @@ Yes:
 
 Also OK:
 
-    ('foo')+  # this is a CAPTURING group in ERE
+    ('foo')+  # but note this is a CAPTURING group in ERE
 
 This is necessary because ERE doesn't have non-capturing groups like Perl's
 `(?:...)`, and Eggex only does "dumb" translations.  It doesn't silently insert
 constructs that change the meaning of the pattern.
 
-### Unicode char literals are limited in range
-
-ERE can't represent this set of 1 character reliably:
-
-    / [ \u{0100} ] /      # This char is 2 bytes encoded in UTF-8
-
-These sets are accepted:
-
-    / [ \u{1} \u{2} ] /   # set of 2 chars
-    / [ \y01 \y02 ] ] /   # set of 2 bytes
-
-They happen to be identical when translated to ERE, but may not be when
-translated to PCRE.
-
-### Don't put non-ASCII bytes in string sets in char classes
-
-<!-- TODO: $'' will be disallowed in YSH -->
-
-This is a sequence of characters:
-
-    / $'\xfe\xff' /
-
-This is a **set** of characters that is illegal:
-
-    / [ $'\xfe\xff' ] /  # set or sequence?  It's confusing
-
-This is a better way to write it:
+### Bytes and Code Points are Limited in Range
 
-    / [ \yfe \yff ] /  # set of 2 chars
+See [re-chars](ref/chap-expr-lang.html#re-chars) in the Oils Reference.
 
 ### Char class literals: `^ - ] \`
 

doc/ref/chap-expr-lang.md

@@ -735,36 +735,66 @@ splicing of strings:
     var dq = "hi $name"    
     var eggex = / @dq /
 
+
 ### class-literal
 
-An eggex character class literal specifies a set.  It can have individual
-characters and ranges:
+An eggex *character class literal* specifies a **set** of code points.  It's
+enclosed in brackets:
+
+    var vowels = / [a e i o u] /  # A set of 5 vowels
+
+A class literal can have individual code points:
 
-    [ 'x' 'y' 'z' a-f A-F 0-9 ]  # 3 chars, 3 ranges
+    [ a e i o u '?' '*' '+' ]
 
-Omit quotes on ASCII characters:
+It can also have ranges of code points, denoted with a hyphen:
 
-    [ x y z ]  # avoid typing 'x' 'y' 'z'
+    [ a-f A-F 0-9 ]
 
-Sets of characters can be written as strings
+To reduce the number of quotes, you can write a set of characters as a string:
 
-    [ 'xyz' ]  # any of 3 chars, not a sequence of 3 chars
+    [ 'xyz' ]  # any of 3 chars, NOT a sequence of 3 chars
 
-Backslash escapes are respected:
+You can also use backslash escapes:
 
     [ \\ \' \" \0 ]
-    [ \xFF \u{3bc} ]
+    [ \y7F \u{3bc} ]     # a byte and a code point
 
-(Note that we don't use `\yFF`, as in J8 strings.)
+    [ \y01 - \y7F ]      # range of bytes
+    [ \u{1} - \u{7F} ]   # range of code points
 
-Splicing:
+The `@` operator lets you refer to string variables:
 
+    var str_var = 'xyz'
     [ @str_var ]
 
 Negation always uses `!`
 
     ![ a-f A-F 'xyz' @str_var ]
 
+### re-chars
+
+Oils usually invokes `libc` in UTF-8 mode.  In this mode, the regex engine
+can't match bytes like `0xFF`; it can only match code points.
+
+    var x = / [ \y7F \u{3bc} ] /     # a byte and a code point
+
+Oils translates Eggex to POSIX extended regex (ERE) syntax.  Here are some
+restrictions when translating bytes and code points to ERE:
+
+- The `NUL` byte `\y00` isn't allowed.
+  - Its synonym, code point zero `\u{0}`, also isn't allowed.
+- Bytes `\y80` to `\yFF` aren't allowed, because they're outside the ASCII
+  range.
+
+Reminders:
+
+- In the ASCII range, bytes and code points are the same
+  - That is, `\y01` to `\y7F` are synonyms for `\u{1}` to `\u{7F}`.
+- Outside of the ASCII range, they are different, so Eggex disallows them.
+  - For example, `\u{FF}` is a code point, and `\yFF` is a byte, but they are
+    not the same.
+
 ### named-class
 
 Perl-like shortcuts for sets of characters:

doc/ref/toc-ysh.md

@@ -314,7 +314,8 @@ X [External Lang] BEGIN   END   when (awk)
                   match-ops     ~   !~   ~~   !~~
   [Eggex]         re-literal    / d+ ; re-flags ; ERE /
                   re-primitive  %zero    'sq'
-                  class-literal [c a-z 'abc' @str_var \\ \xFF \u{3bc}]
+                  class-literal [c a-z 'abc' @str_var \\ \y01 \u{3bc}]
+                  re-chars      \y01 \u{3bc}
                   named-class    dot   digit   space   word   d  s  w
                   re-repeat     d?   d*   d+   d{3}   d{2,4}
                   re-compound    seq1 seq2   alt1|alt2   (expr1 expr2)