Add bidi support and address UAX31/UTS55 requirements (#884)

* Add bidi support and address UAX31/UTS55 requirements

Adds the bidi strong marks ALM, RLM, and LRM plus the bidi isolate controls LRI, RLI, FSI, and PDI to the syntax.

Formally defines optional vs. non-optional whitespace.

Non-optional whitespace must include at least one whitespace character. Optional whitespace may contain only bidi marks (which are invisible)

* Update syntax.md including text from previous PR

* Repair the guidance on strongly directional marks

Include ALM and better specify how to use the marks.

* Fix formatting of the "important"

* Add bidi characters to description of whitespace.

* Permit bidi in a few more places

Add optional whitespace at the start of `variant`

Add optional whitespace around `quoted-pattern`

These changes result in allowing bidi around keys and quoted patterns as intended.

* Update syntax.md ABNF

* Update formatting.md

- Add a note about the difference between formatting and message syntax.
- Clarify the sentence about message directionality.

* Address comment about name/identifier

* Address comments related to bidi in `name`

* Fix variable's location

* Address comment about the list of LRI/PDI targets

* One character typo :-P

* Update spec/syntax.md

Co-authored-by: Eemeli Aro <[email protected]>

* Address comments about rule R3a-1

* Update spec/syntax.md

Co-authored-by: Eemeli Aro <[email protected]>

* Address comment about U+061C

* Change [o]wsp => `o` or `s`

* Match syntax spec to abnf

* Remove *

* Update syntax.md

* Update spec/syntax.md

Co-authored-by: Eemeli Aro <[email protected]>

* Update spec/message.abnf

Co-authored-by: Eemeli Aro <[email protected]>

* Update spec/message.abnf

Co-authored-by: Eemeli Aro <[email protected]>

* Update syntax.md

* Update spec/message.abnf

Co-authored-by: Eemeli Aro <[email protected]>

* Update spec/syntax.md

Co-authored-by: Eemeli Aro <[email protected]>

* Update spec/syntax.md

Co-authored-by: Eemeli Aro <[email protected]>

---------

Co-authored-by: Eemeli Aro <[email protected]>

Author: aphillips

spec/formatting.md

@@ -768,7 +768,16 @@ That is, the text can can consist of a mixture of left-to-right and right-to-lef
 The display of bidirectional text is defined by the
 [Unicode Bidirectional Algorithm](http://www.unicode.org/reports/tr9/) [UAX9].
 
-The directionality of the message as a whole is provided by the _formatting context_.
+The directionality of the formatted _message_ as a whole is provided by the _formatting context_.
+
+> [!NOTE]
+> Keep in mind the difference between the formatted output of a _message_,
+> which is the topic of this section,
+> and the syntax of _message_ prior to formatting.
+> The processing of a _message_ depends on the logical sequence of Unicode code points,
+> not on the presentation of the _message_.
+> Affordances to allow users appropriate control over the appearance of the
+> _message_'s syntax have been provided.
 
 When a _message_ is formatted, _placeholders_ are replaced
 with their formatted representation.

spec/message.abnf

@@ -1,41 +1,41 @@
 message           = simple-message / complex-message
 
-simple-message    = [s] [simple-start pattern]
+simple-message    = o [simple-start pattern]
 simple-start      = simple-start-char / escaped-char / placeholder
 pattern           = *(text-char / escaped-char / placeholder)
 placeholder       = expression / markup
 
-complex-message   = [s] *(declaration [s]) complex-body [s]
+complex-message   = o *(declaration o) complex-body o
 declaration       = input-declaration / local-declaration
 complex-body      = quoted-pattern / matcher
 
-input-declaration = input [s] variable-expression
-local-declaration = local s variable [s] "=" [s] expression
+input-declaration = input o variable-expression
+local-declaration = local s variable o "=" o expression
 
-quoted-pattern    = "{{" pattern "}}"
+quoted-pattern    = o "{{" pattern "}}"
 
-matcher           = match-statement s variant *([s] variant)
+matcher           = match-statement s variant *(o variant)
 match-statement   = match 1*(s selector)
 selector          = variable
-variant           = key *(s key) [s] quoted-pattern
+variant           = key *(s key) quoted-pattern
 key               = literal / "*"
 
 ; Expressions
 expression          = literal-expression
                     / variable-expression
                     / function-expression
-literal-expression  = "{" [s] literal [s function] *(s attribute) [s] "}"
-variable-expression = "{" [s] variable [s function] *(s attribute) [s] "}"
-function-expression = "{" [s] function *(s attribute) [s] "}"
+literal-expression  = "{" o literal [s function] *(s attribute) o "}"
+variable-expression = "{" o variable [s function] *(s attribute) o "}"
+function-expression = "{" o function *(s attribute) o "}"
 
-markup = "{" [s] "#" identifier *(s option) *(s attribute) [s] ["/"] "}"  ; open and standalone
-       / "{" [s] "/" identifier *(s option) *(s attribute) [s] "}"  ; close
+markup = "{" o "#" identifier *(s option) *(s attribute) o ["/"] "}"  ; open and standalone
+       / "{" o "/" identifier *(s option) *(s attribute) o "}"  ; close
 
 ; Expression and literal parts
 function       = ":" identifier *(s option)
-option         = identifier [s] "=" [s] (literal / variable)
+option         = identifier o "=" o (literal / variable)
 
-attribute      = "@" identifier [[s] "=" [s] (literal / variable)]
+attribute      = "@" identifier [o "=" o (literal / variable)]
 
 variable       = "$" name
 
@@ -52,22 +52,22 @@ match = %s".match"
 
 ; Names and identifiers
 ; identifier matches https://www.w3.org/TR/REC-xml-names/#NT-QName
-; name matches https://www.w3.org/TR/REC-xml-names/#NT-NCName but excludes U+FFFD
+; name matches https://www.w3.org/TR/REC-xml-names/#NT-NCName but excludes U+FFFD and U+061C
 identifier = [namespace ":"] name
 namespace  = name
-name       = name-start *name-char
+name       = [bidi] name-start *name-char [bidi]
 name-start = ALPHA / "_"
            / %xC0-D6 / %xD8-F6 / %xF8-2FF
-           / %x370-37D / %x37F-1FFF / %x200C-200D
+           / %x370-37D / %x37F-61B / %x61D-1FFF / %x200C-200D
            / %x2070-218F / %x2C00-2FEF / %x3001-D7FF
            / %xF900-FDCF / %xFDF0-FFFC / %x10000-EFFFF
 name-char  = name-start / DIGIT / "-" / "."
            / %xB7 / %x300-36F / %x203F-2040
 
 ; Restrictions on characters in various contexts
 simple-start-char = content-char / "@" / "|"
-text-char         = content-char / s / "." / "@" / "|"
-quoted-char       = content-char / s / "." / "@" / "{" / "}"
+text-char         = content-char / ws / "." / "@" / "|"
+quoted-char       = content-char / ws / "." / "@" / "{" / "}"
 content-char      = %x01-08        ; omit NULL (%x00), HTAB (%x09) and LF (%x0A)
                   / %x0B-0C        ; omit CR (%x0D)
                   / %x0E-1F        ; omit SP (%x20)
@@ -83,5 +83,15 @@ content-char      = %x01-08        ; omit NULL (%x00), HTAB (%x09) and LF (%x0A)
 escaped-char = backslash ( backslash / "{" / "|" / "}" )
 backslash    = %x5C ; U+005C REVERSE SOLIDUS "\"
 
-; Whitespace
-s = 1*( SP / HTAB / CR / LF / %x3000 )
+; Required whitespace
+s = *bidi ws o
+
+; Optional whitespace
+o = *(ws / bidi)
+
+; Bidirectional marks and isolates
+; ALM / LRM / RLM / LRI, RLI, FSI & PDI
+bidi = %x061C / %x200E / %x200F / %x2066-2069
+
+; Whitespace characters
+ws = SP / HTAB / CR / LF / %x3000