Escaping of non-ascii characters in entry XML ID

[1313ou]

The escape_lemma(lemma) function, whose purpose is to format the lemma so it is valid XML id, is flawed when it comes to escaping non-ascii characters.

It converts any such characters to '-%04x-' % ord(c), which used 4 times withe the current data:
'oewn-Se-00f1-or-n',
'oewn-Se-00f1-ora-n',
'oewn-Se-00f1-orita-n',
'oewn-Capital-003a-_Critique_of_Political_Economy-n',

Decoding would involve the reverse process of converting any r'-[0-9A-Fa-f]{4}-' back to the character.

The snag is such sequences as

-abbe-
-abed-
-face-
-baba-
-bead-
-beef-
-cafe-
-caff-
-dada-
-dead-
-deaf-
-deed-
-fade-
-feed-

also match, qualifying as valid hex sequences (in addition to any four-digit like -1000-).

These sequences will be found in:

oewn-1000-a
oewn-1000-n
oewn-1728-n
oewn-2019-nCoV_acute_respiratory_disease-n
oewn-Adad-n
oewn-Bade-n
oewn-Beda-n
oewn-Bede-n
oewn-Daba-n
oewn-Edda-n
oewn-Rain-in-the-Face-n
oewn-abbe-n
oewn-abed-r
oewn-about-face-n
oewn-about-face-v
oewn-baba-n
oewn-babe-n
oewn-bead-n
oewn-bead-v
oewn-beef-n
oewn-beef-v
oewn-cafe-n
oewn-caff-n
oewn-cede-v
oewn-dace-n
oewn-dada-n
oewn-dead-a
oewn-dead-air_space-n
oewn-dead-burned_lime-n
oewn-dead-end-a
oewn-dead-end_street-n
oewn-dead-man-ap-s-fingers-n
oewn-dead-man-ap-s_float-n
oewn-dead-men-ap-s-fingers-n
oewn-dead-n
oewn-dead-on-a
oewn-dead-r
oewn-deaf-a
oewn-deaf-aid-n
oewn-deaf-and-dumb-a
oewn-deaf-and-dumb_person-n
oewn-deaf-mute-a
oewn-deaf-mute-n
oewn-deaf-muteness-n
oewn-deaf-mutism-n
oewn-deaf-n
oewn-deaf-v
oewn-deed-n
oewn-drop-dead-r
oewn-edda-n
oewn-face-amount_certificate_company-n
oewn-face-harden-v
oewn-face-lift-v
oewn-face-n
oewn-face-off-n
oewn-face-saving-a
oewn-face-to-face-a
oewn-face-to-face-r
oewn-face-v
oewn-fade-n
oewn-fade-v
oewn-feed-n
oewn-feed-v
oewn-force-feed-v
oewn-full-face-a
oewn-in-your-face-a
oewn-lie-abed-n
oewn-pousse-cafe-n
oewn-pudding-face-n
oewn-sick-abed-a
oewn-stone-dead-a
oewn-stone-deaf-a
oewn-stone-face-n
oewn-tone-deaf-a
oewn-volte-face-n

thus making decoding hazardous (because it's impossible to tell the string 'face' from the hex 'face').

Added to that, the '-de.*-' sequences will result in unicode surrogate characters reserved for coding and raising an error when printed.

Te good news is that unicode letters can be be part of an XML ID

Here are regular expressions for valid NameStartChar and NameChar based on the XML 1.0 specification:

name_start_char_re = re.compile(r'^[A-Z_a-z\xC0-\xD6\xD8-\xF6\xF8-\u02FF\u0370-\u037D\u037F-\u1FFF'
                                r'\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF'
                                r'\uF900-\uFDCF\uFDF0-\uFFFD]$')

name_char_re = re.compile(r'^[A-Z_a-z0-9\x2D\x2E\xB7\xC0-\xD6\xD8-\xF6\xF8-\u02FF'
                          r'\u0300-\u036F\u203F-\u2040\u0370-\u037D\u037F-\u1FFF'
                          r'\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF'
                          r'\uF900-\uFDCF\uFDF0-\uFFFD]$')

With this in mind,

'é' is valid in XML ID (including as start character)
'ñ' is valid in XML ID (including as start character)
'汉' is valid in XML ID (including as start character)

That settles the problem for 3 cases out of 4: señor, señora, señorita. Only remaining problem, the colon, currently used only in 'Capital: Critique of Political Economy'
Why not use '-cn' (-cl- is not available being used by cl for centilitre) which would yield

oewn-Capital-cn-_Critique_of_Political_Economy-n

instead of

oewn-Capital-003a-_Critique_of_Political_Economy-n

Risky if a 'cn' is later introduced, abbreviation for China for instance. Personnally I would not accept colons within lemma which has been generating problems from the start, only for a single entry. Besides 'Capital: Critique of Political_Economy' can hardly be argued to be a lemma or a dictionary entry (possibly an encyclopedia entry).

This affects only XML so there is nothing to fix but code because the XML is just derived, not source. However tools that work from XML will have to be reviewed if they try to unescape lemmas in entry ids and work with XML not generated with fixed code.

While correcting, I suggest replacing

    elif c == '-':
            return '-'

which is a NO-OP, with

     elif c == ':':
          return '-cn-'

and change

    else:
        return '-%04x-' % ord(c)

to

    elif name_char_re.match(c) or name_char_re.match(c):
        return c
    raise ValueError(f'Illegal character {c}')

with the regexprs above

[jmccrae]

I think this is a good point and I fully agree. The implementation you propose is good and so please go ahead and make the PR (but check that it is compatible with #1096)

However, I don't think we should assume that IDs should be invertible and unique. For example if we were to introduce accented forms of word, e.g., 'protégé', I think we should at least consider whether we just combine with the existing identifier oewn-protege-n rather than introduced some identifier like oewn-prot-eac-g-eac--n. Of course, this assumes that accents are not meaningful in English ('resumé'/'resume' being maybe an exception but they are also different parts-of-speech).

[jmccrae]

Actually, I see we do have some accented entries already:

oewn-Se-00f1-or-n
oewn-Se-00f1-ora-n
oewn-Se-00f1-orita-n

So maybe we can fix some accented characters as well?

[oewntk]

I don't think we should assume that IDs should be invertible and unique
I agree they don't have to be invertible. However XML IDs have to be unique otherwise the document won't validate. The issue is that the current generation of IDs might lead to collisions and duplicates.
I suspect what you had in mind was more the definition of a lexical entry: is 'protege' a single entry with different forms (protégé, protege) or two different entries with same synset membership (the current scheme in PWN31, it seems)?

[jmccrae]

When I think about it more, I think that we should probably have two lexical entries for 'protege' and 'protégé' so they both appear in the UI (https://en-word.net). Alternative forms are not shown in the interface normally. If XML IDs can support most accented characters as @1313ou says then there is no problem with having nice IDs as well.

[goodmami]

This discussion is related to globalwordnet/schemas#55, right? If so, you can compare to what we do for OMW: https://github.com/omwn/omw-data/blob/main/scripts/util.py

The basic strategy is to allow anything that is allowed in XML IDs, then to use HTML4 entity names with some additional names in a custom mapping. Finally, since - acts as the escaping operator, literal hyphens use the double-escape strategy: --.

>>> import util
>>> util.escape_lemma("Capital: Critique of Political Economy")
'Capital-colon-_Critique_of_Political_Economy'
>>> util.escape_lemma("protégé")
'protégé'
>>> util.escape_lemma("ex-president")
'ex--president'

[1313ou]

Much better to have an escape character instead of a -xy- sequence, with xy thought to be unlikely in the input.

Changing would affect
1310 -ap- sequences (apostrophe)
22 -sl- (slash)
2 -pl- (plus)
1 -ex- (exclamation)

It would also remove the confusion between
-ex- and ex-
-cm- and cm (for centimeter)
-lb- and lb (pound)
-pl- and PL for Poland
-cn- and CN for China

[jmccrae]

@1313ou is this issue closed or do you wish to implement @goodmami's suggestion?

It seems like a more standard and simple solution to me.

[1313ou]

You can close the issue as it was originally about non-ascii characters.

But in the discussion , the issue was also raised about escaping ASCII characters that can't be found in an XML-ID (comma, colon, brackets, exclamation, question ...).

I can implement @goodmami's suggestion (in fact copy his code, though I suspect validation would also have to be tweaked) if you give me some time and if you give the greenlight to changing some 1350 IDs for entries only.

Does it affect both entry IDs (escaped lemmas) and sense IDs (escaped sensekeys) ?

[jmccrae]

Yeah, I don't think it is a problem to change the entry IDs.

This probably doesn't need to affect the sense keys as they do not need to be valid XML ids.

[1313ou]

The sense IDs are mapped/derived from sensekeys through map_sense_key() in wordnet.py. This is to get an acceptable XML ID for senses. What I was meaning is : Do we have to escape them the same way, which does make sense. This affects XML only. Sensekeys in YAML are unaffected.

[1313ou]

This would yield:

protégé -> protégé
señor -> señor
a.b.c -> a.b.c
a-b-c -> a--b--c
a_b_c -> a-lowbar-b-lowbar-c
a b c -> a_b_c
a:b:c -> a-colon-b-colon-c
a/b/c -> a-sol-b-sol-c
a|b|c -> a-vert-b-vert-c
a\b\c -> a-bsol-b-bsol-c
a,b,c -> a-comma-b-comma-c
a;b;c -> a-semi-b-semi-c
a=b=c -> a-equals-b-equals-c
a+b+c -> a-plus-b-plus-c
a(b)c -> a-lpar-b-rpar-c
a{b}c -> a-lbrace-b-rbrace-c
a[b]c -> a-lsqb-b-rsqb-c
a!b!c -> a-excl-b-excl-c
a?b?c -> a-quest-b-quest-c
a$b$c -> a-dollar-b-dollar-c
a#b#c -> a-num-b-num-c
a%b%c -> a-percnt-b-percnt-c
a@b@c -> a-commat-b-commat-c
a^b^c -> a-Hat-b-Hat-c
a*b*c -> a-ast-b-ast-c
a'b'c -> a-apos-b-apos-c
a`b`c -> a-grave-b-grave-c
a´b´c -> a-acute-b-acute-c
a‘quoted’b -> a-lsquo-quoted-rsquo-b
a→b←c -> ERROR '→' [2192] is illegal character in XML ID and no escape sequence is defined