Support for script in addition to language

[kba]

With the @LANG attribute, ISO 639-3 codes can be used to express the language of the String, TextLine, TextBlock, and at some point PAGE (#55). But what about the script the text is written in? With a @SCRIPT attribute in all places where @LANG is allowed, one could express the script e.,g. as a 4-letter ISO 15924 code. PAGE-XML has support for both, roughly representing ISO 639 and ISO 15924.

[mittagessen]

ISO 15924 is a horrible mess and not a good place to start. The 'granularity' of the descriptors varies widely and their semantics are unclear, e.g. there exist separate identifiers for the different styles of Syriac but all Arabic scripts share one identifier. Free text fields might be better in the absence of a good standard.

[artunit]

As per the 2021-04-29 Board Meeting, I am adding a link to BCP 47, which might be one possible approach for script identification, based on positive experience at Google.

[cipriandinu]

I think that xsd:language can cover also the script topic. According with xsd:language definition, this is the set of language codes defined by RFC 1766. RFC 1766 (https://www.rfc-editor.org/rfc/pdfrfc/rfc1766.txt.pdf) defines the language as:

Language-Tag = Primary-tag ( "-" Subtag )
Primary-tag = 18ALPHA
Subtag = 1*8ALPHA

where first subtag is:

• All 2-letter codes are interpreted as ISO 3166 alpha-2
  country codes denoting the area in which the language is
  used.
• Codes of 3 to 8 letters may be registered with the IANA by
  anyone who feels a need for it, according to the rules in
  chapter 5 of this document.
  The information in the subtag may for instance be:
• Country identification, such as en-US (this usage is
  described in ISO 639)
• Dialect or variant information, such as no-nynorsk or en-
  cockney
• Languages not listed in ISO 639 that are not variants of
  any listed language, which can be registered with the i-
  prefix, such as i-cherokee
  - Script variations, such as az-arabic and az-cyrillic

The list of registered codes cand be found here:

https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry

Using this is possible to encode also the scripts - there are defined about 212 possible scripts that can be used as subtag

Changing to a different standard probably will create back-compatibility issues.

[cipriandinu]

After discussion on ALTO board, I propose to close this item, based on the previous comment. @kba please review and give us your feedback. I will set the topic for voting (ACCEPT means agree to close the item)

[kba]

You're right, xsd:language is flexible enough to express both. We will use the RFC 1766 mechanics for language/script. That should also support subtyping Arabic as @mittagessen wanted AFAICT.

[kba]

ACCEPT

[cipriandinu]

ACCEPT

[cneud]

ACCEPT

[ntra00]

ACCEPT

[Haighton]

ACCEPT

[acpopat]

ACCEPT

[cipriandinu]

@kba it looks like in fact I was a bit outdated, the last definition of xsd:language refers BCP47, that replaces older standards. But from ALTO perspective we use xsd:language and this alows to cover any specific script as well (this was the original topic) - either using BCP47 or much older RFC1776 mechanism.

[cowboyMontana]

ACCEPT

[JLoitzenbauer-CRKN]

ACCEPT

[callylaw]

ACCEPT

[hanyelsawy]

ACCEPT

[Haighton]

ACCEPT

[c-sebastien]

ACCEPT

[jukervin]

ACCEPT

[ntra00]

ACCEPT

[cipriandinu]

Closed based on existing votes