Semantic Comments

[zbraniecki]

One of the features we're debating for Fluent is a concept of Semantic Comments.

The idea is to define a syntax for comments that can be easily read in a "text-like" mode, but can also gain semantic meaning that can be additionally interpreted by the tools. Similar to how MarkDown file works maybe.

Generally speaking, semantic comments are meant to not have any meaning at runtime, thus not requiring comments to be parsed or carried into runtime environment.

The original idea has been described in projectfluent/fluent#16 and then extended to cover several areas:

Meta information

# A message displayed in the UI for license acceptance
#
# @context button
# @tone formal, polite
# @mode simple
# @policy firefox-brand-policy
license-accept-button = I have read Firefox Policy and accept the license terms

Original issue: projectfluent/fluent#139

Meta information generally describes some form of key-value pairs where keys are defined to carry a meaning that can be interpreted by tools like CAT tools, validators, etc.

Examples of use cases:

• UI context of the message
• Tone/formality of the message
• Mode/Toolkit - HTML, React, no-html, simple etc.
• License ID
• Policy meta-information (brand usage etc.)
• Message versions (tier 2)
• Variables
• ...

but by design the concept is fairly extensible and the value can be also an array, boolean, number or record, not just a string.

Variables

# Variables:
#   $value (Number) - Value of the unit (for example: 4.6, 500)
#   $unit (String) - Name of the unit (for example: "bytes", "KB")
sitedata-total-size = Your stored cookies, site data and cache are currently using { $value } { $unit } of disk space.

Original issue: projectfluent/fluent#140

This is a particular example of meta-information, which could be desugared to:

#   @var $value (Number) - Value of the unit (for example: 4.6, 500)
#   @var $unit (String) - Name of the unit (for example: "bytes", "KB")
sitedata-total-size = Your stored cookies, site data and cache are currently using { $value } { $unit } of disk space.

And then meta-information key @var has its value with its own syntax containing: name of the variable, type of the variable, description of the variable and a list of examples.

All of those are fully human-readable, and at the same time can be interpreted by the tooling allowing for easy error checks (You're passing a string to DATETIME formatter!), and more WYSIWYG/QA mode where the localizer can see a formatted string with an example value and check if it looks good.

Here's a prototype I once wrote for a tool that guides a localizer through a variant-based translation without expecting the user to step out of the "Q/A" mode - https://labs.braniecki.net/l10n-tool/
Such a UX requires example for variables, and this meta-information could provide it.

String Versions

# @version 2
my-string = This is a slightly updated string

Original issue: projectfluent/fluent#141

This comes from an observation that string changes are of one of the three types

1. Original string spelling update
2. Slight modifications
3. Significant update

In scenario (1) we'd like to update the source string without bothering anyone. Gettext, for example, doesn't allow for that since it uses the source string as an ID, thus any update to it invalidates all translations.
In scenario (3), we'd like to invalidate all "old" translations since they are not applicable to the new meaning of the source string. Fluent handles that by asking developers to change the ID, breaking the social contract and establishing a new one under a new ID.

Scenario (2) is the more nuanced one. The value of the original string does get updated, and we'd like to notify translators that they may want to look at the message again and consider whether their translation should be updated, but since we didn't change the meaning, we don't want to invalidate all translations - we believe the old translation is better for users than fallback to the source language for the string.

Currently Fluent doesn't provide that model, while our initial analysis indicate that at least 10% of changes could fall into that category. Currently, to stay on the safe side, we invalidate such translations accruing cost and requiring translators to provide a new translation.
Some of that cost is mitigated by MT, but not all.

This proposal adjusts it by providing an optional meta-information named @version. The initial, implicit version of every string is 1. When a string gets updated in the (2) scenario, the developer can "bump" its version to @version 2 and tooling can then detect that all translations without @version are on version 1 and let them know that this message has an update they may want to take a look at.
At the same time, software will still use version 1 of the translations since its good enough.
Once the translator takes a look, they either update it or mark it as "good" and @version 2 gets set in the meta-information of the translation of the message. The cycle continues.

I suspect that the 10% is lower than actual value of that mode. I believe that if we were to start using it in production, developers more often would prefer to go for scenario (2) over (3) and in result more translations would be preserved for longer with less cost to the user and localizers.

Title Line

## Privacy Section - Site Data
##
## This sections will contain several messages
## that should be translated by a lawyer if possible.

privacy-msg1 = Foo
privacy-msg2 = Foo 2

##

Original issue: projectfluent/fluent#138

Multiline comments pose an issue for some forms of CAT tool display. In Pontoon (CAT tools developed for Fluent), we could use the concept of Group Comments (#40) to visually cluster lists of messages in the left pane, but since the Group Comment can be multiple, there's no good way to semantically understand how to display a multi-line group comment in a single line for that use case.

Title Line could be an optional logic that states that if the comment is multi line, and the first line is separated from the rest by an empty line, then it is treated as a Title Line.

Such line could serve as a "summary" of the comment, or group, or resource, and have some highlighting markup added when displayed.

In the original issue you can see two mocks of how regular syntax highlighter and Pontoon could benefit from being able to identify title line of a multiline comment.

There are potentially other uses of such meta-information, and as with the rest of the planning, we may decide to just settle on data model, leaving syntax to describe it out of scope, but I think it's useful to imagine some implementable syntax for each data model piece that could actually work.

[Fleker]

Meta Information

I agree having some way to provide extra information in a machine-readable way can be beneficial for translators using tools. Having some sort of examples param for some more complex strings could be useful as well.

String versions

Introducing semantic versioning on a per-string basis may be overly complex, and difficult to enforce. Unlike APIs, which have a very strict definition of v1 -> v2, will a developer know when a string has changed sufficiently? Having some sort of more generic @lastUpdateTime param may be more useful, or perhaps the tools themselves should be more useful in showing recently updated strings that translators should investigate.

Title line

Some of this may be useful to incorporate into #40 to specify groups of strings and provide metadata on that group.

[mihnita]

I think that the feature(s) requested are good, but it is probably going too fast into details :-)

[zbraniecki]

Unlike APIs, which have a very strict definition of v1 -> v2, will a developer know when a string has changed sufficiently?

This is already the question - does the develop know when the change is (1) and when it is (3)? Working on localizable code requires some level of understand what happens with the strings you're adding. If you want to claim that developers have no idea, then they need someone to give i18n-review on their code or they will produce errors (like, preserve the contract while changing the meaning of the string etc.).

Having some sort of more generic @lastUpdateTime param may be more useful, or perhaps the tools themselves should be more useful in showing recently updated strings that translators should investigate.

We explored that and couldn't make it robust enough. Tooling trying to maintain those versions requires a very centralized system that is guaranteed to have control over all strings, versions and their timing to be able to do a "three-way diff" between old-source, new-source, old-translation and produce the information about new-translation.

Introducing semantic versioning on a per-string basis may be overly complex, and difficult to enforce.

The value here is that you don't need to enforce it. If things go wrong, you still have an updated source string and a meaningful translation. I see the value of this model in how gracefully it falls back.

And "overly complex" is, I guess a subjective position that depends on where you're coming from :) The number of issues filled in this repo is a good indicator to me that all stakeholders come with their own features that they successfully use and lack in other localization systems. For others, those features may look "overly complex" since their systems are not designed for that.

I would argue that this is not overly complex any more than other features suggested here. Whether it's worth designing around and for, is a different question. I can see us deciding not to invest in string lifecycle workflow, or design a different one that doesn't need such feature.

I think that the feature(s) requested are good, but it is probably going too fast into details :-)

Agree, I was mostly summarizing ideas that accrued in Fluent repo, for consideration for this work group!

[mihnita]

Some other things that can be expressed as semantic comments, for example width and storage restrictions, like "width less than 30 em" or "less than 80 code units when expressed as UTF-8"
(in a "machine readable form", so that a validation tool can use it.

But in general ++1 for a flexible "annotation" system.
For all the "nodes" (groups of messages / messages / placeholders)

[grhoten]

I've found that variable comments are helpful for the translators and maintainers. They are optional. So when the variable name conveys enough information, then a comment is not needed.

[zbraniecki]

For the example use case of title line in a group semantic comment, see a mockup of its potential use in a CAT tool like Pontoon - projectfluent/fluent#138

[aphillips]

Per discussion, I think we previously decided that comment were out of scope. Marking this issue for potential closure.