Toki

Toki is an i18n framework for Go and a Textual Internationalization Key (TIK) processor implementation.

toki generate parses the source code, lints it reporting any misuse or errors, extracts localized texts and generates a localization bundle.

app-resource-bundle (.arb) files are used as intermediate translation storage for ICU messages generated from TIKs extracted from the source code.

• Quick Start Guide
  • 1. Create a new Go project.
  • 2. Prepare your Go project for internationalization.
  • 3. Localize your application for other languages and regions.
  • 4. Integrate Toki into your CI/CD pipeline.
• Bundle File Structure

Quick Start Guide

1. Create a new Go project.

mkdir tokiexample && cd tokiexample;
go mod init tokiexample;

tokiexample/main.go

package main

import (
	"fmt"
	"time"
)

func main() {
	now, numMsgs := time.Now(), 42
	fmt.Printf(
		"It's %s and you have %d message(s)\n",
		now.Format("Monday, Jan 2, 2006"), numMsgs,
	)
}

2. Prepare your Go project for internationalization.

1. First, we need to generate a new Toki bundle package with default language set to en (English):

go run github.com/romshark/toki@latest generate -l en && go mod tidy

2. Second, you'll need to adjust your messages in the source code to TIKs. Use the TIK cheatsheet for help and guidance.

package main

import (
	"fmt"

	"tokiexample/tokibundle"

	"golang.org/x/text/language"
)

func main() {
	now, numMsgs := time.Now(), 42

	// Get a localized reader for British English.
	// Toki will automatically select the most appropriate translation catalog available.
	reader, _ := tokibundle.Match(language.BritishEnglish)

	// This comment describes the text below and is included in the translator context.
	fmt.Println(reader.String(
		`It's {date-full} and you have {# messages}`,
		now, numMsgs,
	))
}

3. Now regenerate your Toki bundle to include the new localized message:

go run github.com/romshark/toki@latest generate

This will update your catalog_en.arb file and add the new message to it. However, Toki can't translate your messages fully so you will see in the generator report that your translations for en are incomplete yet.

To manually complete the translation, add the missing one {# message} plural option to the ICU message such that it becomes:

"msga5a0f2138b9d6598": "It''s {var0, date, full} and you have {var1, plural, one{# message} other {# messages}}",

You can also make it a bit better by adding a special case for 0:

"msga5a0f2138b9d6598": "It''s {var0, date, full} and you have {var1, plural, =0{no messages} one{# message} other {# messages}}",

TIP: The double apostrophe '' is not a typo, it's ICU's escape for a single quote.

4. After tweaking the catalog files, rerun the generator to update your bundle once again:

go run github.com/romshark/toki@latest generate

3. Localize your application for other languages and regions.

1. Run the generator, but this time, add a new parameter -t en-US -t de -t fr

go run github.com/romshark/toki@latest generate -t en-US -t de -t fr

This will create three new catalogs for American English (en-US), German (de) and French (fr).

2. You can now provide your translators with the new generated .arb files or repeat the same procedure as we did for en yourself.

3. Once you get your hands on translated .arb files, simply replace the ones in the Toki bundle package and rerun the generator to update it:

go run github.com/romshark/toki@latest generate

Coming Soon: LLM-based auto-translator.

TIP: If you're writing your TIKs in a different language than English you may find the CLDR plural rules table helpful. Don't worry, Toki will make sure you're not using illegal options for your default locale and will report any missing options.

4. Integrate Toki into your CI/CD pipeline.

You've now (mostly) mastered the entire i18n workflow for Toki. However, to make sure you never deploy broken or unfinished localizations, add Toki to your CI/CD setup:

go run github.com/romshark/toki@latest lint -require-complete

You may also use toki generate -require-complete and additionally git diff to ensure your generated Toki bundle package is up to date.

Bundle File Structure

• bundle_gen.go contains Toki's core source code and package API.
  • Not editable 🤖 Any manual change is always overwritten. DO NOT EDIT.
• catalog_<locale>_gen.go contains the localized catalog and message writers.
  • Not editable 🤖 Any manual change is always overwritten. DO NOT EDIT.
• catalog_<locale>.arb is an app resource bundle file containing translations in the ICU message format.
  • Editable 📝
  • Changed translations are preserved. You may edit translations.
  • If a new message isn't found in the translation file it's automatically added.
  • If a message is no longer used in the source code it's removed from this file.
• head.txt is a text file defining the head comment to use in generated files.
  • Editable 📝
  • If this file isn't found a new blank file is always automatically created. Use this file for any copyright notices and similar purposes.
• context.txt is a text file defining the overall global context for translators.
  • Editable 📝
  • If this file isn't found a new blank file is always automatically created.

All other files in the bundle package are ignored and preserved. .go files that don't have the following head comment are always preserved:

// Generated by github.com/romshark/toki. DO NOT EDIT.