Migrating internationalization files

During my Ph.D. migration project, I consider the migration of several GUI aspects: Visual, Behavioral and Business. These elements are the main ones. When perfectly considered, you can migrate the front-end of any application. But, we are missing some other stuff 😄 For example, how do you migrate i18N files? In this post, I’ll present how to build a simple migration tool to migrate I18N files from .properties (used by Java) to .json format (used by Angular).

I18N files

First, let’s see our source and target. As a source, I have several .properties files, including I18N for a Java project. Each file has a set of key/value and comments. For example, the EditerMessages_fr.properties is as follow:

##########
#    Page : Edit
##########

pageTitle=Editer
classerDemande=Demande
classerDiffusion=Diffusion
classerPar=Classer Par

And it’s Arabic version EditerMessages_ar.properties

#########
#    Page : Editer
#########

pageTitle=تحرير
classerDemande=طلب
classerDiffusion=بث
classerPar=تصنيف حسب

As a target, I need only one JSON file per language. Thus, the file for the french translation looks like this:

{
    "EditerMessages" : {
        "classerDemande" : "Demande",
        "classerDiffusion" : "Diffusion",
        "classerPar" : "Classer Par",
        "pageTitle" : "Editer"
    }
}

And the Arabic version:

{
    "EditerMessages" : {
        "classerDemande" : "طلب",
        "classerDiffusion" : "بث",
        "classerPar" : "تصنيف حسب",
        "pageTitle" : "تحرير"
    },
}

Once the meta-model is designed, we must create an importer that takes .properties files as input and produces a model.

Properties Importer

To produce a model, I first look for a .properties parser without much success. Thus, I decided to create my own parser. Given a correctly formatted file, the parser provides me the I18N entries. Then, by iterating on this collection, I build an I18N model.

I18N parser

To implement the parser, I used the PetitParser2 project. This project aims to ease the creation of new parsers. First, I downloaded the last version of Moose, and I installed PetitParser using the command provided in the repository Readme:

Metacello new
    baseline: 'PetitParser2';
    repository: 'github://kursjan/petitparser2';
    load.

In my Moose Image, I created a new parser. To do so, I extended the PP2CompositeNode class.

PP2CompositeNode << #CS18NPropertiesParser
    slots: { };
    package: 'Casino-18N-Model-PropertyImporter'

Then, I defined the parsing rules. Using PetitParser2, each rule corresponds to a method.

First, start is the entry point.

start
    ^ pairs end

pairs parses the entries of the .properties files.

pairs
    ^ comment optional starLazy, pair , ((newline / comment) star , pair ==> [ :token | token second ]) star , (newline/comment) star ==> [ :token | 
      ((OrderedCollection with: token second)
           addAll: token third;
           yourself) asArray ]

The first part of this method (before ==>) corresponds to the rule parsed. The second part (after ==>), to the production.

The first part tries to parse one or several comment. Then, it parses one pair followed by a list of comment, newline, and pair.

This parser is clearly not perfect and would require some improvement. Nevertheless, it does work for our context. The second part produces a collection (i.e. a list) of the pair.

Building the I18N model

Now that we can parse one file, we can build a I18N model. To do so, we will first parse every .properties file. For each file, we extract the language and the namespace based on the file name. Thus, EditerMessages_fr.properties is the file for the fr language and the EditerMessages namespace. Then, for each file entry, we instantiate an entry in our model inside the namespace and with the correct language attached.

importString: aString
    (parser parse: aString) do: [ :keyValue | 
        (self model allWithType: CS18NEntry) asOrderedCollection
            detect: [ :entry | 
                "search for existing key in the file"
                entry key name = keyValue key ]
            ifOne: [ :entry |
                "If an entry already exists (in another language for instance)"
                entry addValue: ((self createInModel: CS18NValue)
                    name: keyValue value;
                    language: currentLanguage;
                    yourself) ]
            ifNone: [
                "If no entry exist"
                (self createInModel: CS18NEntry)
                    namespace: currentNamespace;
                    key: ((self createInModel: CS18NKey)
                        name: keyValue key;
                        yourself);
                    addValue: ((self createInModel: CS18NValue)
                        name: keyValue value;
                        language: currentLanguage;
                        yourself);
                    yourself ] ]

After performing the import, we get a model with, for each namespace, several entries. Each entry has a key and several values. Each value is attached to the language.

JSON exporter

To perform the JSON export, I used the NeoJSON project. NeoJSON allows one to create a custom encoder. For the export, we first select a language. Then, we build a dictionary with all the namespaces:

rootDic := Dictionary new.
    (model allWithType: CS18NNamespace)
        select: [ :namespace | namespace namespace isNil ]
        thenDo: [ :namespace | rootDic at: namespace name put: namespace ].

To export a namespace (i.e., a CS18NNamespace), I define a custom encoder:

writter for: CS18NNamespace customDo: [ :mapper | 
    mapper encoder: [ :namespace | (self constructNamespace: namespace) asDictionary 
        ] 
    ].

constructNamespace: aNamespace
    | dic |
    dic := Dictionary new.
    aNamespace containables do: [ :containable | 
        (containable isKindOf: CS18NNamespace)
            ifTrue: [ dic at: containable name put: (self constructNamespace: containable) ]
            ifFalse: [ "should be an CS18NEntry" 
                dic at: containable key name put: (containable values detect: [ :value | value language = language ] ifOne: [ :value | value name ] ifNone: [ '' ]) ] ].
    ^ dic

The custom encoder consists of converting a Namespace into a dictionary of entries with the entries keys and their values in the selected language.

Perform the migration

Once my importer and exporter are designed, I can perform the migration. To do so, I use a little script. It creates a model of I18N, imports several .properties file entries in the model, and exports the Arabic entries in a JSON file.

"Create a model"
i18nModel := CS18NModel new.

"Create an importer"
importer := CS18NPropertiesImporter new.
importer model: i18nModel.

"Import all entries from the <myProject> folder" 
('D:devmyProject' asFileReference allChildrenMatching: '*.properties') do: [ :fileRef | 
    self record: fileRef absolutePath basename.
    importer importFile: fileRef.
].

"export the arabian JSON I18N file"
'D:/myFile-ar.json' asFileReference writeStreamDo: [ :stream |
    CS18NPropertiesExporter new
        model: importer model;
        stream: stream;
        language: ((importer model allWithType: CS18NLanguage) detect: [ :lang | lang shortName = 'ar' ]);
        export
]

Ressource

The meta-model, importer, and exporter are freely available in GitHub.