Chal­lenges of I18n at scale — Part 1

The first part of a two-piece write-up on the inter­na­tion­al­isa­tion of large-scale applic­a­tions in hyper-grow­ing engin­eer­ing companies.

A brief introduction…

Recently at synvert, we had a very excit­ing chal­lenge, one that taught us many les­sons and equipped us with import­ant tools for future endeav­ours regard­ing the inter­na­tion­al­isa­tion of e‑commerce pro­jects on hyper-scal­ing engin­eer­ing envir­on­ments. This art­icle is part of a two-piece write-up that will go over the chal­lenges we’ve faced and the pro­cess we ended up imple­ment­ing. In the second part, we’ll go over the improve­ments we’ve made in the pro­cess and how they improved the speed and qual­ity while redu­cing the costs of translations.

Before we delve deeper into the topic, it is import­ant to set the stage so that we can bet­ter under­stand the envir­on­ment and all its con­straints, which guided the rationale behind our decisions.

We boot­strapped a new team that was integ­rated into a lar­ger eco­sys­tem with over 400 engin­eers, in a com­pany with over 2000 employ­ees dis­trib­uted across mul­tiple ver­tic­als (engin­eer­ing, logist­ics, oper­a­tions, fin­ance, legal, human resources, among oth­ers…). The main focus of this new team was to plan and imple­ment inter­na­tion­al­isa­tion across all the cus­tomer-facing apps and some internal tool­ing, enabling these to dis­play dif­fer­ent pro­pos­i­tions in mul­tiple lan­guages and mar­kets. One crit­ical factor to keep in mind is that this engin­eer­ing branch was grow­ing quickly, and was expec­ted to become even lar­ger in the upcom­ing months, as the com­pany expan­ded into new mar­kets and acquired other com­pan­ies in the pro­cess. This meant the solu­tion had to apply to mul­tiple dif­fer­ent tech­no­lo­gies and be scal­able to accom­mod­ate future expansion.

Kick­ing things off

When we got onboarded onto the pro­ject, our main con­cern was gath­er­ing as much inform­a­tion as pos­sible to under­stand the busi­ness, the struc­ture of the com­pany, the exist­ing code­base, and how this code­base was main­tained and developed.

Dur­ing this period we made a couple of valu­able dis­cov­er­ies, that would be cru­cial to our decision-mak­ing mov­ing forward.

We wanted to devise and imple­ment a single strategy for dif­fer­ent code­bases, writ­ten in dif­fer­ent lan­guages and frame­works, these being:

• Next.js with TypeScript applic­a­tions that powered the cus­tomer-facing side of the website;
• React with TypeScript applic­a­tions for small pieces of logic that are integ­rated onto the main apps;
• React with TypeScript micro fron­tends that were used across the main apps and which shared depend­en­cies with them;
• Lara­vel applic­a­tion for oper­a­tions man­age­ment and product preparation.

Our goal was to provide a solu­tion that could be per­form­ant, simple to use and seam­lessly fit the tech­no­lo­gies lis­ted above (using a sim­ilar file struc­ture both on PHP & JS-based projects).

The decision process

To choose the inter­na­tion­al­isa­tion frame­work and Trans­la­tion Man­age­ment Sys­tem (TMS), it was import­ant to get as much feed­back as pos­sible from the engin­eer­ing teams. To gather this feed­back, we cre­ated an RFC (Request for Com­ments) and made it avail­able for every­one to com­ment on and improve on.

This was a pre­cious tool as it sur­faced inform­a­tion that we didn’t have at the time, allow­ing a team of hun­dreds of engin­eers to dis­cuss a single topic without repe­ti­tion and back­ground noise. Another bene­fit of using this approach is that all con­ver­sa­tions on the topic are cent­ral­ised and saved for his­toric pur­poses, mean­ing that later on, any­one can look at the doc­u­ment and under­stand why some decisions were taken and which altern­at­ives were analysed.

For the inter­na­tion­al­isa­tion frame­work, we decided on using dif­fer­ent approaches depend­ing on the type of applic­a­tion we were work­ing with. For large-scale React/Next.js pro­jects, we would use react-i18next, which ful­filled all our goals and was flex­ible enough not to be tied to a spe­cific frame­work. For smal­ler code­bases like small React applic­a­tions with a very spe­cific scope, we would use a light­weight in-house imple­ment­a­tion to keep them as small and per­form­ant as pos­sible while keep­ing most of the func­tion­al­ity they required in terms of inter­na­tion­al­isa­tion (string inter­pol­a­tion for example).

For the TMS, and after weigh­ing many tech­nical and busi­ness-related para­met­ers, it was decided that we would be mov­ing for­ward with Smart­ling.

Trans­la­tion work­flows at scale

One import­ant topic that has to be addressed when think­ing about inter­na­tion­al­isa­tion at scale is defin­ing a trans­la­tion work­flow to be used to deliver con­tent to dif­fer­ent mar­kets as quickly and effi­ciently as pos­sible. This kind of work­flow will have to meet some import­ant require­ments, which can some­times be chal­len­ging depend­ing on the scale of the pro­ject and the timelines we have for delivery.

Below I will go through these require­ments, explain­ing how we ful­filled them and some of the chal­lenges they posed.

Trans­la­tion Memory

An import­ant tool you should be aware of when start­ing to inter­na­tion­al­ise a web­site is being able to keep a uni­fied Trans­la­tion Memory. This works like a data­base of trans­lated con­tent that trans­la­tion tools to auto­mat­ic­ally sug­gest pre­vi­ously trans­lated identical text, redu­cing the time and cost asso­ci­ated with repeated trans­la­tion efforts.

One of the main bene­fits of trans­la­tion memory is con­sist­ency. By reusing pre­vi­ously trans­lated con­tent, trans­la­tion memory helps ensure that ter­min­o­logy, tone, and style are con­sist­ent across all trans­lated ver­sions of a web­site. This con­sist­ency not only enhances the user exper­i­ence but also helps main­tain the integ­rity of the brand.

In addi­tion to con­sist­ency, trans­la­tion memory also helps to stream­line the trans­la­tion pro­cess. Trans­lat­ors can eas­ily access pre­vi­ously trans­lated con­tent and can quickly identify where changes or updates are needed. This not only saves time but also reduces the like­li­hood of human error, fur­ther improv­ing the qual­ity and accur­acy of the translations.

Finally, trans­la­tion memory also provides valu­able insights into the loc­al­isa­tion pro­cess. By track­ing the use of spe­cific terms and phrases, trans­la­tion memory can help identify areas where addi­tional atten­tion may be needed, such as updat­ing out-of-date trans­la­tions or adjust­ing ter­min­o­logy to bet­ter meet the needs of a par­tic­u­lar audience.

In sum­mary, this is an essen­tial tool for web­site inter­na­tion­al­isa­tion and some­thing you should ensure espe­cially if you are work­ing with a large expand­ing web­site with a strong tex­tual component.

Dif­fer­ence between mar­ket and language

Mar­ket and lan­guage are two dis­tinct but related con­cepts. Mar­ket refers to the group of people or busi­nesses that a product is inten­ded for, while lan­guage refers to the means of com­mu­nic­a­tion used by that group of people. A mar­ket is defined by factors such as geo­graphic loc­a­tion, demo­graphic char­ac­ter­ist­ics, and pur­chas­ing habits, while lan­guage is determ­ined by cul­tural and lin­guistic factors. For example, you might want to sell spe­cific products in the French mar­ket, while allow­ing cus­tom­ers to browse your web­site in both French and Eng­lish lan­guage while pur­chas­ing them.

It is vital that this dis­tinc­tion is made clear to every­one involved in con­ver­sa­tions around inter­na­tion­al­isa­tion, espe­cially in large tech­no­lo­gical envir­on­ments and dur­ing rapid expan­sion phases where you might have slightly dif­fer­ent pro­pos­i­tions and lan­guages for each mar­ket. If this dis­tinc­tion is not made clear from the get-go, it is easy for people to get lost in import­ant con­ver­sa­tions around pro­pos­i­tion defin­i­tion and inter­na­tion­al­isa­tion in general.

Con­text is king

Another cru­cial aspect to be taken into account is ensur­ing trans­lat­ors have the right con­text and under­stand­ing of the pro­pos­i­tion, as this will help ensure that tech­nical terms and phrases related to it are accur­ately con­veyed and that the website’s mes­sage and tone res­on­ate with the tar­get audi­ence. Cul­tural and situ­ational under­stand­ing is para­mount for effect­ively com­mu­nic­at­ing with cus­tom­ers in the tar­get lan­guage. Without proper con­text, a trans­lator may mis­in­ter­pret the mean­ing of cer­tain terms and phrases, res­ult­ing in con­fu­sion and mis­com­mu­nic­a­tion. For example, an online car mar­ket­place tar­get­ing buy­ers in Ger­many should be trans­lated dif­fer­ently than one tar­get­ing buy­ers in the United States, as the Ger­man mar­ket may place more emphasis on fuel effi­ciency and envir­on­mental impact, while the Amer­ican mar­ket may place more emphasis on horsepower and per­form­ance. A trans­lator who doesn’t under­stand the con­text of the web­site may not be able to accur­ately con­vey these cul­tural nuances, res­ult­ing in a mes­sage that falls flat with the tar­get audience.

While this may sound easy, it can prove to be a daunt­ing task when deal­ing with large-scale e‑commerce pro­jects, with mul­tiple dis­trib­uted teams work­ing on dif­fer­ent parts of the sales funnel.

Below are the three main tools we’ve used to deliver a uni­fied and con­tex­tu­al­ised mes­sage to our customers:

• Study­ing the mar­ket and adjust­ing the pro­pos­i­tion
  This involves research­ing the tar­get audi­ence, their needs, and pref­er­ences, as well as under­stand­ing the com­pet­it­ive land­scape and legal con­straints of the mar­ket. By under­stand­ing the mar­ket, a web­site can tailor its pro­pos­i­tion to align with the cul­tural, lin­guistic, and legal nuances of the mar­ket. This can help increase its chances of suc­cess by effect­ively con­nect­ing with its tar­get audi­ence and stand­ing out from the competition.
• Main­tain a doc­u­ment to provide con­text to trans­lat­ors
  A great way to ensure trans­lat­ors have con­text on the pro­pos­i­tion they are trans­lat­ing is shar­ing a doc­u­ment with inform­a­tion about the website’s tar­get audi­ence, its mes­sage and tone, and any industry-spe­cific terms or phrases that may be used on the web­site. This inform­a­tion can help trans­lat­ors under­stand the con­text in which the text is writ­ten and ensure that the mean­ing is accur­ately con­veyed in the tar­get lan­guage. Provid­ing trans­lat­ors with the neces­sary con­text helps to ensure that the website’s mes­sage is effect­ively com­mu­nic­ated in the tar­get language.
• Cre­ate effi­cient com­mu­nic­a­tion chan­nels with the trans­la­tion agency
  Although asyn­chron­ous com­mu­nic­a­tion can work for most scen­arios, hav­ing a dir­ect chan­nel to the trans­la­tion agency can be invalu­able. This can make it easier to deal with issues that may arise dur­ing the trans­la­tion pro­cess, such as ques­tions about ter­min­o­logy or format­ting. It can also be used to align dead­lines between the com­pany and the trans­la­tion agency so that any poten­tial delays are iden­ti­fied and addressed promptly. Over­all, this helps ensure that the trans­la­tion pro­cess runs smoothly and that the final product meets the company’s expectations.

Ini­tial approach

One of our main goals in terms of tech­nical imple­ment­a­tion was ensur­ing a smooth integ­ra­tion of the trans­la­tion pro­cess in the devel­op­ment cycle. There are mul­tiple approaches avail­able to meet this goal, I’ll be going through the one we have put into prac­tice, detail­ing its bene­fits and shortcomings.

Below is an illus­tra­tion of how the ini­tial trans­la­tion work­flow fit­ted into our CI/CD pipelines.

As seen above, when a spe­cific imple­ment­a­tion did not require any trans­la­tions, the deploy­ment cycle was kept unchanged and there was no entropy added to the pro­cess. On the other hand, if a fea­ture added new strings that would be shown to a cus­tomer, these strings would be placed in a resources file (in our case, we would always add the strings to the Eng­lish resources, and then trans­late them into all other lan­guages). The pro­cess mainly revolves around the fol­low­ing steps:

• A mech­an­ism inside the deploy­ment pipeline checks for changes on the Eng­lish resource files, if any file was changed, the fol­low­ing actions happen:
• Gather the strings that were added/updated;
• Send the strings to the Trans­la­tion Man­age­ment System;
• Block the deploy­ment pipeline to ensure we wait for trans­la­tions before mov­ing forward;
• The trans­la­tion agency gets noti­fied about these strings, and will trans­late them dir­ectly in the TMS;
• After the strings are trans­lated, they are sent back to the code repos­it­ory and the deploy­ment pipelines are unblocked, enabling teams to deploy their code to test and pro­duc­tion environments.

The down­sides of the approach

The mech­an­ism above allowed for an effort­less integ­ra­tion between tech teams and the trans­la­tion agency, using the TMS as a bridge to facil­it­ate this, how­ever, it did come with a few short­com­ings we needed to tackle, which are lis­ted below:

• Devel­op­ment cycle delay
  With the above imple­ment­a­tion, our deploy­ment pipelines were blocked until a trans­la­tion was fully com­plete, which pre­ven­ted us from test­ing the imple­ment­a­tions in the test envir­on­ment and con­sequently, drastic­ally slowed down the pace of development.
• Lack of con­text from devel­op­ment teams
  Another pain point early on was the lack of con­text some developers had on this entire pro­cess, which led to the inter­na­tion­al­isa­tion team hav­ing to do mul­tiple ses­sions explain­ing the pro­cess in detail to dif­fer­ent teams to avoid errors and confusion.
• Lack of con­text from the trans­la­tion agency
  In the early stages of the imple­ment­a­tion, there were quite a few issues with mis­trans­la­tions due to the lack of con­text and the desync between the engin­eers and the Trans­la­tion Agency.

These short­com­ings were spot­ted early on and we star­ted work­ing on ways to tackle each one of them to ensure a smoother pro­cess, which we will go over in part two of this article.

The jour­ney con­tin­ues: improv­ing the process

Thank you for tak­ing the time to read this art­icle, we hope you found the inform­a­tion help­ful and informative.

In part 2, we will dive deeper into the improve­ments we made to the pro­cess that helped mit­ig­ate the issues out­lined above, make sure you read through it, as it will also con­tain sug­ges­tions for altern­at­ive approaches to inter­na­tion­al­isa­tion that might fit your pro­ject better.