Introduction | About this playbook

Global App Testing is a community of testing professionals. There’s 90,000 of

us at the time of writing; we run best-in-class functional and user research
around the world, much of which is concerned with localization quality.

Today, in 190 countries we help the biggest software businesses deliver

programs worth hundreds of billions of dollars. With that knowledge, we
wanted to build this playbook.

What is this playbook?

Recently, we ran an interview series: Around the World in 7 Localization

Interviews. It includes interviewees from HubSpot, Deliveroo, Slack, Google,
Pleo, Pinterest, and Shopify.

We’ve gone back and looked at some of the highlights of those interviews.
We’ve also supplemented it with our own knowledge and research from working
alongside major localization teams from global businesses.

And here’s what you’ll learn:

1 Different stages of maturity for l10n teams and where you are

2 2.5x faster go-to-market time and better l10n delivery

3 Framing localization as an investment to be more valued

4 Understanding your investments, your ROI and what to do next

5 Decision-making data and identifying value-add resolutions

6 Generating leadership buy-in for your team as a strategic leader

7 How to be a strategic leader in your business

1
Driving Localization Leadership

Table of contents

Introduction | Why do l10n teams fail

Chapter 1 | The delivery narrative

- What is the delivery narrative in localization?

- Our interviewees’ comments on challenges in delivery

- Takeaways from chapter 1

Chapter 2 | Ace delivery; improve your speed by 2.5x

- Our interviewees' advice on rapid delivery

- Localization QA & Localized QA

- Takeaways from chapter 2

Chapter 3 | Beyond delivery: moving to the investment narrative

- How the investment narrative is different

- Identifying & communicating ROI

- ROI: advice from interviewees

- Making basic, low-friction, tactical recommendations

- Takeaways from chapter 3

Chapter 4 | Becoming a strategic partner

- How is strategy different from investment?

- Our interviewees’ advice on governance

- Generating the data for strategy

- Our interviewees’ advice on localization

- Takeaways from chapter 4

Global App Testing | 2023 2

Driving Localization Leadership

Chapter 1 | What is the delivery narrative?

In this chapter, we’ll cover the following:

1 Different team maturity stages. What they are and how

do we recognize them?

2 What’s the delivery narrative? The first stage of maturity

and how to ace it.

3 The challenges as we understand them. Here’s the

challenges businesses describe in delivery.

What is a narrative?

In our Amazon bestseller: Leading Quality, we referred to ”narratives” around

software quality which a business shares.

A “narrative” is a set of ideas and principles your team has, and it can be about
anything. What do they believe about quality? Security? Team building? Even if
you don’t deliberately encourage specific views, your team has a narrative, and
you should know what it is.

And in localization?

Because it’s such a cross-functional job, the narrative about localization was
even more important.

We identified 3 belief areas:

1. Your business believes something about what localization is

2. Your business believes something about what a localization team is for

And therefore

Global App Testing | 2023 3

 Driving Localization Leadership

3. Your business believes something about whether a localization team has

been successful

What is the delivery narrative in localization?

The first localization narrative is the delivery narrative.

Put simply, the narrative says: “the localization team delivers localization”

“We need to translate our product, our support material, and our marketing
content to local users. So we’ve hired someone to sort that out.”

See how we tease that out against the other narratives below:

What’s a localization What is Has the l10n team

team for? localization? been successful?

The To deliver localization; A necessary evil They deliver fast l10n

(usually translations) (sometimes,
delivery (Mostly) translating continuously)
narrative things
There’s no quality
A commodity issues

The An investment in A nice-to-have They achieve ROI, and

international explain it convincingly
investment effectiveness The more you localize,
narrative the more customers like
you

Enhances our ROI – but

employees don’t trust
the figures

The The voice of local users Essential to our We achieve our

performance mission in
strategy international markets
narrative A process which involves
data gathering &

Global App Testing | 2023 4

Driving Localization Leadership

governance

A strategic investment

Among our clients, we sometimes find that l10n professionals were more likely
to be critical of the delivery narrative than the others. In particular, they are:

1) Frustrated by what they perceived as an early-stage narrative and

answering to others;

2) More likely to report a difference between their own understanding of

that role; and that of the business.

But the advice from the top performers?

3) It’s no good to resent this narrative. You need to ace delivery, especially
delivery which is requested by other departments. We’ll show you how to
do it below.

Challenges with the delivery narrative

From our own knowledge and our recent interview series, here are some of the
challenges with the delivery narrative.

1. Negative perceptions of localization

I. A necessary evil

Some localization professionals are concerned that delivery-only departments

are perceived as a “necessary evil.” Those were the words of one Localization
Director Global App Testing spoke to on background.

“I want to advocate for deeper localization,” he told us, “but engineers perceive
us as a kind of necessary evil. I want to make the argument that there’s an ROI
on deeper localization.”

Global App Testing | 2023 5

Driving Localization Leadership

It also cropped up in our recent interviews:

‘It’s very easy to get pulled into being this in-house translation agency,’ Iggy,
Localization Manager @ Deliveroo had told me. ‘Or a necessary evil at the end
of a process. I’ve tried to position my team as a deeper value.”

One of the consequences of regarding l10n as a barrier-to-release rather than

an investment is that it comes to be perceived as a cost center, which can cap
the value you can deliver.

II. A commodity

A related idea would be that localization is a “commodity”.

“I feel that localization in general is perceived as a commodity”, said Natalia, the

Terminology Manager at Google, in her interview with GAT, although she added
“not at Google”.

To spell out what this means, a commodity is a good which is fungible or largely
fungible – for example, that translations are all equivalent, in the same way that
gallons of crude oil would be equivalent.

Selling a “commodity” is not necessarily a negative thing, but does transform

how we might look at our understanding of success. As a commodity,
localization success would be about delivering localization quickly and
cheaply.

2. Wildly different workflow expectations

If localization is about delivering high quality quickly, it raises a question. How

fast should you be delivering things?

In his interview with Global App Testing, Robert Bauch, a Product Program
Manager at HubSpot described the challenges of product and marketing
working through i8n challenges together.

Marketing copy needed to be “polished to the max” said Robert; his world of
product needed to be more consistent to specialized language libraries – and
much faster.

Global App Testing | 2023 6

Driving Localization Leadership

“In product, we needed to keep moving. We were pushing code to production

hundreds of times a day. 3-400 times a day. We needed [a translation process]
to keep up with that.”

Natalia, a Terminology Manager at Google, used to work for a language service

provider. In her interview, she said that they sometimes had to educate
marketing teams.

“It could be difficult to have conversations”, she said, with teams who
“understand tone of voice – but not products, development and release cycles,
or anything else from a best practices standpoint.”

How do you thread that line? We explain below.

3. A lack of awareness of non-American user needs

Steve Jobs articulated one of the most famous ideas in product.

“You have to start with the customer experience” he said, “and work backwards
to the technology.”

Does that happen in localization teams? Consider the following categories of

product localization:

A B C

🇺🇸 🇺🇸→🇧🇷 🇧🇷
English-only content or Localized content or Local content or
product product product built from
scratch

With the exception of Slack, every business we spoke to varied the extent to
which they localized depending on things like market size. But teams focused
on “B” don’t introduce users from local markets at their planning stage, so the
content or product is likely to be further from local user needs.

Natalia, Terminology Manager @ Google said: “Let’s be frank: it’s very difficult
for someone to think of terminology for global audiences. When content
creators, or the content owner thinks about what they’re going to produce, they

Global App Testing | 2023 7

Driving Localization Leadership

usually do it in their language… so that means they might focus all their efforts
into that.”

4. A trade-off between speed and quality

A trade-off between speed and quality is well-known to anybody who works in

software. In fact, many engineering ideas seem to be creeping into a space
formerly run by translators. “We work in CI/CD” said Diana, Content Localization
Manager @ Pleo, who was not from a technical background.

”Continuous Integration & Continuous Delivery” is a framework which describes

how you can deliver software effectively in a software team. It makes
recommendations about automation in your release cycle, for example; but we’ll
talk more about that later.

It’s also a challenge to get good translations from third-parties at pace.

Natalia, Terminology Manager @ Google, described the “Holy Trinity” of using

third parties to deliver your translations – speed, quality, price. “You can’t have
all three”.

We know that quality assurance is often a bottleneck. Diana, the localization

content manager @ Pleo, described a process whereby “every word” was
checked internally despite a lack of in-house translators. Pleo was fortunate,
said Diana, “we have a lot of people that are very interested… and very proud of
bringing Pleo to their home country.”

Global App Testing | 2023 8

Driving Localization Leadership

Key takeaways from chapter 1

1 Yours might be a delivery narrative business.

The “delivery narrative” is the most common localization

team narrative that we encountered among teams – this
is likely to include you

2 Delivery is about speed & scale

As a commodity, teams will appreciate rapid delivery and

will understand “quality” as lack of mistakes; rather than
e.g. deeper rapport with users

3 Delivery narratives are fraught with challenges

Beyond being squeezed to go faster, professionals report

feeling undervalued and businesses fail to accommodate
for non-english-speaking users

4 To get beyond delivery you need to ace it

It’s not acceptable to opt out of delivery – you need to

ace delivery in order to move on to better narratives

Global App Testing | 2023 9

Driving Localization Leadership

Chapter 2 | Ace the delivery narrative;

improve your go-to-market speed by 2.5x
Here’s what we’re covering in this chapter:

1 How can you improve your release speed by 2.5x?

Localize faster, and localize smarter.

2 Automating more of your localization workflow. Here’s a

model of how greater automation can work.

3 How to categorize and split your deliverables. We’ll talk

about how to devise different playbooks per stream and
begin to split them.

In our recent interview series, we talked with Pleo about how it had improved its
go-to-market time on localization changes. The European business expenses
app had a bumper globalization year in 2022, and had launched in 10 new
markets bringing their total to 15 in total. “It was one market per month,
sometimes two.”

Over the year, Diana had improved the app’s go-to-market speed by 2.5x.

“The first market took us seven weeks, and then the last one took us three,”
said Diana. It’s 2023 where the team are starting to build out a deeper
localization maturity and focus on a deeper rapport with users.

There’s plenty of advice about speed-to-market; which we have amalgamated

below.

1. Ensure you invest in the core materials

When it comes to a localization delivery pipeline, every time you redouble your
efforts, you are wasting time.

Global App Testing | 2023 10

Driving Localization Leadership

When we asked Diana the main way Pleo improved its go-to-market speed,
here’s what she said:

Having “a ready playbook, a core asset list, and all the dedicated resources
frontloaded… having glossaries and style guides prepared.”

Natalia, the Terminology Manager @ Google, used to work for a major LSP. We
asked her what she’d advise about LSP engagement and she said:

“The one thing I would see recurrently is the assumption that a localization
services provider is going to know everything without actually getting any
information from the client. There’s an expectation… that a linguist would know
what they’re going to encounter, how to translate it.”

Lots of that can come from core materials. “Style guide, glossary, identification
of content types, formats, context, audience type,’ listed Natalia. “That’s usually
one of the biggest mistakes I’d encourage everyone to invest time on.”

Core materials

📚 Playbook What is the nature of the l10n you undertake in different

circumstances?

📝 Core asset list What assets do you need to launch somewhere new?
How far is each market localized?

📘 Glossary What key terms do you have? How are they translated?

🧍 Audience Who are we talking to? What’s the audience here?

📻 Contexts What are the contexts and formats for the different types
of content? Where and how are users digesting it?

🎨 Brand What are your brand guidelines around things like tone of
voice?

Global App Testing | 2023 11

Driving Localization Leadership

2. Playbooks should be different per asset stream

In HubSpot in 2013, they began to separate the marketing and product asset
streams within localization.
Robert, Product Program Manager @ HubSpot described the split:
​
​“As we got the product internationalized, we could stay fairly close because we
were operating at a really small scale. That made it easier to keep the quality
tight… over time, we needed to ramp up the volume of content we were
pumping into these markets.”

In time, marketing and product content split their workflows and their
processes. “The needs of scale began to separate us… and then also the
quality requirements started to change in your product.”

“Public-facing marketing content needs to be polished to the max” whereas

with the product content, they found they were better able to use automated
tools to generate some of the content.

Teams and stream pods

A B C D

📘 📙 📗 📕
Website Product Campaign Help
Content Content Content Content

In Pinterest, there are also pods to help them handle different requirements of
these streams.

“One pod works on product internationalization” said Francesca, the Head of

Internationalization & Design Program Management there. “Which means
international product UI strings […] We do functional and linguistic QA on our
core product and features. We internationalize the UI, the iconography if
needed, taxonomy, and so forth.”

“I have a team working on Pinterest scaled platforms. So they translate,

internationalize the Pinterest websites. That includes the Business Site, the
Help Centre, educational platforms, and seasonal websites.”

Global App Testing | 2023 12

Driving Localization Leadership

Different depths per country

With the exception of Slack, all of the businesses moderated the level of
localization they provided depending on the country.

Robert @ HubSpot explained: “With Japanese, you have to have somebody

who is ingrained in Japanese culture and language to have an app which is
perceived as high quality. You don’t necessarily need someone in Finnish. We
don’t have an internal Swedish speaker, but we offer a Swedish UI.”

3. Your product localization playbook should be automated as

much as possible

Availability Requires Methodology

📙
Product Continuous; Consistency, Agile delivery +
Content always-on speed machine learning

There is less of a distinction between content and software than there once
was.

CI/CD stands for “Continuous Integration / Continuous Delivery” (sometimes,

“Continuous Deployment”). There are lots of articles which describe in more
detail what CI/CD means, but it refers to a framework for how teams should
plan and release code, with emphasis on speed, accuracy, and automation in
deployment.

Whether or not you choose machine translation, machine learning is a must for
this kind of content.

Robert, Product Program Manager @ HubSpot argues: “add machine learning

as soon as you can. There are so many tools and this was ten years ago – the
landscape wasn’t as robust as it is now. Add machine learning and automation

Global App Testing | 2023 13

Driving Localization Leadership

to your process, because you can leverage so much of the work you’ve done in
the past to ensure quality with speed in the future.”
More automation in product localization

Back in 2013, HubSpot’s experiments with machine translation were “slightly

below the quality we were shooting for” – but machine translation is improving,
and the guide is being written in Q2 2023, following the breakthrough in LLM
technologies pioneered by OpenAI with ChatGPT.

Robert explained the Babel system which Hubspot used for machine
translation. Babel simultaneously provides a machine translation offer which is
live for the customer and routes the translation to a third party translation
vendor for QA.

“Babel allows us to get code to translation and get it back within minutes for
the machine translation layover. And it very rarely takes more than 24 hours to
get a solid human translator on the string. The engineer or dev doesn’t even
see it happen, he or she just sees the code come in, and they pull it into their
repository.”

Global App Testing | 2023 14

Driving Localization Leadership

Pleo aims to automate away the function as much as possible.

‘If you ask the engineers’, said Diana, Localization Content Manager @ Pleo,
everything will be automated, and there will be no longer need for us. So that's
kind of the end-goal for engineering.’

4. Is it a project? Think carefully about scope

Many of the professionals I spoke to referred to a lack of time or a high-paced

environment as a factor.

“It’s tech, so everything was supposed to be done yesterday” said Francesca,

Head of Internationalization and Program Design Management @ Pinterest.

In our forthcoming report based on a survey of localization professionals, we’ll

show that there is broad agreement that pressure on the speed of release has
an impact on quality of localized changes.

Additionally, most of our interviewees referred to scale and the difficulties of

scaling their process. The advice in this section works well for scale, because
focusing on low-cost continuous localization which is mostly automated can
handle.

“You can definitely get to a level with too much complexity,” argued Iggy,
Localization Manager @ Deliveroo. “We have this concept of globalization –
your user experience should in theory be the same in every language we offer.”

Global App Testing | 2023 15

Driving Localization Leadership

Diana @ Pleo had come to the conclusion that localizing fewer things, to a
higher standard, is better. ‘I prioritized an MVP of localization based on what we
had produced. So the apps, the website, the most popular landing pages, the
most popular help center articles,’ she said.

Split your asset streams by longevity and price to produce, and begin to
attempt to reduce the scope of things which can’t scale.

Cost Longevity

Machine translated Near-free

Machine translated & checked Low

Human translated Medium Depends on asset

Transcreated High

Built from scratch High

5. You’re going to need LSPs, so manage the relationship

carefully

Natalia, Terminology Manager @ Google and former Director of Quality at an

LSP, argued LSP use is “an absolute need. If localization is not your core
business it’s too hard.” (She added that this was not true of Google, arguing it
was a special case.)

“If you really want to produce large amounts of content in all the languages, it’s
going to be very difficult to sustain that internally. So you need to really
understand what your resources are.”

“To make the most out of those resources, which might not be having [an LSP]
localizing content, but thinking of how the shape of that market is going to look
in a language.”

More than one interviewee argued that long-term engagement with an LSP was
preferable if possible.

Global App Testing | 2023 16

Driving Localization Leadership

“We got really tight with a [translation] vendor,” said Robert, Product Program
Manager @ HubSpot. “That part of the quality process we took on early was
finding a good vendor and sticking with them.” Robert argued that a better
depth of knowledge on their part improved the quality of their work.

6. Make requests easier, not harder

Many l10n pipelines are dominated by requests from other departments.

A common failure mode in any business is for a service department to keep on

top of their requests by making it extremely difficult to request things from
them. The best performing localization teams made it easier to request
localized changes to features or translations of their content.

It was true of our interviewees:

- Slack automated the localization request system within the Slack app, so
requesters didn’t have to leave their channel.

- Pleo made it possible to tag a feature or piece of content for l10n in

Github and to automate the delivery flow to and from the development
environment.

- HubSpot used the Babel system described in detail above.

Robert, Product Program Manager @ HubSpot argued: “If you start to scale,
and there’s long times between questions getting asked and answered, trust
gets diminished. At that point, you start to incur debt, because you have
problems that are going unsolved.”

7. Continuous localization quality assurance should be

elastic, and be more than translation

Quality controls are essential in localization, especially if you are looking at

routing more of your localized content through fast, automated or even
machine translated systems.

Elastic LQA

Global App Testing | 2023 17

Driving Localization Leadership

Where you’re delivering localization quickly, you should consider elastic LQA.

Elastic LQA is an approach to LQA which emphasizes availability of QA supply,

thus reducing the friction for a buyer who needs to access that supply. In the
case of Global App Testing that means “crowdtesting”, where a large group of
users and testers are available to review different software products, sites, and
assets, in 190 countries and territories.

Because that pool contains over 90,000 people, there are LQA experts available
in any one of those countries and it means we can guarantee review of a
localized asset or feature within 48 hours.
LQA, Localized QA, Translation QA

Some nomenclature, quickly.

- “Localized QA” refers to functional QA in a specified local environment,

and is great for engineering-focused teams to do things like compatibility
testing, and ensuring that your software still works in general as
expected in a local environment.

- LQA, or “localization QA” (we don’t use “language QA”) refers to the
process of checking localized changes outlined below.

Why not language QA? Well, many early-stage localization attempts are focused
exclusively on translation. And whether you’re translation-first or not, it is not
worthwhile to limit LQA solely to translation checks.

In our view as a localization QA vendor, translations create the risks outlined in

the table on the next page.

Global App Testing | 2023 18

Driving Localization Leadership

Localization QA

Translation Date formats and foreign measurements are very easy for
adjacent translation-oriented QAs to miss. Have you ever read about
mistakes the Turkish İ, for example?

Look & feel Translations have a habit of ruining the design of things.
issues “Overlong or truncated strings” are strings of text which
extend beyond the bounds, or are cut off at the bounds, of
their respective boxes. But these are just two examples –
your site or product design is likely to be impacted by
translation in various ways, generally to do with text
spacing.

Cultural The immediate way that businesses understand cultural

issues & checks is to do with sentiment risks, for example, images
mistakes which are considered explicit in some cultures creeping into
multimedia. But things like assumptions made during
onboarding can be just as important. “Required” last names,
address formats, will all lead to customer dropoffs during
forms.

Functional Although it is not (usually) the job of a localization manager

bugs to chase up your engineers, a QA bug can have a bearing on
the measurements above. In particular, integrations and
access. Local banking and checkout partners are one of the
most common reservoirs of functional bugs.

When you are looking at localization as a commodity, then you want QA which
is fast, thorough, and cost-effective. That’s where Global App Testing can help
you, and we’re only an email away.

Get Started with Global App Testing →

Global App Testing | 2023 19

Driving Localization Leadership

Key takeaways from chapter 2

1 Ready your core materials ASAP

Get all of the core materials in a centralized place; this

will make strategic governance easier as well as
improving your localization speed

2 Divide your delivery streams

Identify “commodity” localization and work as hard as

you can to automate that, including by experimenting
with machine translation tooling

3 Choose elastic LQA

Get QA which is flexible and demand-oriented, instead of

relying on a supplier used to walking slowly in waterfall

4 Calculate your costs

Our next chapter will be about getting to the investment

narrative; for which you need to understand your costs

Global App Testing | 2023 20

Driving Localization Leadership

Chapter 3 | Beyond delivery: moving to the

investment narrative

Evolving your maturity level

In addition to acing delivery (Chapter 1-2), there’s lots of advice available on

how you can evolve your maturity level. We’ve listed them below, and they will
form the basis for the second half of this playbook. Here’s what we’ll cover in
chapters 3-4.

Chapter 3 | Moving to the investment narrative

1 Identifying your ROI. How to calculate your ROI and

describe a value narrative for your business.

2 Building a data picture. How do you understand the

impact you’ve had? How do you experience local users’
experiences?

3 Turning stakeholders into clients. Here’s how you begin

to transform your role in the business.

Chapter 4 | Becoming a strategic partner

4 Offering strategic advice. How do you begin to start

identifying strategic opportunities?

5 Creating & influencing policy. How do you represent

local users to your organization?

6 Being a leader. Here’s how you can exercise amazing

leadership in localization

Global App Testing | 2023 21

Driving Localization Leadership

How is the investment narrative different

After delivery, the second narrative is the investment narrative. The

“investment” narrative says, “we invest more in our local offering by investing
more in localization.”

Delivery Investment
Localization is a Commodity Activity differentiated by
depths & standards

Localization acts as A barrier to releases An investment in

effectiveness

The L10n team is a Cost center Revenue generator

You are measured Expense & speed ROI

by

Successful teams Run workflows better Identify opportunities

will better

“Better quality” Fewer bugs or errors Improvement in local

means metrics

You need to focus Solid core materials Understanding ROI

on….
Differentiated playbooks Prioritizing localization
activities
Automation & machine
translation Building value narratives

Agile deployment Opportunity-based

quality insight
Scope & suppliers

Elastic LQA

Global App Testing | 2023 22

Driving Localization Leadership

Identifying & communicating ROI

Identifying & communicating ROI is the easiest way to shift the dial in terms of
how your team is perceived. But calculating ROI is much harder in localization
– which is why our interviewees had a lot of advice for how to think about it.

What people say about ROI

The formula for ROI is

ROI = added value of l10n / costs of l10n

The formula seems simple, but it becomes much harder when we get to the
metrics and costs involved in localization, on the next page.

A complicating factor is that you don’t “own” these metrics, you’re merely
inferring an effect on these metrics. That would imply measuring the metrics
before and after a localization activity, which is tough – especially if you’re
localizing from the outset.

Added value metric examples

- Pageviews - Additional MRR - NPS

- Keywords - Total MRR - Customer
- Signups - User churn satisfaction
- Monthly Active - Checkout completion - Reviews per
Users [MAUs] - Revenue platform
- User session time - Profit - Resource use
- Bounce rate - Qualitative
feedback
- Market share
- Engagement

Global App Testing | 2023 23

Driving Localization Leadership

Cost component examples

Cash Time

- Initial i18n investment - Delays

- Translation costs - Labor borrowed from other
- Engineering costs teams
- Other analyst costs
- Other labor costs
- Project costs

It is not possible in a small l10n team to calculate all those numbers. For l10n
teams with fewer resources, l10n relies often on a fuzzy sense of their value
and a corresponding mistrust in their numbers.

Some of the l10n professionals we have spoken to off-the-record have

described a struggle to articulate the ROI: they struggle to calculate it first of
all; and then they struggle to generate credibility that their numbers are
accurate.

Here’s what our interviewees had to say. 👇

ROI: the advice from our interviewees

1. Use analysts for the best results

Where possible, you should use analysts to do the work for you.

Francesca, Head of Internationalization and Program Management @ Pinterest

said, “I’d work with our team of [business] analysts and ensure that they had
analysts dedicated to international [users]”.

Localization Operations Lead at Shopify said: “Return on investment is elusive.

Everybody says there’s a formula. But at [former employer & HR management

Global App Testing | 2023 24

Driving Localization Leadership

platform] Ceridian, I asked a colleague who calculates ROI for customers for a
living to help.”

A HubSpot localization manager in another podcast referred to the use of

analysts, too: “HubSpot Research is doing a lot of really original, data-based
research…” he also describes the findings. “We’ve found that for certain locales,
we’ve found that native-language content vastly outperforms localized
content.”

Having somebody outside of the l10n department marking your effectiveness is

also best practice in getting other teams to buy into the ROI figure. Not
everybody has access to analysts, but you should at the least try to influence
your data operations to create an international dashboard.

2. Attune your value frameworks to your organization

“You need to talk about [ROI] in your organizational language,” argued

Oleksandr, the Localization Operations Lead @ Shopify.

“Let’s say [your employer is] product- and technology-obsessed. Then, you
have to talk in those terms… create a roadmap. Talk about short- and long-term
parts, present the benefits and communicate, communicate, communicate.
Never shut up about that roadmap.”

“The biggest mistake [that new localization managers make] is not aligning with
the goals and principles of the company you were hired into. Wanting to build
things the right way, the localization way. You’re hired, and you’re paid, you're
employed to do things for the business,” said Oleksandr.

Robert, Product Program Manager @ HubSpot, argued that you should reduce
the complexity across functions by ensuring that you are unified on a single
organizational value framework. “I think one of the main challenges you run into
in a company is that sometimes teams can work at cross-purposes,” said
Robert.

HubSpot’s framework: “What’s the value to the customer? That question starts
to bring everybody into alignment. At the end of the day, if you’re not building
things your customers need, neither of those matter.”

Global App Testing | 2023 25

Driving Localization Leadership

3. Ask not what your stakeholders can do for you…

By the same token, you can win the trust and appreciation of your stakeholders
by looking at their metrics in closer detail.

“I think you need to identify the KPIs of your stakeholders,” argued Iggy,
Localization Content Manager @ Deliveroo, “and help them to be successful
eventually. That's the only way that you can start changing the attitude of your
stakeholders as well – who might think about you as a contributor.”

This mirrored the advice of Francesca, Head of Internationalization @ Pinterest.

“When I have [the NPS] for all the markets I’m serving, I can finally start to
speak the same language that a PM is speaking.”

Oleksandr, the Localization Operations Lead @ Shopify, said: “learn the

language of your internal customers.”

There is a point where generating value for stakeholders needs to be more than
reactive. Imagine for a second that your function is an external agency hired by
your organization and that your job within it is the account manager.

If you were trying to grow the account, you would need to sell the value of your
work. That would involve not only identifying key success KPIs, but also
suggesting new lines of work. Unfortunately, sales is a part of every job – yours
too.

4. Don’t neglect the cost in your ROI calculations

It’s important to not simply advocate for l10n in a blanket way without referring
to the cost.

That’s for two reasons. First, “you can definitely get to a level with too much
complexity.” Argued Iggy, Localization Content Manager @ Deliveroo. Nearly all
interviewees tiered localization investment by country; there is no way to make
practical decisions without tracking costs.

But second, it’s vital to advocate for initiatives; and it reinforces the idea that
localization is not a commodity but an investment with a return.

Global App Testing | 2023 26

Driving Localization Leadership

Francesca, Head of Internationalization and Program Design Management @

Pinterest, described the way that they calculated cost for the decision to make
native [non-localized] content.

“We partnered with local teams and teams across the company to understand
their needs. Local teams lamented the fact that our narratives were very
US-centric. But we also have limited resources.”

“So we ran a number of pilots on some specific programs – we hired

copywriters, put together a workflow, so that we’re able to create native
content. Now we create native content for 10 markets.”

You can also calculate the opportunity cost more accurately. Iggy, Localization
Manager @ Deliveroo points out: “the opportunity cost is huge. It would be
super interesting to measure the difference between localization levels of a
new non-English spelling market. What’s the impact of not localizing or
translating any of the product?”

5. Focus on generating an international data picture

Francesca, the Head of Internationalization @ Pinterest explains the problem:

“Sometimes, you have access to very little data in localization. Dashboards are
built by the US teams for the US teams. So, if you manage to have access to an
international dashboard, that’s going to advocate for your initiatives.”

Using data, “I can corroborate my strategic instinct… if you ask to develop a

feature, the data has to corroborate what you’re asking for.”

Here’s our advice for getting the data to proceed with recommendations. 👇

Global App Testing | 2023 27

Driving Localization Leadership

Making basic, low-fiction, tactical recommendations

The next chapter will be all about recommendations. But as we come to talk
about international data, it’s useful to start with recommendations which don’t
have lots of friction.

Let’s start with something easy. The first stage of your l10n recommendation
engine should be to identify quick-wins and low-hanging fruit. Find the
lowest-friction, highest-return recommendation you can make, find a local,
short-term, high-downside problem and identify the cause.

Here’s the kind of statement we’re looking for:

“There’s low checkout completion in France”

1. The problem is local, because it seems to be specific to France

2. The problem is product-local, because it seems to be specific to the

checkout

3. It’s short-term because the checkout is not a lag indicator. Something

like churn or NPS may give a less precise indication of what could be at
fault

4. It’s high-downside because poor checkout conversion has a clear,

measurable impact on a metric somebody cares about

5. It’s a problem because it’s on the “objective” end of the

objective-to-subjective continuum. This is a mistake, a bug, or something
done poorly

Global App Testing | 2023 28

Driving Localization Leadership

The what vs the why

Many of the dashboards built by local professionals focus on the “what”, which
is to build a data picture of performance in different markets. For the “why”, it
depends a little on your business how to calculate it. For example, HubSpot
described an engaged B2B user base which were happy to act as beta testers
and would communicate their issue in some detail.

But other teams, we need a “local story” – your explanation for why a
localization activity might have an effect on a metric.

But a “local story” could be something broad, or could be something narrow.

Here’s a shortlist.

Narrow, Local functional bugs, availability issues, integration

functional issues, translation mistakes, [some] legal issues, currency
suggestions availability, overlong & truncated strings, hardware access
issues including e.g. bandwidth, WiFi

The messy Look and feel post-translation issues, insensitive media &
middle: LQA user sentiment risks, translation flow or inconsistent
translations, assumptions made during onboarding.

Broad user The product and marketing doesn’t feel like it’s designed
problems for local users, including things like US-centric case
studies & marketing.

As a side effect, we can see that the role of quality has changed. In the “narrow
functional bugs” space, localization is binary and commodified, and a quality
issue is a mistake. In the “broad user problems” area, localization is about
users, and quality is about user fit.

Global App Testing | 2023 29

Driving Localization Leadership

Tests appropriate for low-friction recommendations

Localized product flow testing

One of the most popular kinds of tests on live products that we conduct for
localization teams at Global App Testing is localized user flow testing. This is
common to businesses of all sizes, but investment and strategy-stage
businesses tend to do product flow testing as a matter of routine.

To do this, send test cases [steps a tester should follow during a test] to real
individuals in desired locations via the Global App Testing crowd. We set these
projects up as a mixture of local user survey, functional test, and LQA, in order
to give a 360 degree view of why performance is poor.

Exploratory testing

Where you’re unable to isolate the issue so precisely – e.g. to the checkout –
we’d recommend running exploratory tests. With the same mixture of survey,
functional exploratory, and LQA, you can find possible explanations for poor
metrics in a 48-hour timeframe over the weekend.

Our advice to do these well

👉 Real users in real places | It can be tempting to try to run the functional
element of this kind of QA out of an emulator. But functional bugs can be hard
to emulate. Choose QA which involves routing your test through the country or
region in question.

👉 Native speakers and cultures | To execute on the LQA element to the

highest standard, you should focus on people experienced in local language &
culture. We summarized our LQA approach in chapter 2, but be sure to find
testers who are genuinely local as well as operating locally.

❌ Avoid slow panel approaches | For a fast or iterative product environment,

panels are too slow; and they are also typically not sufficiently specialized to
give practical, actionable feedback. By using a local software crowd, you can
go faster, cheaper, and more focused than a local panel.

Global App Testing | 2023 30

Driving Localization Leadership

Key takeaways from chapter 3

1 Align to the value-add KPIs of your organization

Identifying the KPIs which make a difference to your

organization and stakeholders and express your
value-add in those terms

2 Make tactical recommendations to help people

Before you start making deep user-oriented

recommendations, start with narrow and bug-oriented
QA to find recommendations which are easy to enact.

3 Get your hands on data

It’s absolutely key to get your hands on data. Work with

analysts, build local performance dashboards, and
undertake investigative testing, to deliver the best
recommendations.

Global App Testing | 2023 31

Driving Localization Leadership

Chapter 4 | Becoming a strategic partner

Here’s a recap of what we’ll cover in chapter 4:

4 Offering strategic advice. How do you begin to start

identifying strategic opportunities?

5 Creating & influencing policy. How do you represent

local users to your organization?

6 Being a leader. Here’s how you can exercise amazing

leadership in localization

Offering strategic advice

What is a strategic partner? Here’s our idea of the difference between

investment and strategy:

An investment is transactional. A strategy is about the bigger picture.

An investment works on its own. A strategy works in coordination.

An investment is about return. A strategy is usually about users.

On strategic advice for localization, our interviewees had various things to say.

Francesca, the Head of Internationalization and Program Management @

Pinterest, said:

“The actual decision [to launch in a market] might be made at a higher level, so
if you want to have that strategic presence in the company, your language
strategy must entail something else.”

Global App Testing | 2023 32

Driving Localization Leadership

“You could [argue to] invest in content safety in an unmanaged market – or we

could help define the legal policy for data compliance in a high-risk market. We
might suggest an ad hoc [built from scratch; non-localized] app for a market
where the infrastructure is poor, but which is a big market. There’s a lot you can
do to help other teams manage risk, and it depends on how much you want to
step up and do that kind of work.”

Another angle is to create localization which is more than the sum of its parts.
An investment narrative organization might see localization activities as a
number of one-off investments, for example.

Robert, Product Program Manager @ HubSpot, said:

“Our approach [to localization] was initially quite cosmetic. I think that’s pretty
common – we were in a similar situation and businesses get focused on TAM
[Total Addressable Market] and try to get really quick wins… there’s nothing
wrong with that at all.

“One of the first pushes within the first couple of years was driving away from
the cosmetics. Towards the needs of our global customers. It stops becoming
just about content… that’s the start, but it becomes more about, “how do I
create a full experience for this particular market?”

The strategy difference

Here’s a table of how we think the “strategic” narrative is different to the

previous narratives.

Delivery Investment Strategy

Localization Commodity Activity Process of
is a differentiated by identifying policy /
depths & standards strategy &
governing

Localization A barrier to An investment in Core business

acts as releases effectiveness tenet

Global App Testing | 2023 33

Driving Localization Leadership

The L10n Cost center Revenue generator Voice of local users

team is a

You are Expense & ROI Global business

measured by speed success

Successful Run workflows Identify Understand users

teams will better opportunities better well; influence
stakeholders
successfully

“Better Fewer bugs or Improvement in local Experiences which

quality” errors metrics feel natural to users
means

You need to Solid core Understanding ROI Understanding

focus on…. materials users
Prioritizing
Differentiated localization activities Exercising
playbooks leadership
Building value
Automation & narratives User-based quality
machine insight
translation Opportunity-based
quality insight
Agile
deployment

Scope &
suppliers

Elastic LQA

Advice on strategic governance

What is strategic governance?

Global App Testing | 2023 34

Driving Localization Leadership

What percentage of your assets are non-localized; are localized; or are built
natively for a particular market, is an open question for any localization team
and depends on lots of factors. But as businesses get larger it is likely to invest
in more native content, which can create questions over who does what.

When Oleksandr, the Localization Operations Lead @ Shopify, first joined

former employer and HR SaaS software product Ceridian, he was asked to
create a l10n department and found everything was being built natively.

“There were already product managers for payroll for Germany that delivered
Germany-specific features”, he said. Instead, Oleksandr built an
internationalization team instead, advising other engineers. “My team would
show them how to do things, to create tools, and APIs, and libraries.”

Strategic governance is the process of encouraging the rest of the business to

deliver in line with best practice; and requires managing this kind of
environment.

Advice on strategic governance

One business which does strategic governance exceptionally well is Google.

Natalia, the Terminology Content Manager @ Google, described her approach
to governance, which we’ll quote at length:

“The biggest challenge in Google is scale”, said Natalia. “We put several
approaches [to governance] on the table at the same time.”

“The first is self-service. We have some processes which are automated. We

have some processes as part of the overall localization playbook – where
stakeholders can go, and know that…I find the information here, these are the
processes I need to follow, this is what I need to submit. This is the most
transactional part.”

“The second approach is partnering with the teams that have a direct influence
over the main decision makers, and that's usually marketing. Marketing have
their own playbooks – so embedding localization there is an incredible way to
reach wider audiences.

“The third approach is getting the support with, and of, any project owner. So
we start seeing what our key products are based on. What's the priority for the

Global App Testing | 2023 35

Driving Localization Leadership

organization? We identify a main point of contact. Those are going to be our

sponsors.”

What does great policy look like?

Policy tends to involve decisions made at the strategic level and then adapted
into processes; once there, policy is harder to change.

Great policy processes optimize for clarity, communication, and solid process
thinking. It might be contained in a single-source of truth [SSOT]; pinned in a
clear central place; and should embody the right level of detail that it gets read
and tells you something useful.

Great policy documentation also tends to ensure that at the moment an

employee needs the information in the policy (e.g. the correct translation of a
specialized word) the correct document is presented to them (e.g. a glossary in
a process embedded in a TMS).

What does great strategy look like

Great strategy is harder, but at its core, we believe that great strategy is about
users. Our advice for generating as rich an understanding of your users as is
practical is below.

Data for strategy: what you should have when you start

In chapter 3, we identified the following things that all businesses need when
they begin to

International performance dashboards

to represent your product performance in different countries, preferably

focused on the KPIs which drive your organization or your stakeholder teams.

- Release-oriented QA to ensure that your localized product changes are

getting released to the highest standard, preferably within an elastic

Global App Testing | 2023 36

Driving Localization Leadership

format to avoid slowing down the amazing speed which enables you to
focus on strategy.

- Some investigative and responsive QA, based on things like tight local
user metrics in areas where your product seems to be exhibiting sub-par
performance based on metrics in other markets. For this, we’d
recommend product flow testing.

Data for governance: what you should consider

As we arrive at data for governance, the approaches taken by our team are
wide-ranging.

1. Deeper sentiment analysis

Many businesses which work with Global App Testing are able to use sentiment
analysis to identify what users are saying about their brands and products, and
use that information to supplement their QA to make decisions around what to
prioritize.

If you’re a B2C brand and you don’t have access to the level of feedback that
B2B businesses use to improve their product, sentiment analysis is a good
place to start.

2. In-house expertise centers

For your larger markets, it is likely to be too thin to ask analysts or third parties
to tell you what a local user thinks. You’re going to need product managers or
local speakers on the ground in those countries to continuously assess user
values and needs.

Rober, Product Program Manager @ HubSpot, explained how this might be

more relevant in some countries than others, even after market size is taken
into account. “With Japanese, you have to have somebody who is ingrained in

Global App Testing | 2023 37

Driving Localization Leadership

Japanese culture and language to have an app which is perceived as high

quality.”

Most of our interviewees hired experts for their top relevant countries; even
where they could not afford a translator for every market they used.

3. Local user research

It’s tough to generalize about the way that top teams conduct local user
research, much of which is sensitive and highly bespoke.

With Global App Testing, there are some off-the-shelf products like qualitative
user surveys in which you can get in touch directly with users in any locale in a
48h timeframe; or think-out-loud testing.

Here’s an example of what they mean:

+ Think-out-loud testing

For think-out-loud testing, we set up a panel of local users to talk through their
use of your product as you undertake it. This gives you a chance to observe
your users in incredible detail, and get a picture of local users which your
business is likely to only process at the product level

Because of the bimodal structure of Global App Testing – with the delivery
“crowd” around the world and a team of analysts in London – we’d pull out
insights based on the finding in addition to the raw qualitative feedback.

+ Competitor benchmarking

One thing that often gets lost, even among extremely sophisticated localization
teams, is the different competitor mixes in different markets. This is because of
the tremendous complexity involved in expanding the scope of “quality” from
narrow questions like “does it work” to broad questions like “how does it sit in
the market?” – for hundreds of markets which might matter for. That’s why
Global App Testing produces detailed and rapid-turnaround competitor
benchmarking reports.

Global App Testing | 2023 38

Driving Localization Leadership

What our interviewees said about leadership

One of the recurring themes of this playbook has been leadership. A final area
we want to highlight is the role of leaders in encouraging the maturity and
growth of localization teams.

Robert, Product Program Manager @ HubSpot, described how i18n efforts were
being thought about from the top of the company. “The COO, our CEO,” were in
the room, said Robert. “Both of them were very interested in this… We had our
MD of Global Sales, and then our VP of Engineering, who was my manager at
the time.”

Oleksandr, Localization Operations Lead @ Shopify, spoke to how much weaker

leaders can be than they appear: ‘Underneath themselves, executives see
huge ambiguity. They don’t know all the details, but they trust the experts on
the respective teams. Executives have to be consistent and believe in the
mission and the goal. And continuously validate that their team understands
the goals and are aligned.’

It was Slack which really stood out to me as an example of great leadership. “I

was lucky to work with leaders who took a chance on me,” said Anca, now the
Director of Localization at Slack. “It would have made sense for them to hire a
person with experience. I was not that person.”

‘I kept proving myself, taking on more work. Maybe they just thought “she’s
figuring it out. Let’s keep it going.”

We’re excited to know generations of localization leaders at Global App Testing.

Global App Testing | 2023 39

Driving Localization Leadership

Key takeaways from chapter 4

1 Pivot to the bigger picture

Moving from a model of simple return to strategic

investment can make your department more invaluable at
a higher level

2 Gear up, start making recommendations

You can start identifying opportunities for your business

and begin to make recommendations right now.

3 Drive policy and convey leadership

Using data, you can build a strategic partnership with

multiple departments to convey ensure that localization
is a leadership department

Global App Testing | 2023 40

 Driving Localization Leadership

Our global growth toolbox

Check out our toolbox for global growth with Global App Testing.

FUNCTIONAL QA LOCALIZATION QA LOCALIZATION QA LOCALIZATION QA

Fast, thorough, Real, physical Native-speaker Cultural

localized tests availability translation bulletproofing
checks checks
Your full suite of Identify anything 48-hour translation Ensure your workflows
functional tests taken blocking your users checking by native are free from bad
to 190 countries with from using your speakers in 160 assumptions &
real users and real product in a target languages sentiment risks.
devices country

LOCALIZATION QA LOCAL USER INSIGHT LOCAL USER INSIGHT LOCAL USER INSIGHT

Post-translation Product flow Qualitative user Think-out-loud

UI cleanup assessments feedback user testing

Eliminate overlong Identify why some Ask local users Observe local users as
strings and date user flows are working whatever you like, they think aloud about
format snafus with the better than others either moderated or your product with
final stage of with a metric-oriented raw feedback format. think-out-loud testing.
localization QA local review.

SERVICES SERVICES

Competitor Insight structure

Let’s go →
benchmarking advice
Ask us about building your own toolbox.
Retain your local We can tell you how to
context with a build the ultimate local
benchmark to help insight machine based
you thrive on your business.

Global App Testing | 2023 41

Driving Localization Leadership

Information
The lead writer of this playbook was Adam Stead. Please direct any questions
to [email protected].

Some of the material in this playbook is taken from our recent interview series
“Around the World in 7 Localization Interviews” which can be found here.
Participation in the interview series does not imply endorsement of this
playbook.

If you are interested in learning more about how we can help you with your
localization strategy and implementation, get in touch here.

Global App Testing | 2023 42