Crowdsourced Localization

Tips on Launching and Testing Apps for a Global Audience

White Paper
November 2011

White Paper: Crowdsourced Localization Testing

White Paper

Crowdsourced Localization Testing

Launching and Testing Apps for a Global Audience
Table of Contents Introduction............................................................. 3 - What is Localization? ........................................... 3 - Testing Challenges ................................................ 4 - Crowdsourcing 101................................................ 5 Lessons in Localization ......................................... 7 1. Start With Why ..................................................... 7 2. Get What You Pay For ........................................ 7 3. Identify Fringe Use Cases ................................... 8 4. Think Ahead......................................................... 9 5. Mirror Real-World Conditions .............................. 10 6. Collaboration is Critical ........................................ 11 7. Call the Shots ...................................................... 11 8. Startups and Enterprises ..................................... 12 Crowdsourced L10N Testing................................. 12 Case Study .............................................................. 14 About uTest............................................................. 15

Not

surprising, our global customers have different demands of our products. We want products to feel local and we need to support features that may be unique to specific markets. - Patrick Copeland Sr. Engineering Director, Google

You might also like.

White Paper: Crowdsourced Localization Testing

Introduction
What is Localization? In the context of software development, localization also known as L10N is the process of making an application usable with a given language or culture. Perhaps more than any type of testing, localization is a mark of true success, as it signifies that your application was popular enough to warrant expansion into multiple regions. Unfortunately, its also among the most difficult QA tasks to perform, since it requires highly specialized skills from software-savvy native speakers. More on that in a second. But first, since localization is one of the least performed (and hence, least understood) types of testing, its important to first explain some basic concepts and terminology. Heres a good overview on the main localization nomenclature from Wikipedia: A localized system has been adapted or converted for use in a particular locale (other than the one it was originally developed for), including the language of the user interface (UI), input, and display, and features such as time/date display and currency. Each instance of the system only supports a single locale, and there is no explicit support for languages that are not part of that locale Multilingualized software supports multiple languages for display and input, but has a single UI language which cannot be changed after installation of the software. Multi-locale support for other features like date, time, number, and currency formats varies as the system tends towards full internationalization. An internationalized system is equipped for use in a range of "locales" (or by users of multiple languages), by allowing the co-existence of several languages and character sets for input, display, and UI. In particular, a system may not be considered internationalized in the fullest sense unless the UI language is selectable by the user at runtime.

In each case, one commonality exists: youre launching an application outside of your comfort zone - literally! So how should you go about testing the context and accuracy of your localized product? Well, thats the subject of this brief whitepaper. Written for companies of all shapes and sizes, this whitepaper will outline some battletested strategies for achieving localization testing across a wide spectrum of use cases. Well provide key tips from localization veterans (i.e. companies like Google, Microsoft and Mozilla), but well also highlight several also real-world examples from smaller firms establishing a global presence. Lets get started

White Paper: Crowdsourced Localization Testing

Localization Testing Challenges Lets say, hypothetically, that youve just launched a Mandarin version of your mobile app. Like the rest of your applications, this one was developed and tested using inhouse resources. But for this assignment, you needed a native Mandarin speaker to confirm that nothing was lost in translation. Luckily, Jenny in the marketing departments speaks, reads and writes fluent Mandarin - problem solved. Right? Not so fast! While translation validation is indeed part of the localization process, its not the only part. First, is Jenny fluent enough in Mandarin to fully understand metaphors, similes and slang? Will Jenny be pulled from her day-to-day job to validate every content revision throughout the SDLC? Can Jenny vet every part of your localized application including your mainstream app content, your error messages, the system-generated emails your app sends? Is Jenny really going to localize ever-changing content like comments, message boards and ratings (sounds like Jennys got herself a new full-time job)? And what about when the app is launched in German, French or Portuguese (including Brazil vs. Portugal) are you really going to become Rosetta Stones best customer in order to keep up?

Assuming you (or your CFO) have already rejected the idea of hiring an in-house team of linguists, what other options do you have? Free translation tools while innovative are often overtly literal and ill-equipped to handle localization on such a huge scale. Other options include outside consulting services, which specialize in localizing your content, but often lack an effective third-party mechanism to validate their own work. Aside from matters of expertise, cost and personnel, theres that small issue of localization testing criteria, which includes: Content Static & dynamic content like catalogs, search results, metadata, etc. Dates Is the date January 1 or 1 January? Characters Diffrent lnguages have ifferent set f characters Postal codes In some countries, postal codes contain letters Phone numbers Different formats for different markets Direction Some languages are written left to right, others are right to left: Currency conversion Especially important for internet retailers Tax calculation VAT, sales tax and others vary from country to country

White Paper: Crowdsourced Localization Testing

So now that you have a better idea of what youre up against, how can you expect to build an in-house staff of software-savvy linguists? Answer: Most companies cant - at least not without some serious investments in both money and time. But what you can do to overcome the challenge of localization is to leverage a crowdsourced community one that includes native speakers who live in-market all around the globe.

What is Crowdsourcing?
Traditionally, decision-makers in the software industry (in fact, decision-makers in all industries) had two basic options for completing assigned tasks: through an in-house staff, or with an outsourcing firm. With the introduction of crowdsourcing, theres now a viable third option one thats much less costly and time-consuming. Heres the formal definition of the term, as coined by journalist Jeff Howe in his book, Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business: [Crowdsourcing is]the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call. Whats the significance? Well for one, it enables you to make best decisions on a perproject basis - which comes in especially handy in terms of localization projects, since they are often times too infrequent to warrant building out an entire staff of linguists. Software teams can now build in-house expertise while maintaining the operational flexibility that comes with external providers. The task and opportunity for executives today is to get the most bang for their buck by building in-house strengths around key employees and core competencies, while also tapping into variable resources like crowdsourcing.

Any darn fool can make something

complex; it takes a genius to make something simple. - Albert Einstein

Key Benefits from Crowdsourced Localization The numerous benefits that can be achieved when adopting crowdsourcing for localization testing include: Cost containment: Crowdsourcing allows managers to utilize lower-cost outside support without being tied down by the long-term commitments of outsourcing firms. The fundamental structure of online communities creates competition, making it more accountable and cost-effective than traditional outsourcing.

White Paper: Crowdsourced Localization Testing

Dealing with increased complexity: Significant quality improvement is achieved when localization testing is done across a wider set of platforms, locations and languages. Localization testers native speakers who live in your targeted market can be easily recruited on a per-project basis. Faster time to market: Since localization testing tends to occur during the latter phases of the SDLC a time when deadlines are fast-approaching crowdsourcing can alleviate delays that often stifle companies around peak release times. Meeting expectations for higher quality: Todays customers no matter where they are expect apps to be complete and functional from day one. The era of beta-testing in foreign markets as a substitute for QA is long gone. Keeping your customers (and converting your prospects): Companies who venture into localization often have trouble retaining users outside of their home market. By crowdsourcing the localization process, you get a first hand, more indepth look at the relative strengths and weakness of your application. Empowering the user: Users that feel that they have a hand in the success of a product will have a stronger emotional commitment to the product and the company, creating powerful connections and a potential army of fans.

To summarize, only crowdsourcing meets the in-the-wild contextual demands of localization. By utilizing a community of diverse and talented professional testers, crowdsourced testing provides companies with native speakers worldwide to run complete localization: validating any translated website, product or app; verifying accuracy and context across currency conversions, dates, special characters and static content; as well as other trouble areas of localized products. With that in mind, lets take a look at some key lessons to ensure you get the most out of your crowdsourcing efforts.

White Paper: Crowdsourced Localization Testing

Lessons in Crowdsourced Localization Testing

Lesson #1: Start With Why In a lot of ways, testing is about finding (and potentially fixing) what doesnt work. Testers seek out problems because nobody wants to use or sell buggy software. But with localization testing, theres an added reason to care: youre protecting the very things that are valuable to an entire culture of people. Cultural identity is an extraordinarily important component of who we are, and bugs that impact cultural identity can often times be the most frustrating. For example, take the Cherokee tribe of Native Americans. Like many Native American tribes, their community was displaced by the advancement and settling of Europeans. However, one of their members, a man named Sequoyah, had the foresight to invent a written system for the Cherokee language. With that system, the Cherokee were able to keep detailed records of their history and communicate with other members of their tribe who were now spread across the country on different reservations. Today, Cherokee is spoken by thousands of people in the United States, and it still uses Sequoyahs complicated writing system. Incredibly, its possible today for Cherokees to communicate in their language using any modern communication device, including iPads and iPhones. Thats because Unicode includes support for the Cherokee syllabary ( ), and any modern device with good Unicode support can handle the Cherokee language. So why do we care about localization? Because entire cultures with thousands of years of history rely on being able to communicate with each other. Testing is crucial for more reasons than just making sure an app works. Testing is crucial because an entire culture may depend on your app to communicate. As such, your testing team needs to be a diverse as humanly possible. Lesson #2: Get What You Pay For Crowdsourcing has been viewed by many critics as little more than a cost-cutting measure. Due to the contest-based, winner-take-all nature of many crowdsourcing initiatives, some decision-makers have come to regard this method as an easy way to extract value from the masses while only compensating a select few, usually for pennies on the dollar if at all. In reality, crowdsourcing allows companies to utilize lower-cost outside support without being tied down by long-term commitments (aka traditional outsourcing). The fundamental structure of online communities creates competition, making it more accountable, efficient and cost-effective than traditional outsourcing. So while it is true that crowdsourcing has done much to lower costs in certain areas, this does NOT equate to it being free.

White Paper: Crowdsourced Localization Testing

Crowdsourcing expert Brent Fei makes an important distinction between these two forms of crowdsourcing: The novelty of free crowdsourcing (getting something useful done by the masses for nothing) has been the romantic aspect most often studied and written about. Free differs from paid crowdsourcing in that free work gets accomplished only if its entertaining, emotionally fulfilling, or leads to recognition. The less sexy and decidedly more complex paid cousin, which uses money as leverage to generate results, has been more of a mystery. Ask 10 business managers whether theyve used on-demand workers through an online service, and nine will cock their head like a lab that just heard a dog whistle. As mentioned previously, localization requires a highly specific set of skills. So while you can take advantage of lower costs (compared to the alternatives), dont expect to get something for nothing. Lesson #3: Identify Fringe Use-Cases Fringe use cases (i.e. unexpected scenarios) occur more in the localization than perhaps any other type of testing. Take for instance, the case of localized URLs. Up until last year, Internet domain names were a pretty mature business. Then the folks at ICANN decided to shake things up by enabling non-Latin character ccTLDs (country code Top Level Domains like .co.il and .co.uk ). What does that mean for you? Well, heres a quick test. Try visiting this URL: http://. . /-. What youre looking at is an Internationalized Domain Name, or IDN for short. It doesnt contain western or Latin letters, and chances are everything you know about URLs is about to get turned backwards (literally, in this case). Whats worse is that different browsers handle this kind of domain name differently, and theres no one single solution. Are you a software tester? Then your ship has come in because IDNs open up a whole new category of software bugs. Lets take a look at a few big trouble areas, but hang on tight because this gets goofy fast. Right to Left TLDs Take a look at the URL in the second paragraph (which goes to the Egyptian Ministry for Communications and Information Technology). After the http:// youll see the Misr (Egypt) TLD, followed by a period, and then the domain name. This makes sense because Arabic is written right-to-left, but it would be like reading

White Paper: Crowdsourced Localization Testing

the BBCs URL as http://uk.co.bbc.www. Of course, you cant write out any old URL right-to-left just those from certain languages. Which means that when it comes to parsing domain names, figuring out the language is an important first step to knowing whether the TLD comes first or last. New Opportunities for Phishing The next problem is even worse, and theres no good solution. If you open the first URL in Firefox, youll notice that the URL bar shows it as a long string of Arabic text. Safari, on the other hand, displays it properly. Why does Firefox break the URL? Because IDNs have the potential to be very dangerous for web security and phishing. As more languages are approved for IDNs by ICANN, the number of valid character sets will grow. This introduces conflicts with international characters that look very similar to Latin characters. Conclusion Software testing just got a lot more complicated, but here are a few ideas to get you started with IDNs:

First, does it matter if the app handles IDNs at all? Not every app cares about URLs. Use good judgment before testing. Next, does the app handle international URLs with IDNs correctly? Does the web app handle right-to-left domain names correctly? Again, use the URL above as a test. How does the app handle domains that could be phishing targets? Is the app able to differentiate between the international and Latin versions of a domain?

Instances like this while they might be considered fringe are nevertheless an important part of any localization test cycle. Lesson #4: Think Ahead Crowdsourcing is seen as a quick and easy solution to a temporary problem, such as the need to solicit feedback on a recently launched website or generating ideas for an upcoming marketing campaign. Because of the on-demand nature of crowdsourcing, it is rarely considered to be a legitimate, long-term solution in the business world. In terms of localization, however, this couldnt be further from the truth. When crowdsourcing first came on to the scene, many companies were hesitant to even consider using it, let alone making it a core component of their business strategy. But as crowdsourcing has evolved, so too have the perceptions of upper and middle management. No longer considered a one-off solution, companies now engage the crowd on a regular basis.

White Paper: Crowdsourced Localization Testing

In fact, a common denominator among successful crowdsourcing operations is long-term engagement on the part of the community. It is a mistake for a company to leverage crowdsourcing once before exiting. The real benefits are only realized after the crowd becomes familiar with your business and its goals. Crowdsourcing scholar Ankit Sharma explains: It is critical that the crowd is viewed as a partner in the initiative. The needs, aspirations, motivations and incentives of the crowd to participate in the initiative must remain the most important consideration while developing the crowdsourcing initiative. The practitioners must understand the crowd motivation and align their goals according to it. This is especially true in the context of localization, where the more testers can interact with your application, the higher quality results youre likely to receive. Lesson #5: Mirror Real-World Conditions When Possible Forget the fact that you are paying these people to write test cases or find bugs. While thats the primary job, a secondary (but powerful) benefit is that your product is being validated in the hands and minds of real-world users in real conditions. Select members of the community that best match your customer base: by location, language and expertise. The testers that work on your app should match the users you hope to target.

Free Tools for Localization

Are you writing a web application? Do you have a global audience? Then you probably care deeply about localizing or internationalizing your application so it works smoothly for all your customers. What you want is to make sure your application as well as your web server stack supports UTF-8. Fortunately, almost every modern component involved in building a web application supports UTF-8. Unfortunately, getting each individual component to use UTF-8 requires a lot of configuration kicking and screaming. This article by Jonathan Rentzsch describes all the steps to getting a typical LAMP stack working with UTF-8, end-to-end. Jonathan recommends starting with two important tools: a hex editor (so you can see the raw data exactly as its stored) and a Unicode test string that can test to see if an article is being stored and shown with correct UTF-8 encodings. For the test string, he proposes (and this is where we see if our config is correct): Itrntinliztin including the curly quotes. The rest of his post includes configuration tips for MySQL, Java and others.

10

White Paper: Crowdsourced Localization Testing

Lesson #6: Collaboration is Critical When selecting a community to work on your products, choose one that will work closely with your in-house specialists. Some communities tend to be noisy, with lots of opinionating, but not much actual productive work. Choose a community that has proven success and exposes the past performance and reputation of community members. Also look for those communities that can point to clear, demonstrable wins other customers that theyve worked with to solve real business problems. And what about your in-house experts? What role should they play in the process of crowdsourced localization? Heres a great tip from the writers at Going Social Now: Anytime you run a crowd-sourcing initiative, it is important to connect the employees who are normally tasked with the jobs that are being crowdsourced with the participants. This is for three reasons. Firstly, the employees have valuable insights to share and can guide the process to better solutions. Secondly, the employees will be more accepting of the feedback if they are a part of the process start to finish. And thirdly, it'll show the participants that the brand is a more human one with real people trying to solve the very same problems. If expanding into multiple regions, it would be wise to assign one in-house employee to oversee the crowdsourced community in each region. Whatever direction you decide on, be sure to keep internal stakeholders involved throughout the process. Lesson #7: Call the Shots Due to the open call nature of crowdsourcing - where virtually everyone has the opportunity to participate - the practice is regarded by many as a chaotic free-for-all with a high signal-to-noise ratio. As a result, this affords the project owner little control over the process. To find a valuable contribution requires that one sort through hundreds of worthless submissions, right? Wrong! Crowdsourcing does not change a fundamental truth: Effective, detailed communication and project management are key to any successful project. This is true in managing in-house resources or outsourced partners. Crowdsourcing is no exception. In fact, many companies today have assigned an internal project owner to manage the crowd and keep the information flowing. While you dont always have to micro-manage the crowd in terms of tactical execution and idea generation, strong management and executive buy-in enables processes, plans and deadlines to still remain firm. Crowdsourcing writer Allen Day nails it in 8 Keys to Effective Crowdsourcing: Know what you expect the worker to do for you. Make each task so simple that its virtually impossible for a worker to do it incorrectly. Break up complex tasks into their most elementary pieces. Ideally one task = one decision. Make each task closed-ended. Do not leave any room for ambiguity.

11

White Paper: Crowdsourced Localization Testing

Lesson #8: From Startups to Enterprises Though crowdsourcing may be thought of as a solution for startups and small businesses, we find that many large organizations are also turning to the crowd for localization testing. Thoughts from Googles Sr. Engineering Director, Patrick Copeland: Not surprising, our global customers have different demands of our products. We want products to feel local and we need to support features that may be unique to specific markets. For instance, in Indic-based languages using a standard keyboard is difficult, so we develop strategies like virtual keyboards or category browsing for search. As we specialize our products for certain markets, it introduces more challenges for testing (eg. requiring special cultural knowledge). When we cant find internal talent, community-based testing is an interesting solution to this challenge. Thoughts from Mozilla QA Director, Matt Evans: I think the biggest misconception is that it is all lumped under one label and that crowdsourcing is done according to some crowdsourcing manual or standard. There is such a big spectrum of crowdsourced project implementations and it isnt correct to have crowdsourcing and community-driven projects under the same umbrella. Strict crowdsourced projects tend to be targeted towards tasks that are discrete and distributed in nature. Community-based approaches tend to be longer lived projects and are typically for public benefit. Membership is considered a privilege and betterment of the community as a whole and is usually at the top of the list of shared goals amongst the members of the community project. In addition, you find the rise of leaders within community projects that drive the project forward. Mozilla certainly is an example of a community-based project. Many people would be surprised at our mission statement: To promote openness, innovation and opportunity on the web. It just so happens that building a world-class competitive browser helps us achieve the mission. Community citizenship and passion for the project has been the key factors for success. So is crowdsourcing the right solution to your localization testing challenges? If so, the next section will help you get started. uTest: Crowdsourced Localization Testing Because every countrys language, culture, currency, taxes and standards are different, its imperative for companies to test how well their applications perform within different aspects of international culture. However, unlike some other forms of testing, the process of testing and the knowledge required is highly specialized.

12

White Paper: Crowdsourced Localization Testing

With the help of a dedicated uTest project manager and team lead for each language (or market), companies can run complete localization testing of their products and applications, verifying accuracy and context across currency conversions, dates, characters and other trouble areas of localized product: uTests services include:

Translation Validation: With native speakers in more than 180 countries worldwide, uTest can select a custom team of testers - located in your specific market - to validate any recently translated website, application or product. Full L10N Testing: Testers from virtually any market or region will review your application for context and accuracy across all aspects of the localization process, including static content, special characters and other areas.

For more information, including details on pricing and deliverables, visit: www.utest.com/localization-testing

13

White Paper: Crowdsourced Localization Testing

Appendix: ReviverSoft Crowdsources Localization Testing (Case Study) When web-performance leader ReviverSoft localized its website and product into 22 languages, the company chose a major localization services company to do the translation work. After their website and product had been localized, Mark Beare the companys Founding Partner needed a way to test the accuracy of these changes. He decided on an independent 3rd party to validate the accuracy and completeness of the localization efforts. Localization is a very complex task, explained Mark. Theres a lot of room for error. You have to make sure the translations are accurate and contextual. Since none of our staff is located in Germany, France and many other countries where we were offering our products at the time, crowdsourcing was an ideal solution. For the same reason that writers shouldnt edit their own work, translators should not review their own translations, so Mark adopted crowdsourcing to find native speakers who live in-market to ensure ReviverSofts apps were ready for localized success focusing on formatting, dates, punctuation, currency, etc. We really needed native speakers for this assignment, said Mark. These are the countries where were selling our products, so it was extremely important that the material was accurate and made sense. With the help of his dedicated project manager, Mark and his team would assemble a team of diverse testers in terms of language and location. With the setup complete, this team of experts would spend the next several days reviewing the various localized sites, with a particular focus on the German, French, Danish and Japanese versions. We had been sending a lot of long files to the translators, so you never know when text is going to be cut off in certain areas of the site, since length often varies, he explained. We needed to make sure that the text rendered correctly, and we needed to know where text should be changed to make it more relevant. Within a matter of days, Mark and his team had received complete feedback on all of their recently translated versions. What we found was that, for the most part, testers were able to use the software with the translations provided, so that was good confirmation, said Mark. We did, however, find a few issues where the text was not rendered correctly and where certain content needed to be changed completely. Without some of that feedback, our product instructions would not have made a lot any sense to some people. Were a young company, so it didnt make sense for us to go out and hire a full team of testers right away, since they wouldnt have had a whole lot to do, explained Mark. For us, crowdsourcing was a much quicker, much more efficient way of performing our testing.

14

White Paper: Crowdsourced Localization Testing

About uTest uTest provides in-the-wild testing services that span the entire software development lifecycle including functional, security, load, localization and usability testing. The companys community of 70,000+ professional testers from 190 countries put web, mobile and desktop applications through their paces by testing on real devices under real-world conditions. Thousands of companies -- from startups to industry-leading brands rely on uTest as a critical component of their testing processes for fast, reliable, and cost-effective testing results. More info is available at www.utest.com or blog.utest.com, or you can watch a brief online demo at www.utest.com/demo.

uTest, Inc. 153 Cordaville Road Southborough, MA 01772 p: 1.800.445.3914 e: [email protected] w: www.utest.com

Case Study Case Study

15