Background

Cultural Differences

Localization is about more than cosmetic differences: forget whether the year comes before or after the month in a date, basic beliefs about how the world works vary significantly

1. Geography's pretty universal, right?

   • Places have only one official name
   • Place names follow the character rules of the language
   • Place names can be written with the usual character set of a country

2. Well, what about someone's name?

   1. People have exactly one canonical full name.
   2. People have exactly one full name which they go by.
   7. People’s names do not change.
   8. People’s names change, but only at a certain enumerated set of events.
   20. People have last names, family names, or anything else which is shared by folks recognized as their relatives.

3. What about something as simple as a person's gender? That's just biology, right?

   • There are two and only two genders
   • Okay, then there are two and only two biological genders.
   • Gender is determined solely by biology.

Dealing with Cultural Differences

• The easiest way to spend less time dealing with complex data is not to ask for it: do you really need to know your users' gender? This is also a good way to avoid your signup process feeling nosy
• If you do need data, ask how much structure you need: a simple “What name should we display on your profile?” field is easy to build and much easier than trying to migrate a simplistic system after it's full of user data
• If you have to model something complicated, pay the cost upfront: use a library or service, test major assumptions and potential outliers regularly and think about how you'll deal with problems

Writing

An Abbreviated History of Electronic Text

When people started building electronic communication systems, it was easy to continue assigning each distinct character a number. Since early systems needed to be simple each character was assigned a fixed-length binary number

• Baudot code (1870) to ITA2 (1930): 5 bits - just enough for the English alphabet
• TeleTypeSetter and BCD (1928): 6 bits allowed punctuation. Unfortunately, different manufacturers used different schemes, making it difficult to exchange data or even mix computers and printers from different manufacturers!
• ASCII (1963): 7 bits allow both upper and lower case! Standardization should also help avoid painful conversion issues between manufacturers…

• … but almost everyone outside the United States needs more characters and uses 8-bits to store extended characters beyond basic ASCII. Worse, it's frequently possible to exchange text incorrectly until someone notices the first document using one of the different characters!

  Since there are individual languages which need more than 256 characters, there's no possibility of a standard 8-bit encoding emerging

The Range of Human Writing

Unicode

Starting in the 1980s, engineers from various companies started working on an ambitious project: a universal 16-bit character set which could represent every character used in human writing. At some point it expanded beyond 16 bits but the goal hasn't changed

Terminology

Collation

Determining how strings compare for the purposes of sorting

Directionality

Unicode Encodings

The Unicode standard describes abstract characters but we need a way to convert them into bytes for storage and exchange. Early experiments which simply doubled 8-bit ASCII to 16-bits revealed significant problems:

1. Not every processor stores the bytes comprising a 16-bit integer the same way — big-endian (“UNIX”) or little-endian (“NUXI”) — necessitating a special byte-order mark (BOM) at the start of the string simply to decode it
2. All existing text would need to be converted!
3. All text becomes more expensive to store and process
4. It wasn't enough: Chinese alone would require at least 3 bytes!

UTF-8 was developed to avoid these problems. It's a very clever variable-length encoding scheme under which all existing 7-bit ASCII is valid, all common non-Asian characters require only 2 bytes, common CJK still needs only 3-bytes, using 4 bytes only for rare and historical characters. Because it's read one byte at a time, there's no need for a BOM.

State of Unicode

• In 2008, Google announced that for web content UTF-8 had surpassed ASCII in popularity. In 2010, it was approaching 50%, and as of February 2012 it has passed 60%

• All major operating systems and programming languages support Unicode and UTF-8, although compatibility is still a consideration for the more recent features added in version 6 such as emoji (🌏) or regional indicators ( 🇺 🇸 = 🇺🇸, 🇫 🇷 = 🇫🇷, etc.)

Complex scripts

We previously discussed how accented characters can be formed by combining a base character with the desired diacritic. The concept of multiple characters producing a visually distinct glyph is relatively unusual in English, where only a few ligatures are at all commonly used - perhaps the best known being the “ae” in encylopædia - but other languages depend on this behaviour.

A complex text layout system allows the visual display to be significantly altered based on the context. If you need to support complex languages this will affect your font choices and design options!