Google Translate and r?maji

The following is a guest post by Professor J. Marshall Unger of the Ohio State University’s Department of East Asian Languages and Literatures.

The challenge

On 18 November 2009, Mark Swofford posted an item on his website pinyin.info criticizing the way Google Translate produces Hanyu Pinyin from standard Chinese text. He concluded by saying, “Google Translate will also romanize Japanese texts written in kanji and kana, Russian texts written in Cyrillic, etc. But I’ll leave those to others to analyze.” So I decided to take up Swofford’s challenge as it pertains to Japanese. Using Google Translate, I romanized a news item from the Asahi of 6 December 2009:

Original Google Translate
???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? roku nichi gogo yon ji san go fun goro , t?ky? to chiyoda ku k?kyogaien no tod? ? uchibori d?ri ? no nij?bashi zen k?saten de , ch?goku kara no kank? kyaku no yon zero dai no dansei ga j?y?sha ni hane rare , zenshin wo tsuyoku u~tsu te mamonaku shib? shi ta . kuruma wa hod? ni noriage te arui te i ta dansei ? roku ky? ? mo hane , dansei wa atama wo tsuyoku u~tsu te ishiki fumei no j?tai . marunouchi sho wa , unten shi te i ta t?ky? to minato ku hakkin san ch?me , kaisha yakuin takahashi nobe tsubuse y?gi sha ? ni yon ? wo jid?sha unten kashitsu sh?gai no utagai de genk? han taiho shi , y?gi wo d? chishi ni kirikae te shirabe te iru .
???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? d?sho ni yoru to , shib? shi ta dansei wa ?dan hod? wo arui te wata~tsu te i ta tokoro wo chokushin shi te ki ta kuruma ni hane rare ta . kuruma wa hidari ni ky? handoru wo kiri , shad? to hod? no sakai ni oka re ta kasetsu no saku wo haneage , hod? ni noriage ta to iu . saku wa hod? de ran’ningu wo shi te i ta dansei ? san yon ? ni atari , dansei wa ry?ashi ni karui kega .
???????????????????????????????????????????? d?sho wa , shib? shi ta dansei no mimoto kakunin wo susumeru totomoni , t?ji no k?saten no shing? no j?ky? wo shirabe te iru .
????????????????????????????????????????? genba sh?hen wa t?ky? kank? no supotto no hitotsu da ga , saikin wa jogingu wo tanoshimu hito mo fue te iru .

Google’s romanization algorithm does a thoroughly mediocre job compared with what a human transcriber would do. To see this, compare the following:

Google Translate human transcriber
roku nichi gogo yon ji san go fun goro , t?ky? to chiyoda ku k?kyogaien no tod? ? uchibori d?ri ? no nij?bashi zen k?saten de , ch?goku kara no kank? kyaku no yon zero dai no dansei ga j?y?sha ni hane rare , zenshin wo tsuyoku u~tsu te mamonaku shib? shi ta . kuruma wa hod? ni noriage te arui te i ta dansei ? roku ky? ? mo hane , dansei wa atama wo tsuyoku u~tsu te ishiki fumei no j?tai . marunouchi sho wa , unten shi te i ta t?ky? to minato ku hakkin san ch?me , kaisha yakuin takahashi nobe tsubuse y?gi sha ? ni yon ? wo jid?sha unten kashitsu sh?gai no utagai de genk? han taiho shi , y?gi wo d? chishi ni kirikae te shirabe te iru . Muika gogo yo-ji sanj?go-fun goro, T?ky?-to Chiyoda-ku K?kyo Gaien no tod? (Uchibori d?ri) no Nij?bashi-zen k?saten de, Ch?goku kara no kank?-kyaku no yonj?-dai no dansei ga j?y?sha ni hanerare, zenshin o tsuyoku utte mamonaku shib?-shita. Kuruma wa hod? ni noriagete aruite ita dansei (rokuj?ky?) mo hane, dansei wa atama o tsuyoku utte ishiki fumei no j?tai. Marunouchi-sho wa, unten-shite ita T?ky?-to Minato-ku Shirogane san-ch?me, kaisha yakuin Takahashi Nobuhiro y?gisha (nij?yon) o jid?sha unten kashitsu sh?gai no utagai de genk?han taiho-shi, y?gi o d?-chishi ni kirikaete shirabete iru.
d?sho ni yoru to , shib? shi ta dansei wa ?dan hod? wo arui te wata~tsu te i ta tokoro wo chokushin shi te ki ta kuruma ni hane rare ta . kuruma wa hidari ni ky? handoru wo kiri , shad? to hod? no sakai ni oka re ta kasetsu no saku wo haneage , hod? ni noriage ta to iu . saku wa hod? de ran’ningu wo shi te i ta dansei ? san yon ? ni atari , dansei wa ry?ashi ni karui kega . D?-sho ni yoru to, shib?-shita dansei wa ?dan hod? o aruite watatte ita tokoro o chokushin-shite kita kuruma ni hanerareta. Kuruma wa hidari ni ky?-handoru o kiri, shad? to hod? no sakai ni okareta kasetsu no saku o haneage, hod? ni noriageta to iu. Saku wa hod? de ranningu o shite ita dansei (sanj?yon) ni atari, dansei wa ry?ashi ni karui kega.
d?sho wa , shib? shi ta dansei no mimoto kakunin wo susumeru totomoni , t?ji no k?saten no shing? no j?ky? wo shirabe te iru . D?-sho wa, shib?-shita dansei no mimoto kakunin o susumeru to tomo ni, t?ji no k?saten no shing? no j?ky? o shirabete iru.
genba sh?hen wa t?ky? kank? no supotto no hitotsu da ga , saikin wa jogingu wo tanoshimu hito mo fue te iru . Genba sh?hen wa T?ky? kank? no supotto no hitotsu da ga, saikin wa jogingu o tanoshimu hito mo fuete iru.

For the sake of comparison, I have retained Google’s Hepburn-style romanization. The following changes have been made in the text in the righthand column:

  1. Misread words have been rewritten. Many involve numerals; e.g. muika for “roku nichi”, yo-ji for “yon ji”, sanj?go-fun for “san go fun”. The personal name Nobuhiro is an educated guess, but “Nobetsubuse” is certainly wrong. Shirogane for “hakkin” is a place-name (N.B. Google did not produce *hakukin, indicating that the algorithm does more than just character-by-character on-yomi).
  2. False spaces and consequent misreadings have been eliminated. E.g. hanerare for “hane rare”, wattate ita for “wata~tsu te i ta”.
  3. Run-together phrases have been parsed correctly. E.g. to tomo ni for “totomoni”.
  4. Capitalization of proper nouns and the first words in sentences has been introduced.
  5. Hyphens are used conservatively for prefixes and suffixes, and for compound verbs with suru.
  6. Obsolete “wo” for the particle o has been eliminated. (N.B. Google did not produce *ha for the particle wa, so “wo” for o is just the result of laziness.)
  7. Apostrophes after n to indicate mora nasals in positions where they are not needed have been eliminated.
  8. Punctuation has been normalized to match for romanized format and paragraph indentations have been restored.

One could make the romanized text more easily readable by restoring arabic numerals, italicizing gairaigo, and so on. Of course, if the reporter knew that his/her copy would be reported orally or in romanization, s/he might have chosen different wording to avoid homophonic ambiguities. E.g., Marunouchi-sho could be Marunouchi Keisatsu-sho, though perhaps in the context of a traffic accident story, it is obvious that the suffix sho denotes ‘police station’. Furthermore, in a digraphic Japan, homophones might not be such as great problem. If, for instance, readers were accumstomed to seeing d?sho for ?? ‘same place’, then d?-sho would immediately signal that something different was meant, which, given context, might be entirely sufficient to eliminate misunderstanding.

But having said all that, my guess is that the romanization function of Google Translate was programmed with some care. Rather than criticize the quality Google’s algorithm, I suggest pursuing the logical consequences of assuming that it deserves about a B+ by current standards.

Analysis

Clearly, there is a vast amount of knowledge an editor needs if s/he wants to bring Google’s result up to an acceptable level of romanization for human consumption. That minimal level, in turn, is probably a far cry from what a committee of linguists might decide would be an ideal romanization for daily use in 21st-century Japan. It is quite obvious why Google’s algorithm blunders — the reasons were well understood and described long ago (e.g. in Unger 1987) — and though the algorithm can be improved, it can never produce perfect results. Computers cannot read minds, and mindreading is ultimately what it would take to produce a flawless romanization.1

Furthermore, imagine the representation of the words of the text that presumably takes shape in some form or other in the mind of the skilled reader of the original text. Given that Google’s programmers are doing their best to get their computers to identify words and their forms from Japanese textual data, it is clear that readers, who achieve excellent comprehension with little or no conscious effort, must be doing vastly more. The sequence of stages — from (1) the original text to (2) the Google transcription, (3) the better edited version, (4) some future “ideal” romanization scheme, and onward to (5) whatever the brain of the skilled reader ultimately distills and comprehends — concretely illustrates how, at each stage, different kinds of information — from the easily programmable to genuine expert knowledge — must be brought to bear on the raw data.

Of course, something similar can be said of English texts as well: like Chinese characters, orthographic words of English, even though written with letters of the roman alphabet, typically function both logographically and phonographically. The English reader has to do some work too. But how much? Think of the sequence of stages just described in reverse order. The step from the mind of an expert reader (5) to an ideal romanization (4) is short compared with the distance down to the crude level of romanization produced by Google Translate (2). Yet Google does quite a bit relative to the original text (1). It does not totally fail, but rather makes mistakes, which, as just demonstrated, a human editor can identify and correct. It manages to find many word boundaries and no doubt could do better if the company’s programmers consulted some linguists and exerted themselves more. The point is that Japanese readers must cover the whole distance from the text to genuine comprehension, a distance that must be much greater than that traversed by the practiced reader of English, for all its quaint anachronistic spellings. With a decent, standardized roman orthography, the Japanese reader would have a considerably shorter distance to negotiate.

Note

  1. Indeed, starting in the 1980s, Asahi pioneered in the use of an IBM-designed system called NELSON (New Editing and Layout System of Newspapers) that uses large-array keyboards (descriptive input) rather than the sort of kanji henkan methods (transcriptive input) common on personal computers and dedicated word-processing systems. Consequently, the expedient of storing the underlying roman or kana input stream alongside the selected characters is not available for Asahi stories. Of course, such information is routinely thrown away by many other input systems too.

Journal issue focuses on romanization

cover of this issue of the Journal of the Royal Asiatic SocietyThe most recent issue of the Journal of the Royal Asiatic Society of Great Britain and Ireland (third series, volume 20, part 1, January 2010) features the following articles on romanization movements and script reforms.

  • Editorial Introduction: Romanisation in Comparative Perspective, by ?lker Aytürk
  • The Literati and the Letters: A Few Words on the Turkish Alphabet Reform, by Laurent Mignon
  • Alphabet Reform in the Six Independent ex-Soviet Muslim Republics, by Jacob M. Landau
  • Politics of Romanisation in Azerbaijan (1921–1992), by Ayça Ergun
  • Romanisation in Uzbekistan Past and Present, by Mehmet Uzman
  • Romanisation of Bengali and Other Indian Scripts, by Dennis Kurzon
  • The R?maji movement in Japan, by Nanette Gottlieb
  • Postscript from the JRAS Editor, Sarah Ansari

Unfortunately, none of these cover any Sinitic languages or the case of Vietnam. And Gottlieb’s take on r?maji is certainly more conservative than Unger’s. But I expect this will all make for interesting reading.

I am able to view all of the articles on my system. But perhaps others will run up against a subscription wall.

I thank Victor H. Mair for drawing this publication to my attention.

that horizontal feeling

The Yomiuri Shimbun reports, “A series of classic works by renowned novelists is proving popular due to innovative designs and the fact the text is printed using lateral text rather than the vertical columns usually used for Japanese novels.”

The first two books in the Meisaku Bungaku (Masterpiece Literature) series are single volume editions of Soseki Natsume’s “Kokoro” (Heart) and Osamu Dazai’s “Ningen Shikkaku” (No Longer Human), both published on Aug. 1.

The venture by the publisher, Goma Books, is aimed at getting young people to read classic fiction in a similar manner to the way they read novels on mobile phones.

The two books feature photographs of actresses on their front covers, and the type is not the usual black, but features colors such as orange and bright green to give the books a casual feel. Such designs, coupled with the horizontal text, have helped the publisher sell more than 50,000 copies of the novels since they were put on sale.

The two books were among 60 novels made available on the Goma Books mobile phone Web site in April last year. They were selected due to their great popularity.

Copyright on all the site’s books has expired because at least 50 years have passed since the death of their authors.

Some site users said they found it easy to read the masterpieces when they were written horizontally rather than vertically. The site attracts about 100 million hits a month, prompting the publisher to put out printed forms of the works.

As well as the switch from vertical to horizontal text, other ideas also were adopted.

Reading ease was taken into account, with the publisher using fewer words per page and more space between lines. Kana syllables are also frequently printed alongside kanji to aid readers.

My favorite bit, in part because I wonder if the first sentence had ever been uttered before, comes next. Or is this a topic that has been hotly debated among the Japanese literati?

“The emotions [of the work] are not lost with lateral writing,” said Yutaka Akiyama–a former editor at publisher Iwanami Shoten–who was responsible for compiling the complete works of Soseki. “Soseki himself wrote his notes horizontally.”

The second batch of three works, which include Ryunosuke Akutagawa’s “Kumo no Ito” (The Spider’s Thread), came out Friday.

source: Laterally printed classics prove hit, Daily Yomiuri Online, August 23, 2008

Compensation for kanji-input basic technology subject of lawsuit

A Japanese man who says he invented the technology behind the context-based conversion of a sentence written solely in kana into one in both kanji and kana, as well as another related technology, filed suit against Toshiba on December 7, seeking some US$2.3 million in compensation from his former employer.

Shinya Amano, a professor at Shonan Institute of Technology, said in a written complaint that although the firm received patents for the technologies in conjunction with him and three others and paid him tens of thousands of yen annually in remuneration, he actually developed the technologies alone.

Amano is claiming 10 percent of an estimated ¥2.6 billion in profit Toshiba made in 1996 and 1997 — much higher than the roughly ¥230,000 he was actually awarded for the work over the two-year span.

His claim is believed valid, taking into account the statute of limitations and the terms of the patents.

“This is not about the sum of the money — I filed the suit for my honor,” Amano said in a press conference after bringing the case to the Tokyo District Court.

“Japan is a technology-oriented country, but engineers are treated too lightly here,” he said.

Toshiba said through its public relations office that it believes it paid Amano fair compensation in line with company policy. The company declined to comment on the lawsuit before receiving the complaint in writing.

Amano claims that he invented the technology that converts a sentence composed of kana alone into a sentence composed of both kanji and kana by assessing its context, and another technology needed to prioritize kanji previously used in such conversions.

Using theories of artificial intelligence, the two technologies developed in 1977 and 1978 are still used today in most Japanese word-processing software, he said.

source: Word-processor inventor sues Toshiba over redress, Kyodo News, via Japan Times, December 9, 2007

names of love hotels in macho kanji and other scripts

Donald Ritchie’s recent review of Japanese Love Hotels: A Cultural History, by Sarah Chaplin, has the following interesting section:

The contemporary love hotel is now much more kawaii (cute) than kinky.

Among the the reasons offered for this is that there has been something of a power shift in love-hotel choice. It used to be the male half that decided. Back then the places had hopeful macho monikers — Empire, Rex, King. Then the female half began to choose. Love hotels started calling themselves “fashion hotels” or “boutique hotels,” and began to have lavish lobbies with theme-shops, colors like beige and lavender, and decor like Laura Ashley.

This change can be documented in the Meguro Emperor (still in Meguro), which began in 1973 as a he-man fort before it slowly metamorphosed into a romantic Disneyland castle. The interior has been several times revised to segue from male- to female-friendly. Even the name has changed. It is now Gallery Hotel.

In most love hotels “macho” kanji has been replaced by “feminine” hiragana, trendy katakana or, more often, romaji, that romanized script that carries no male/female associations at all.

source: It’s ladies first now in Japanese love hotels, Japan Times, August 26, 2007

scripts related to Chinese characters — an article

sample of some of the scripts discussed in the paper; click to view the articleThe most recent rerelease from Sino-Platonic Papers is The Family of Chinese Character-Type Scripts, by Zhou Youguang, one of the main people behind the creation of Hanyu Pinyin. So it’s no surprise that his name has come up before in Pinyin News.

This article, from September 1991, categorizes and briefly discusses more than a dozen scripts derived from Chinese characters, most of which were used inside China by non-Han people.

The link above is to an HTML version. The original format of the article is preserved in the PDF file (650 KB).

Japanese literacy–an SPP reissue

Here’s another re-release from the archives of Sino-Platonic Papers: Computers and Japanese Literacy: Nihonzin no Yomikaki Nôryoku to Konpyûta, by J. Marshall Unger of the Ohio State University’s Department of East Asian Languages and Literatures. The link above is to the PDF version (1.2 MB), which reproduces the original exactly.

This is a parallel text in Japanese (in romanization) and English, so if any of you want to practice reading romaji, here’s your chance.

The English text alone is available in HTML: Computers and Japanese Literacy.

The essay touches on many of themes Unger explores in depth in his books, all of which have excerpts available here on Pinyin Info: The Fifth Generation Fallacy, Literacy and Script Reform in Occupation Japan, and Ideogram: Chinese Characters and the Myth of Disembodied Meaning.

Here is the opening, in both English and Japanese (in romanization).

Watakusi wa saikin, gendai no konpyûta siyô to Nihongo ni tuite kenkyu site orimasu. Gengogakusya mo konpyûta no nôryoku ya mondaiten ni tuite iken o happyo suru sekinin ga aru to omou kara desu. I am currently engaged in research on contemporary computer usage and the Japanese language. Linguists too, I believe, have a responsibility to present their views on the potentials and problems of computers.
Sate, Amerika no zen- Kôsei Kyôiku tyôkan, John Gardner-si no kotoba de hazimetai to omoimasu. Sore wa “aizyô nasi no hihan to hihan nasi no aizyô (Eigo de iu to, “unloving criticism and uncritical love”) to iu kotoba desu. Gardner-si wa, Amerikazin no aikokusyugi ni tuite Amerika o sukosi de mo hihan site wa ikenai to syutyô suru hito wa kangaetigai da, aizyô nasi ni syakai ya bunka no ketten o hihan bakari suru koto wa motiron warui keredo, hihan sore zitai o kiratte kokusuisyugi o susumeru koto mo syôrai no tame ni yoku nai, to iimasita. Kono koto wa bokoku igai no syakai to bunka ni tai suru baai de mo onazi de wa nai desyô ka? Gengogakusya ya rekisigakusya mo “aizyô nasi no hihan to hihan nasi no aizyô” to iu ryôkyokutan o sakeru yô ni sita hô ga ii to omou no desu. Watakusi wa Nihon no gengo to bunka o senmon ni site, Nihon ni tai site aizyô o motte orimasu kara koso, Nihongo no hyôkihô ya Nihonzin no yomikaki nôryoku ni tuite no teisetu o mondai ni site iru wake desu. Iwayuru zyôhôka syakai no zidai ni hairi, ippan no hitobito ga pasokon ya wâpuro o kozin-yô ni tukau yô ni naru ni turete, nettowâku tûsin, kyôiku-yô sohutowea, sôzôteki na puroguramingu nado ga yôkyû sarete kite iru desyô. Mosi sono konpon ni aru yomikaki nôryoku no henka to genzyô o gokai sureba, gôriteki na konpyûta siyôhô o kaihatu dekinai darô to omou kara desu. Let me begin by quoting the former U.S. Secretary of Health, Education, and Welfare, John Gardner. I am thinking of his phrase “unloving criticism and uncritical love.” By this, he meant that it was wrong for proponents of American patriotism to oppose even the slightest criticism of the United States: although it is bad to dwell unsympathetically on finding fault with social and cultural shortcomings, it is equally bad for the future of society to advance nationalism and eschew all criticism. I think that this is also true when considering foreign societies and cultures. Linguists and historians would do well to avoid the twin extremes of “unloving criticism and uncritical love.” As someone professionally involved with the language and culture of Japan, I have an affection for the country, but for that very reason, I wish to call into question the accepted theory of Japanese script and literacy. As we enter the age of the so-called informational society, and as more and more ordinary people begin to use computers on an individual basis, demands on network communications, educational software, creative programming, and so on, will steadily increase. Unless we understand the present situation and history of literacy, which underlies all these applications, we cannot hope to develop a rational basis for computer usage.
Sate, hyôi mozi to iu kotoba wa Nihongo ni tuite no hon ni yoku dete imasu kara kokugogaku no yôgo da to itte mo ii hodo desu ga, hyôi mozi to iu mono wa zissai ni sonzai site iru desyô ka? Kyakkanteki ni kangaete miru to, dono gengo mo konponteki ni wa hanasu mono desu. Mozi wa syakaiteki, rekisiteki na men ga arimasu ga, mozi wa kotoba no imi no moto de wa arimasen. Tatoeba, itizi mo yomenai mômoku no hito de mo, hoka no syôgai ga nai kagiri, bokokugo ga kanzen ni hanaseru yô ni narimasu. Sitagatte, hanasi-kotoba to wa mattaku kankei ga nai mozi nado to iu mono wa muimi na gainen desu. Gengo no imi wa gengo no kôzô kara hassei si, mozi wa sono han’ei de sika nai wake desu. Kore wa toku ni kore kara no konpyûta o kangaeru toki ni wasurete wa ikemasen…. The term “ideographic characters” appears so often in books on the Japanese language that one might say it has become a stock phrase of Japanese linguistics. I wonder, however, whether such things as “ideographs” actually exist. When examined objectively, all languages are fundamentally speech. Characters are not the source of the meanings of words, although they do have their social and historical aspects. For example, blind people who cannot read a single character can nonetheless speak their native tongues perfectly, unless they suffer from some other handicap. The very idea of characters totally divorced from speech is therefore meaningless. For the meaning of language emerges from the structure of language, of which writing is merely a reflection. It is particularly important that we not forget this when we consider the computers of the future….

This was first published in January 1988 as issue no. 6 of Sino-Platonic Papers.

make romanizations, not war

closeup of the romanization discussed in this postWhile a lot of things might be unusual about this old print of a Japanese soldier having sex with or simply raping a Western soldier, what particularly startled me is the use of romanization. Given that much of the text in the full image (note: definitely not safe for work) isn’t accompanied by romanization, it appears the intent is to help indicate which lines are being said by the Westerner. (But I can’t read Japanese, so I don’t know for sure.)

Has anyone noticed this practice — the romanization, y’all — in other Japanese prints?

Commenters on Eros Blog (again, not safe for work) translate the text and place the cartoon from the time of the Russo-Japanese War of 1904–05.