{"id":657,"date":"2007-03-05T21:49:10","date_gmt":"2007-03-05T13:49:10","guid":{"rendered":"https:\/\/pinyin.info\/news\/2007\/indicator-of-character-frequency-a-suggestion-for-programmers\/"},"modified":"2018-08-22T16:45:20","modified_gmt":"2018-08-22T08:45:20","slug":"indicator-of-character-frequency-a-suggestion-for-programmers","status":"publish","type":"post","link":"https:\/\/pinyin.info\/news\/2007\/indicator-of-character-frequency-a-suggestion-for-programmers\/","title":{"rendered":"indicator of character frequency: a suggestion for programmers"},"content":{"rendered":"<p>It occurred to me the other day that many people, especially language learners, might find it useful to have a tool that would take text written in Chinese characters and mark it up according to the frequency of use of the individual characters within. <\/p>\n<p>Here&#8217;s a sentence from a recent <span style=\"text-decoration: line-through;\">CCP rant<\/span> news item that can serve as an example: <\/p>\n<blockquote>\n<p>&#38750;&#39540;&#38750;&#39532;&#30340;&#8220;&#32593;&#35821;&#8221;&#19981;&#20877;&#28385;&#36275;&#20110;&#20559;&#23433;&#32593;&#32476;&#19968;&#38533;&#65292;&#27491;&#36805;&#36895;&#21521;&#30528;&#20854;&#23427;&#23186;&#20307;&#28183;&#36879;&#65292;&#22240;&#32780;&#21152;&#21095;&#20102;&#25253;&#32440;&#30005;&#35270;&#31561;&#25991;&#23383;&#35821;&#35328;&#30340;&#28151;&#20081;&#65292;&#29623;&#27745;&#20102;&#27721;&#35821;&#35328;&#25991;&#21270;&#30340;&#32431;&#27905;&#12290;<br \/>\n(<span class=\"py\">F&#275;il&#472;f&#275;im&#462; de &#8220;w&#462;ng y&#468;&#8221; b&#249;z&#224;i m&#462;nz&#250; y&#250; pi&#257;n&#8217;&#257;n w&#462;nglu&#242; y&#299;y&#250;, zh&#232;ng x&#249;ns&#249; xi&#224;ngzhe q&#237;t&#257; m&#233;it&#464; sh&#232;nt&#242;u, y&#299;n&#8217;&#233;r ji&#257;j&#249; le b&#224;ozh&#464; di&#224;nsh&#236; d&#283;ng w&#233;nz&#236; y&#468;y&#225;n de h&#249;nlu&#224;n, di&#224;nw&#363; le H&#224;ny&#468; y&#225;n w&#233;nhu&#224; de ch&#250;nji&#233;.<\/span>)<\/p>\n<\/blockquote>\n<p>Predictably, many of the characters here are extremely common. Others, however, would not even be covered under China&#8217;s definition of literacy. I&#8217;ve separated these characters into different classes, based on their frequencies of usage and applied different colors to each class: <\/p>\n<ul id=\"char-freq2\">\n<li><span style=\"background-color: lightgreen;\">character frequency: 1-100 (class i-c)<\/span><\/li>\n<li><span style=\"background-color: yellow;\">character frequency: 101-500 (class c-d)<\/span><\/li>\n<li><span style=\"background-color: gray;\">character frequency: 501-1000 (class d-m)<\/span><\/li>\n<li><span style=\"background-color: orange;\">character frequency: 1001-1500 (class m-md)<\/span><\/li>\n<li><span style=\"background-color: lightblue;\">character frequency: 1501-2000 (class md-mm)<\/span><\/li>\n<li><span style=\"background-color: red;\">character frequency: beyond 2000 (class mmplus)<\/span><\/li>\n<\/ul>\n<p>So the sample sentence would look like this: <\/p>\n<blockquote id=\"char-freq\">\n<p><span style=\"background-color: yellow;\" title=\"101-500\">&#38750;<\/span><span style=\"background-color: red;\" title=\"2001+\">&#39540;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#38750;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#39532;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#30340;&#8220;<\/span><span style=\"background-color: orange;\" title=\"1001-1500\">&#32593;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#35821;&#8221;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#19981;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#20877;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#28385;<\/span><span style=\"background-color: gray;\">&#36275;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#20110;<\/span><span style=\"background-color: orange;\" title=\"1001-1500\">&#20559;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#23433;<\/span><span style=\"background-color: orange;\" title=\"1001-1500\">&#32593;<\/span><span style=\"background-color: orange;\" title=\"1001-1500\">&#32476;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#19968;<\/span><span style=\"background-color: red;\" title=\"2001+\">&#38533;<\/span>&#65292;<span style=\"background-color: yellow;\" title=\"101-500\">&#27491;<\/span><span style=\"background-color: orange;\" title=\"1001-1500\">&#36805;<\/span><span style=\"background-color: gray;\">&#36895;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#21521;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#30528;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#20854;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#23427;<\/span><span style=\"background-color: red;\" title=\"2001+\">&#23186;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#20307;<\/span><span style=\"background-color: red;\" title=\"2001+\">&#28183;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#36879;<\/span>&#65292;<span style=\"background-color: yellow;\" title=\"101-500\">&#22240;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#32780;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#21152;<\/span><span style=\"background-color: gray;\">&#21095;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#20102;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#25253;<\/span><span style=\"background-color: gray;\">&#32440;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#30005;&#35270;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#31561;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#25991;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#23383;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#35821;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#35328;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#30340;<\/span><span style=\"background-color: orange;\" title=\"1001-1500\">&#28151;<\/span><span style=\"background-color: gray;\">&#20081;<\/span>&#65292;<span style=\"background-color: red;\" title=\"2001+\">&#29623;<\/span><span style=\"background-color: lightblue;\">&#27745;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#20102;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#27721;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#35821;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#35328;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#25991;<\/span><span style=\"background-color: yellow;\" title=\"101-500\">&#21270;<\/span><span style=\"background-color: lightgreen;\" title=\"1-100\">&#30340;<\/span><span style=\"background-color: orange;\" title=\"1001-1500\">&#32431;<\/span><span style=\"background-color: orange;\" title=\"1001-1500\">&#27905;<\/span>&#12290;<\/p>\n<\/blockquote>\n<p>(Those of you reading this through RSS may need to visit the site to see what I&#8217;m talking about.)<\/p>\n<p>The coding I used looks like this, though other approaches are possible:<\/p>\n<blockquote>\n<p>&lt;span class=&#8221;c-d&#8221; title=&#8221;101-500&#8243;&gt;&#38750;&lt;\/span&gt;&lt;span class=&#8221;mmplus&#8221; title=&#8221;2001+&#8221;&gt;&#39540;&lt;\/span&gt;&#8230;.<\/p>\n<\/blockquote>\n<p>I added titles to make this more accessible. <\/p>\n<p>Perhaps adding a summary would be useful:<\/p>\n<pre style=\"border: 1px solid green; padding: .5em; margin: .5em;\">1-100              24.6%\r\n101-500           42.1%\r\n501-1000          8.8%\r\n1001-1500         14.0%\r\n1501-2000          1.8%\r\n2001+              8.8%<\/pre>\n<p>This approach could also be used for Japanese &#8212; for example, to highlight all kanji not included in the J&#333;y&#333; kanji, or to highlight different sets of the Ky&#333;iku kanji. For that matter, it could also be applied to written words in English or other languages that use alphabets, though conjugutions, plurals, and the like would complicate matters. <\/p>\n<p>So, would anyone like to try coming up with one of these? Or has it been done already?<\/p>\n<p>one possible resource: <\/p>\n<ul>\n<li><a href=\"http:\/\/technology.chtsai.org\/syllable\/\">Mandarin Syllable Frequency Counts for Chinese Characters<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>It occurred to me the other day that many people, especially language learners, might find it useful to have a tool that would take text written in Chinese characters and mark it up according to the frequency of use of &hellip; <a href=\"https:\/\/pinyin.info\/news\/2007\/indicator-of-character-frequency-a-suggestion-for-programmers\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12,15,113,41,29,13,28,55,32],"tags":[],"class_list":["post-657","post","type-post","status-publish","format-standard","hentry","category-chinese","category-chinese-characters","category-computers","category-english","category-japanese","category-kanji","category-languages","category-literacy","category-mandarin"],"_links":{"self":[{"href":"https:\/\/pinyin.info\/news\/wp-json\/wp\/v2\/posts\/657","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pinyin.info\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pinyin.info\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pinyin.info\/news\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/pinyin.info\/news\/wp-json\/wp\/v2\/comments?post=657"}],"version-history":[{"count":4,"href":"https:\/\/pinyin.info\/news\/wp-json\/wp\/v2\/posts\/657\/revisions"}],"predecessor-version":[{"id":7825,"href":"https:\/\/pinyin.info\/news\/wp-json\/wp\/v2\/posts\/657\/revisions\/7825"}],"wp:attachment":[{"href":"https:\/\/pinyin.info\/news\/wp-json\/wp\/v2\/media?parent=657"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pinyin.info\/news\/wp-json\/wp\/v2\/categories?post=657"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pinyin.info\/news\/wp-json\/wp\/v2\/tags?post=657"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}