Marvelous Moonglyphs: Match Kana To Kanji
natural-language
For people who are curious, like to do research, and want to learn something new.
This is a somewhat real-world example that isn't to hard to implement, but it may seem difficult because many people around here won't be familiar with the topic.
Overview
Recently your company started to expand its business to the Asian market. Nobody volunteered, so you have been asked to come up with some Japanese text processing code. Japanese addresses often come as a bunch of squiggly moon-glyphs, with the prefecture, district, and town name all mangled together. On the net, you found a list that tells you how to read that bunch, but you (and your Japanese customers) would like to know how to pronounce the district and town name by itself.
A very brief, over-simplified explanation of the Japanese writing system:
Japanese consists 100~200 syllables. They can be written with 48 kana, similar to our alphabet. There are two versions, Hiragana
and Katakana
, like lowercase and uppercase letters. Kanas
are like a phonetic transcription. A word can also be written with meaning-based kanji. Each kanji may possess multiple readings. Given a word with many Kanji
and its reading in Kana
, determine which Kanas
belong to which Kanji
. Look up Kanji
on wikipedia if you want to know more.
All Hiragana
s ["lower case"] are
がぎぐげござじずぜぞだぢづでどばびぶべぼぱぴぷぺぽあいうえおかきくけこさしすせそたちつてとなにぬねのはひふへほまみむめもやゆよらりるれろわをんぁぃぅぇぉゃゅょっゐゑゔ
And the Katakana
s ["upper case"] are
ガギグゲゴザジズゼゾダヂヅデドバビブベボパピプペポアイウエオカキクケコサシスセソタチツテトナニヌネノハヒフヘホマミムメモヤユヨラリルレロワヲンァィゥェォャュョッヰヱヴ
They correspond to each other in the order given above.
Scoring
Feature-challenge.
Your program should implement the basic feature described below. Your basic score is 20.
You will receive additional points for each feature you implement. In case of a tie, code length in bytes decides.
Disclaimer
Standard loopholes shall (not) apply.
First, I will provide you with the information needed to define the task.
After that, I shall add some notes, examples, and hints for those of you not familar with Japanese. If you want to challenge yourself, and do some research yourself, do not read this.
Task
It is your task to write a program that will take as its input a string of MOONGLYPHS
, its READING
, and the moonglyphs separated into PARTS
whose readings your program should output. You already found a dictionary file with all possible readings for each MOONGLYPH
. (see below). All examples are formatted as follows:
MOONGLYPHS
READING
PART1
,PART2
,PART3
,...
EXPECTED_OUTPUT
A simple example:
成田
[Narita, name of a town]
なりた
[na-ri-ta]
成
, 田
- [
成
,なり
],[田
,た
]
The MOONGLYPH
s 成田
are read なりた
. The parts 成
and 田
are read なり
and た
.
I/O source and destination
Up to you, as long as it a complete program, ie you may read from
- a file
- stdin
- network
- keyboard
Same for the output.
Input and output format
Input:
All strings may be encoded in the encoding of your choice. (eg UTF-8, Shift-JIS etc.)
MOONGLYPH
and READING
are strings (or an equivalent in the language of your choice).
PARTS
are an array, or an equivalent data structure in the language of your choice. Each entry is a string.
MOONGLYPH
only contains MOONGLYPH
s found in the Dictionary File
. (see below)
It may also include other characters, if your program implements the corresponding feature.
If you support all features, it may include KATAKANA
, HIRAGANA
, and various full-width symbols and punctuation marks as well.
It will never contain any half-width letters, numbers or marks. (such as ,.=?)agE234\
)
READING
only contains HIRAGANA
.
If you implement the corresponding feature, it may contain the same full-width symbols and numbers that MOONGLYPH
s may contain.
The array of PART
s, when joined in the given order, will result in MOONGLYPH
. For example, if MOONGLYPH
is 日本語, then parts may be [日本,語]
or [日,本,語] - but not [語,本,日] (reversed order) or [日本] (missing 語).
Example:
日本語
(Japanese)
にほんご
[ni-hon-go]
日本
, 語
The MOONGLYPH
string is 日本語
, the READING
is にほんご
, and the parts are 日本
and 語
.
Output
An array, or equivalent data structure.
Each entry contains one of the input PARTS
, as well as the corresponding part of the READING
- in the same order as PARTS
. Joining all parts results in MOONGLYPH
s, and joining all readings results in the READING
.
If there is no match, your program must behave in a way that is distinguishable from when it finds at least one match - including outputting nil
, an empty array
, or crashing
.
Example:
日本語
(Japanese)
にほんご
日本
,語
- [
日本
,にほん
], [語
,ご
]
All of the following cannot be valid outputs under any circumstances, irrespective of the dictionary data:
[語
,ご
], [日本
,にほん
] (reversed order)
[日本
,に
], [語
,ご
] (joining the readings results in にご
, which is not equal to the READING
, にほんご
)
[日
,にほん
], [語
,ご
] (joining the moongylphs results in 日語
, which is not equal to the MOONGLYPH
s, 日本語
)
Dictionary File
The dictionary file is called KANJIDIC
(not KANJDIC212
) and can be found on this page (English):
It comes in a few different formats, choose one you like. Treat suffixes
and prefixes
as regular readings, strip the okurigana
off the reading.
I also added the files on this github.
Basic Feature
Score = 20
Output the readings for each part, as described in the Input/Output section.
A somewhat longer example:
京都府京都市下京区大黒町仏光寺通御幸町西入
[Kyoto, Shimo-Gyouku District, Daikoku]
きょうとふきょうとししもぎょうくだいこくちょうぶっこうじどおりごこまちにしいる
京都府
,京都市下京区
,大黒町
,仏光寺通御幸町西入
- [
京都府
,きょうとふ
], [京都市下京区
,きょうとししもぎょうく
], [大黒町
,だいこくちょう
], [仏光寺通御幸町西入
,ぶっこうじどおりごこまちにしいる
]
The only possible combination, given the dictionary data, is that きょうとふ
belongs to 京都府
, etc.
Optional Features.
No need to implement all features if you don't understand one of them. Remember, have fun.
壱 (1) +15
Implement Rendaku (Voicing). Handakuten count as voicing as well. No ヴ.
To keep it simple, we are going to assume that this voicing may always occur, except for the KANA
at the beginning of the READING
string.
弐 (2) +10
Support and ignore these punctuation symbols.
─〜、・()。!?「」/〒【】『』0123456789
These appear both in the MOONGLYPH
s, READING
s, and PART
s at the same abstract position and should be ignored. That is, your program does not need to handle unmatched punctuation. You may assume that punctuation characters will always agree between MOONGLYPHS
and READING
.
桜川市(亀岡)
["Cherry Flower River", "Turtle Hill"]
さくらかわし
, (かめおか)
Invalid input:
参 (3) +10
Support ケ
, ヶ
, and ヵ
.
All three may be read か
and が
. ケ
and ヶ
may also be read げ
and こ
.
肆 (4) +10
Support omitted genitive markers の
between MOONGLYPHS
. An addtional +5 if you support が
as well.
油小路
あぶらのこうじ
油
, 小路
- [
油
,あぶら
], [<empty>
,の
], [小路
,こうじ
]
伍 (5) +15
Support full-width roman numbers. You only need to support integers >0
and <1E12
, and do not include any separators at any power of 10.
There shall be no 一
before 百
, 千
, 万
, and 億
.
An addtional +5 if you support an optional 一
before 百
, 千
, 万
, and 億
. That is, 102番
may be read either as 百番
or 一百番
.
12月
じゅうにがつ
12
, 月
- [
12
,じゅうに
], [月
,がつ
]
陸 (6) +15
Add support for KANA
. Including the now deprecated four ゑ
, ゐ
, ヱ
, ヰ
read as い
and `え.
は
, へ
, を
will never appear as わ
, え
, or お
in the output.
岩月町かしわ野
[City of Iwatsuki "Moon Rock", Kashiwano "Evergreen Oak Plains"]
いわつきまち
, かしわの
漆 (7) +5
Add the additional MOONGLYPH
s found in KANJIDIC212
. You can download it from the same page as KANJIDIC
, see above. XML here.
鱏八軟骨魚綱板鰓亜綱仁属為
(Batoidea are Chondrichthyes, Elasmobranchii)
えいはなんこつぎょこうばんさいあこうにぞくす
捌 (8) +10
Prefer on-on
and kun-kun
readings, and sort the results accordingly.
This requires a metric. To keep things simple, set the likelihood
to zero, add +1
for each on/on or kun/kun pairs.
So for example,
- ON ON KUN KUN => likelihood 2
- ON KUN ON KUN => likelihood 0
- ON ON ON KUN => likelihood 3
Punctuation symbols and KANA
are be ignored for this calculation.
玖 (9) +15
Implement the MOONGLYPH
doubler sign 々
.
When the MOONGLYPH
repeater 々 occurs m*n
times, it may stand for the last n
MOONGLYPHS
occuring m
times.
月光綺麗々々々々々々
[The moonlight. Beautiful, beautiful, beautiful, beautiful.]
げっこうきれいきれいきれいきれい
月光綺麗
, 々々々々々々
- [
月光綺麗
,げっこうきれい
], [々々々々々々
,きれいきれいきれい
]
The input MOONGLYPH
s shall never be such that any possible choice for n or m results in a previous occurence of 々 getting repeated. Thus, 木々日々々々
would not be a valid input.
拾 (10) +10
Support the voiced kana repeater ゞ
. ひゞ
shall stand for ひび
or ひぴ
, ごゞ
for ごご
, and ぱゞ
for ぱば
or ぱぱ
.
It may not occur after syllables that do not accept dakuten, eg まゞ
will is invalid input.
陰 (Final) +20
Support Ateji
, Gikun
readings, that is support multi-MOONGLYPH
words.
Dictionary File EDict. Use either edict.gz
or edict2.gz
(custom format); or JMdict.gz
or JMdict_e.gz
(xml). The download page also contains links to the documentation of the dictionary format.
(1)
(2)
This word is found only in EDICT2
, but not in EDICT
.
Complex Example
This example requires features 1,2 5, and 9.
173〜190番地「鉢伏峠」等々
ひゃくななじゅうさん〜いっぴゃくきゅうじゅうばんち「はちぶせとうげ」とうとう
173〜190
, 番地
, 「鉢伏峠」
, 等々
- [
173〜190
,ひゃくななじゅうさん〜いっぴゃくきゅうじゅう
], [番地
,ばんち
], [「鉢伏峠」
,「はちぶせとうげ」
], [等々
,とうとう
]
Tutorial
Do not read any further if you want to challenge yourself, or do the research yourself.
Moved here to keep this short.
May your journey to the moon be successful and fruitful, brave adventurer!