Late last year I released posuto, a package presenting Japanese postal codedata in an easy-to-use format. It's based on data released by Japan Post, which is infamous for being widely used but hard to parse. I first became aware of the postal data when I entered my postal code in an online form and it auto-completed my address as "XXX-borough (except the following buildings)".

I had no idea what that parenthetical was referring to, so I looked for a common source of postal data, found the CSV, and found the issue. It turns out the CSV file contains parenthetical notes for anyone reading the CSV file and makes reference to the order of the rows. This causes problems.

The data is mainly useful one row at a time, where the parenthetical is meaningless. Since CSV is a field-delimited format, there's also no need for parentheticals - you could just add a note field. This is only one of many issues with ken_all.csv.

You can find people complaining about it regularly on Twitter, and there was even briefly a blog just collecting posts from all over the web about it. A particularly amusing tweet describes people who expect computers to bend to the will of humans being punished in Hell by having to parse ken_all.csv forever. The README for the file explains that lines with overly long fields will be broken up into multiple lines.

Specifically, if the neighborhood name is over 38 characters, or if the half-width katakana pronunciation field is over 76 characters, the line will be split into two lines. The overly-long neighborhood field will be continued and all other fields will be duplicated. It should also be noted that while the length limits are as stated, the location where line breaks are inserted in long lines appears random, occurring neither at the character limit nor at normal word boundaries.

It's worth noting not all the issues with the CSV are inherently technical; postal codes are always complicated. The postal code with the most rows in the CSV - a stunning 66 - is 〒452-0961, which refers to the Haruhi region of Kiyosu City in Aichi Prefecture. This has that many lines because every neighborhood gets a separate line.

In contrast, the longest continued line, using the line break rules above, is the entry for 〒602-8368 or 〒602-8374, both with eight lines. These are both in one of a few areas in Kyoto that uses a unique, bizarre system of intersection-based addressing. There are catch-all postal codes for many areas, where the neighborhood is given as "except the following", and the only thing to do is look for that exact string and exclude it.

There's a variety of similar strings, and it's hard to be sure I've caught them all. An example of another comment is 一円. Normally this would mean "one yen", but it also means "the area surrounding", and is a note in the CSV that should be removed from neighborhood names, except...