HeadlinesBriefing favicon HeadlinesBriefing.com

UTF-8000: Unlimited UTF-8 Encoding Proposal

Hacker News •
×

UTF-8000 proposes an unlimited extension of UTF-8 encoding that preserves all existing properties while removing the 4-byte limit. The specification maintains ASCII ⊆ UTF-8 ⊆ UTF-8000 compatibility with no special cases introduced beyond those inherited from UTF-8. A reference implementation is available via pipx install UTF-8000, though the project is not endorsed by the Unicode Consortium.

The encoding extends UTF-8's pattern logically: 5-byte sequences use prefix 111110xx, 6-byte uses 1111110x, continuing up to 22-byte sequences and beyond. The system relies on UTF-8's self-synchronization prefixes (0 for ASCII, 10 for continuation bytes, 11 for multi-byte first bytes) to achieve byte-level self-synchronization. Key innovations include clarifying the split between self-synchronization bits and start bits in the first byte, then striping start bits across continuation bytes for arbitrarily large code units.

The glossary defines terms like codepoint, code unit, first byte, continuation byte, and self-synchronization prefix. Notably, byte four in a 22-byte sequence serves as both a continuation byte and final start byte with straddled mandatory content bits. Only two special cases exist: ASCII preservation and 2-byte UTF-8's 4 mandatory content bits versus 5 for longer sequences.