HeadlinesBriefing favicon HeadlinesBriefing.com

Mojibake: A C11 Unicode Library

Hacker News •
×

Mojibake is a new, low-level Unicode library written in C11 and compatible with C++17, designed for efficient text processing without external dependencies. It comprises just two files, `mojibake.h` and `mojibake.c`, and integrates essential Unicode algorithms such as normalization, case conversion, segmentation, bidirectional text, collation, and confusable character detection.

The library supports Unicode 17.0 and is released under the MIT License. It aims to be small, easy to use, and fast, running on various operating systems including Linux, macOS, FreeBSD, OpenBSD, NetBSD, and Windows 11. A WASM demo is available online, showcasing the public API functions and documentation.

Mojibake offers comprehensive features including normalization (NFC, NFD, NFKC, NFKD), case conversion with special casing, text filtering, character database properties, segmentation (grapheme clusters, words, sentences), bidirectional text handling, emoji analysis, and display width calculations. It also includes collation for sorting and comparison, and confusable detection for security applications.

Integration is simplified with support for multiple encodings (UTF-8, UTF-16LE/BE, UTF-32LE/BE), parsing functions, and locale tag parsing. The library is embeddable with custom allocators and build-time feature flags to optimize size. It has undergone extensive testing with over 1.5 million assertions, including official Unicode conformance suites, and is fuzzed with libFuzzer.