HeadlinesBriefing favicon HeadlinesBriefing.com

Running Rust JSON Parsers in Python with PyO3

Hacker News •
×

Every time you validate data with Pydantic v2, the data-validation library most Python apps reach for, a Rust extension does the work. Its core, pydantic-core, is built with Py O3, the same toolchain this post uses to expose a small Rust JSON parser as a Python package.

Getting Rust code into Python takes four steps: write a normal Rust module, annotate it with Py O3 macros, let maturin compile and install it, and import the result. #[pyfunction] and #[pymodule] add the glue, type conversions, and reference counting. Maturin then compiles the crate to a shared library and drops it into your virtual environment.

The parser produces a plain Rust enum: Null, Boolean(bool), Number(f64), String(String), Array(Vec<Json Value>), or Object(Hash Map<String, Json Value>). A Rust attribute macro rewrites the function it sits on, like a Python decorator. In our Python to Rust cohort, students spend six weeks writing a JSON parser from scratch in Rust, a hand-rolled tokenizer and recursive-descent parser with no serde, then expose it to Python through Py O3. Josh's version beat CPython's C json module on real-world fixtures; Jochen's ran up to 3.5x faster than the Python version.

The public reference implementation is the clean version students start from. parse_json<'py>(py: Python<'py>, input: &str) -> Py Result<Bound<'py, Py Any>> returns the value or raises a Python exception. On traditional Python builds, this access is associated with holding the GIL. Py O3 hands it to you and you pass it along wherever you touch a Python object.