HeadlinesBriefing favicon HeadlinesBriefing.com

SQLite Production Learnings

Hacker News •
×

The author shares insights from using SQLite in production for a Django site, acknowledging that databases are complex despite SQLite's reputation for simplicity. A key discovery was the importance of the `ANALYZE` command. Initially, a full-text search query on a table of 4000 rows took 5 seconds. After running `ANALYZE`, the same query improved to approximately 0.05 seconds. The command generates statistics to help the query planner optimize performance.

Another challenge encountered is cleaning up large amounts of data. Accidental row insertions led to slow `DELETE` statements (over 5 seconds), causing other workers to time out and crash due to write contention. The current workaround involves processing deletions in small batches to avoid exceeding the 5-second timeout. This experience highlights the limitations of SQLite with concurrent writes compared to databases like Postgres.

For backups, two methods are discussed: `restic` and `litestream`. The `restic` method involves `VACUUM INTO` and then compressing and uploading the file, with commands to manage the repository. The author also experimented with `litestream` for incremental backups, finding it potentially more efficient, though unsure about its retention functionality. An interesting note is the possibility of using multiple database files, as demonstrated with the "Mess with DNS" project, which has run on SQLite for 4 years.