Comment by rich_sasha
Comment by rich_sasha 7 hours ago
I concur that sqlite is quite amazing. That said, I was a heavy user and have grown some skepticism as well:
- it is not that hard to lock the db. Usually killing the process that caused the deadlock solves the issue - but you need to identify it / monitor for it. And yes, it happens with WAL too
- but when it does happen, it is quite scary. Simply, anything that touches your DB suddenly stops working - can't read, can't write.
- in some cases, WAL does not checkpoint. This leads to drastic growth in the size of the WAL file, and down the line in catastrophic slowdown of queries - things that take 10ms suddenly take 10 seconds. In my particular case, no tweaking of SQLite params fixed it. I had to monitor for it, and periodically force WAL file to be rolled into the main DB.
- all of this gets harder on Windows, where eg.you cannot just 'lsof' a file.
- the performance stats change somewhat for the worse in the cloud on drives that look local but actually aren't. Of course that is not sqlite's fault, but the blazing fast performance doesn't apply to all commonly encountered environments that look like real local drives.
I'm not dissing SQLite, I use it despite these shortcomings. Equally, I'm happy to reach for something like Postgres, which, well, hasn't burned me yet.
EDIT I should add that despite all this I never managed to corrupt the DB, or break any of the SQL promises - never messed up indices, never saw broken ACID compliance etc. And that's a massive endorsement, on reflection.
These are some really good points.
- WAL checkpointing is very important (litestream handles this well). As you said not checkpointing can cause massive query slow down.
- SQLITE_LOCK and SQLITE_BUSY can be avoided by ensuring your application only has a single write connection ideally behind an MPSC queue. After WAL this is probably one of the biggest SQLite quality of life improvements.
- 100% avoid cloud drives in this context you ideally want attached NVME.
- Postgres is great and there's nothing wrong with using it!