Comment by williebeek

Thanks for the tip, I will check if inserting rows with Go is any faster. For reference, inserting a log takes three steps, first the log data is stored in a Redis Stream (memory), a number of logs are taken from the stream and saved to disk and finally inserted in batches in ClickHouse. I've created it so you can take the ClickHouse server offline without losing any data (it will be inserted later).

For reference, moving about 4k logs from memory to disk takes less than 0.1 second. This is a real log from one of the webservers:

Start new cron loop: 2024-12-18 08:11:16.397...stored 3818 rows in /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc (0.0652 seconds).

Storing this data in ClickHouse takes a bit more than 0.1 second:

Start new cron loop: 2024-12-18 08:11:17.124...parsing file /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc

* Inserting 3818 row(s) on database server 1...0.137 seconds (approx. 3021.15 KB).

* Removed /var/www/txtlog/txtlog/tmp/txtlog.rows.2024-12-18_081116397_ES2gnY3fVc

As for Docker, I'm too much of a Docker noob but I appreciate the suggestion.