Read-only disk check-up for ClickHouse® inside self-hosted Langfuse, SigNoz and ClickStack
The disk fills up, but your own data is small. Usually the space goes to ClickHouse's own log tables, which have no size limit by default. diskvet shows what eats the disk and prints the exact commands to fix it. Nothing runs by itself.
Free, Apache-2.0, no registration. Two short files you can read before you run them. The script talks only to your ClickHouse; nothing is sent anywhere else.
The usual story
- 66.86 GiB of
system.trace_lognext to about 51 MiB of Langfuse tables (langfuse#13123); - 80+ GB of system logs next to less than 500 MB of telemetry (SigNoz#12050);
- a 10 Gi volume full in 10 days (ClickStack-helm-charts#275).
The main fix is known: TRUNCATE the logs and add a TTL. The report adds the
traps that fix hits in practice: the 50 GB limit on TRUNCATE, old
*_log_N copies after a restart, the special config of
opentelemetry_span_log, Docker logs without rotation, and deleted rows that
still hold space.
Quick start: Langfuse with docker compose (30 seconds)
On the machine where Langfuse's docker-compose.yml runs:
cd langfuse # the folder with Langfuse's docker-compose.yml
curl -fsSLO https://github.com/Protemir/diskvet/releases/latest/download/diskvet.sh
curl -fsSLO https://github.com/Protemir/diskvet/releases/latest/download/checks.sql
curl -fsSLO https://github.com/Protemir/diskvet/releases/latest/download/SHA256SUMS
sha256sum -c SHA256SUMS # optional: both files match the release
less checks.sql # read it first: SELECTs from system.* only
sh diskvet.sh report --docker auto > report.md--docker auto finds the ClickHouse container itself and runs
clickhouse-client inside it, so you need neither a client on the host nor an
open port. Without --user it uses the container's own
CLICKHOUSE_USER / CLICKHOUSE_PASSWORD, and the password never
leaves the container.
Other ways to connect:
sh diskvet.sh report --docker signoz-clickhouse # a container by name
sh diskvet.sh report --host 127.0.0.1 --user diskvet --password '...' # local clickhouse-clientNeeds POSIX sh, awk, sed,
od, date, and docker or clickhouse-client.
No jq, no Python. Latest release:
GitHub releases.
What it checks
| # | Check | Reads from | Typical fix in the report |
|---|---|---|---|
| 1 | System logs without TTL | system.tables, system.parts | TRUNCATE (with the one-time flag over 50 GB), a generated config.d TTL file, old *_log_N copies to drop |
| 2 | Disk space not in ClickHouse table parts | system.disks vs all parts | Docker log rotation and where else to look |
| 3 | Disk usage and rough forecast | system.disks, system.asynchronous_metric_log | what to free first |
| 4 | Growth per day | system.part_log | which table wrote the most in 24 h |
| 5 | Too many parts | system.parts, system.merge_tree_settings, system.events | distance to the table's insert limits |
| 6 | Inactive and detached parts | system.parts, system.detached_parts | stuck parts, DROP DETACHED PART |
| 7 | Deleted rows and stuck mutations | system.parts, system.mutations | APPLY DELETED MASK for the exact partitions, KILL MUTATION |
Each check says OK, INFO, WARN or CRITICAL, with the thresholds in the
README. Every fix comes with
how safe it is and whether it needs a restart. A check whose query fails shows
NOT_RUN with the reason; the others still run.
What it reads, and what it never reads
Reads
- metadata from
system.tables,system.parts,system.disks,system.detached_parts,system.merge_tree_settings,system.mutations,system.part_log,system.asynchronous_metric_log,system.asynchronous_metrics,system.events; - if allowed,
system.server_settings(onlymax_table_size_to_drop); - one probe,
SELECT getSetting('readonly'), to confirm the session is read-only.
Never reads
- rows of your own tables: no
FROMorJOINon them, no table functions, nodictGet; the test suite enforces this; system.query_log, query texts, mutation commands or error texts.
The local report shows real database and table names, because you need them to run the fixes. It stays on your machine.
Two ways to run it safely
- With an existing user. Queries run with
readonly=2and limits (30 s, 10 000 rows, 2 threads, 500 MB). If the server refuses read-only mode for this user, the script stops without running anything. - With a dedicated user, for security reviews: narrow grants on the system tables above and no access to product data. The exact SQL is in the README.
Tested on stock clickhouse/clickhouse-server images 24.1, 24.8,
25.12 and 26.9, with Langfuse-like and SigNoz-like setups. Not tested yet: replicated clusters,
Kubernetes, disks on object storage, macOS.
Prefer to fix it by hand?
Why ClickHouse fills the disk in self-hosted Langfuse and SigNoz, and how to fix it
One read-only query to confirm it, how to free the space now, and how to stop it coming back without the known traps. Every command in it was run on ClickHouse 24.8, 25.12 and 26.9. You don't need diskvet to follow it.
Hourly check-ups: free beta in October
The script is free and stays free. A single run can't tell when the disk will really run out. That is the part I'm building now:
- an hourly snapshot: exactly the JSON that
sh diskvet.sh --print-payloadprints, nothing more; - an email before the disk fills, only when a state changes;
- a signal when snapshots stop arriving;
- a short report every Monday.
No hosting on your side and no open port: your server sends a small signed snapshot out, nothing connects in. Host names, IPs, users, query texts and errors are never sent; your own table names leave your server only as salted hashes.
Want in? Comment in Early access (discussion #1) with what runs your ClickHouse and roughly how big the disk is. You'll get a reply there when the beta opens. Please don't post company or host names there: it's a public thread.
The beta runs for 30 days; after that the hourly part becomes a paid plan, and the script stays free and open source. Servers in Kazakhstan: local report only, for now.
Found a problem the script misses, or a wrong fix? Open an issue. That's the most useful thing you can do.