Symptom: the Yealink dashboard showed data, then panels went empty. Not data
loss — Loki has every line (per-device counts only ever grew; no ingestion
errors, no rejections, compactor deleted nothing). Loki was rejecting the
dashboard's own queries with HTTP 429 "too many outstanding requests" for ~45%
of requests (53 rejected vs 64 OK in 30 min), so on each refresh a random
subset of panels won the race and the rest blanked.
Cause: split_queries_by_interval defaulted to 30m, so each 24h panel query was
split into 48 sub-queries. 14 panels refreshing together = ~672 sub-queries
against a scheduler queue of 100, on a 2-core host.
Reproduced deterministically: 14 identical queries fired sequentially = 14/14
OK; the same 14 fired concurrently = 5/14 OK, 9 rejected.
Fix:
- split_queries_by_interval: 24h — 48 sub-queries become 1, and it matches the
schema_config index period. This fleet produces ~35k lines/day; splitting
buys nothing at this scale.
- query_scheduler.max_outstanding_requests_per_tenant: 2048 (was 100) — headroom
so one refresh cannot overflow the queue.
- max_query_parallelism: 8 (was 32) — realistic for 2 cores.
Validated on a disposable loki:2.9.8 container with its own data dir before
commit: starts clean, and /config confirms split_queries_by_interval=1d,
max_query_parallelism=8, max_outstanding_requests_per_tenant=2048.
Note for future dashboard work: the panel queries were individually verified
against live Loki before the dashboard shipped, but sequentially — which never
exercised the concurrency a dashboard actually creates. Verify dashboards by
loading them, not by looping their queries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator requirement: logs must persist at least one year. Applied
live (data-preserving restart, verified prior data survived) and
here for the next redeploy to stay in sync.
:latest images (loki 3.7.3/go1.26.4, presumably grafana/alloy too) reset
every TCP connection on this host — kernel is 5.16.7 (Fedora 35, EOL
since 2023), too old for whatever the newer Go toolchains need.
Confirmed via a disposable grafana/loki:2.9.8 test container: same
config, same host, works cleanly. Pin all three images to older stable
releases instead of chasing this again on every redeploy.
Also fixes an unrelated config bug: instance_addr belongs directly
under `common:`, not nested under `common.ring:` — the misplaced field
was silently ignored, which didn't help but wasn't the actual cause.
Alloy receives syslog on 514/udp and 1514/tcp from network devices,
forwards to a local Loki instance; Grafana provisioned with Loki as
default datasource. Compose bind-mounts config from
/mnt/containers/logging/... on the host (staged separately) since
Portainer's GitOps pull only fetches the compose file, not the repo's
other files.
pipistrelloandClaude Opus 4.8