pipistrelloandClaude Opus 4.8
bb4c92d5e5
Symptom: the Yealink dashboard showed data, then panels went empty. Not data loss — Loki has every line (per-device counts only ever grew; no ingestion errors, no rejections, compactor deleted nothing). Loki was rejecting the dashboard's own queries with HTTP 429 "too many outstanding requests" for ~45% of requests (53 rejected vs 64 OK in 30 min), so on each refresh a random subset of panels won the race and the rest blanked. Cause: split_queries_by_interval defaulted to 30m, so each 24h panel query was split into 48 sub-queries. 14 panels refreshing together = ~672 sub-queries against a scheduler queue of 100, on a 2-core host. Reproduced deterministically: 14 identical queries fired sequentially = 14/14 OK; the same 14 fired concurrently = 5/14 OK, 9 rejected. Fix: - split_queries_by_interval: 24h — 48 sub-queries become 1, and it matches the schema_config index period. This fleet produces ~35k lines/day; splitting buys nothing at this scale. - query_scheduler.max_outstanding_requests_per_tenant: 2048 (was 100) — headroom so one refresh cannot overflow the queue. - max_query_parallelism: 8 (was 32) — realistic for 2 cores. Validated on a disposable loki:2.9.8 container with its own data dir before commit: starts clean, and /config confirms split_queries_by_interval=1d, max_query_parallelism=8, max_outstanding_requests_per_tenant=2048. Note for future dashboard work: the panel queries were individually verified against live Loki before the dashboard shipped, but sequentially — which never exercised the concurrency a dashboard actually creates. Verify dashboards by loading them, not by looping their queries. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
67 lines
1.8 KiB
YAML
67 lines
1.8 KiB
YAML
auth_enabled: false
|
|
|
|
server:
|
|
http_listen_address: 0.0.0.0
|
|
http_listen_port: 3100
|
|
grpc_listen_address: 0.0.0.0
|
|
grpc_listen_port: 9096
|
|
log_level: info
|
|
|
|
common:
|
|
instance_addr: 127.0.0.1
|
|
path_prefix: /loki
|
|
storage:
|
|
filesystem:
|
|
chunks_directory: /loki/chunks
|
|
rules_directory: /loki/rules
|
|
replication_factor: 1
|
|
ring:
|
|
kvstore:
|
|
store: inmemory
|
|
|
|
schema_config:
|
|
configs:
|
|
- from: 2024-01-01
|
|
store: tsdb
|
|
object_store: filesystem
|
|
schema: v13
|
|
index:
|
|
prefix: index_
|
|
period: 24h
|
|
|
|
limits_config:
|
|
reject_old_samples: true
|
|
reject_old_samples_max_age: 168h
|
|
retention_period: 8760h # 365 days — required minimum; monitor disk usage as fleet onboarding grows (host has ~40GB free)
|
|
max_query_series: 5000
|
|
|
|
# --- Dashboard concurrency ---
|
|
# Default 30m split a 24h dashboard panel into 48 sub-queries. With 14 panels
|
|
# refreshing together that is ~672 sub-queries against a scheduler queue of
|
|
# 100 -> HTTP 429 "too many outstanding requests", and panels blank out at
|
|
# random on each refresh (the "data appears then disappears" symptom).
|
|
# Measured 2026-07-15: 14 identical queries fired sequentially = 14/14 OK;
|
|
# the same 14 fired concurrently = 5/14 OK, 9 rejected.
|
|
# This fleet produces ~35k lines/day, so splitting buys nothing here.
|
|
# 24h also matches the schema_config index period above.
|
|
split_queries_by_interval: 24h
|
|
# Default 32 is far too optimistic for this 2-core host.
|
|
max_query_parallelism: 8
|
|
|
|
query_scheduler:
|
|
# Default 100. Headroom so one dashboard refresh cannot overflow the queue.
|
|
max_outstanding_requests_per_tenant: 2048
|
|
|
|
compactor:
|
|
working_directory: /loki/compactor
|
|
compaction_interval: 10m
|
|
retention_enabled: true
|
|
retention_delete_delay: 2h
|
|
delete_request_store: filesystem
|
|
|
|
ruler:
|
|
storage:
|
|
type: local
|
|
local:
|
|
directory: /loki/rules
|