Fix Loki 429s that blanked out dashboard panels at random

Symptom: the Yealink dashboard showed data, then panels went empty. Not data
loss — Loki has every line (per-device counts only ever grew; no ingestion
errors, no rejections, compactor deleted nothing). Loki was rejecting the
dashboard's own queries with HTTP 429 "too many outstanding requests" for ~45%
of requests (53 rejected vs 64 OK in 30 min), so on each refresh a random
subset of panels won the race and the rest blanked.

Cause: split_queries_by_interval defaulted to 30m, so each 24h panel query was
split into 48 sub-queries. 14 panels refreshing together = ~672 sub-queries
against a scheduler queue of 100, on a 2-core host.

Reproduced deterministically: 14 identical queries fired sequentially = 14/14
OK; the same 14 fired concurrently = 5/14 OK, 9 rejected.

Fix:
- split_queries_by_interval: 24h — 48 sub-queries become 1, and it matches the
  schema_config index period. This fleet produces ~35k lines/day; splitting
  buys nothing at this scale.
- query_scheduler.max_outstanding_requests_per_tenant: 2048 (was 100) — headroom
  so one refresh cannot overflow the queue.
- max_query_parallelism: 8 (was 32) — realistic for 2 cores.

Validated on a disposable loki:2.9.8 container with its own data dir before
commit: starts clean, and /config confirms split_queries_by_interval=1d,
max_query_parallelism=8, max_outstanding_requests_per_tenant=2048.

Note for future dashboard work: the panel queries were individually verified
against live Loki before the dashboard shipped, but sequentially — which never
exercised the concurrency a dashboard actually creates. Verify dashboards by
loading them, not by looping their queries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

logging/loki-config.yaml

@@ -35,6 +35,23 @@ limits_config:
   retention_period: 8760h  # 365 days — required minimum; monitor disk usage as fleet onboarding grows (host has ~40GB free)
   max_query_series: 5000
 
+  # --- Dashboard concurrency ---
+  # Default 30m split a 24h dashboard panel into 48 sub-queries. With 14 panels
+  # refreshing together that is ~672 sub-queries against a scheduler queue of
+  # 100 -> HTTP 429 "too many outstanding requests", and panels blank out at
+  # random on each refresh (the "data appears then disappears" symptom).
+  # Measured 2026-07-15: 14 identical queries fired sequentially = 14/14 OK;
+  # the same 14 fired concurrently = 5/14 OK, 9 rejected.
+  # This fleet produces ~35k lines/day, so splitting buys nothing here.
+  # 24h also matches the schema_config index period above.
+  split_queries_by_interval: 24h
+  # Default 32 is far too optimistic for this 2-core host.
+  max_query_parallelism: 8
+
+query_scheduler:
+  # Default 100. Headroom so one dashboard refresh cannot overflow the queue.
+  max_outstanding_requests_per_tenant: 2048
+
 compactor:
   working_directory: /loki/compactor
   compaction_interval: 10m