The Yealink dashboard treated nearly every level-3 line as actionable, making routine boot and firmware chatter look like fleet-wide failures.\n\nReplace generic error counts with recent LDAP, provisioning, and internal-database signals. Add dedicated operational trend and log panels, and relabel raw severity as diagnostic-only while retaining web-password, registration, and DHCP context.\n\nValidated all 17 LogQL targets directly against Loki 3.7.2. In the inspected 24-hour sample, provisioning had settled, LDAP failures remained active, and one handset reported a malformed internal database.
Two changes driven by the fleet syslog rollout (~4 -> 106 senders) and the
operator enabling Yealink's prepend-MAC on the autoprovisioned handsets.
1. Alloy: lift the MAC into a real `mac` label.
UDP syslog offers only three possible identities: sender IP, RFC3164
HOSTNAME, message body.
- source_ip is DHCP (observed lease 1200s) so it is NOT stable: a re-IP
looks like a new device, and a recycled IP silently inherits another
phone's history.
- HOSTNAME is useless: Yealink puts its subsystem there (sua/GUI/cfg/sys),
and after prepend-MAC the bracketed MAC lands in that slot and is rejected
as a hostname, so the label is simply absent on new lines.
So the body is the only place identity exists. A loki.process regex stage
extracts it. Non-Yealink senders (the D-Link switches) don't match and pass
through with no mac label. Cardinality is safe: mac is 1:1 with a device and
does not multiply against source_ip.
Verified on a disposable alloy+loki rig by replaying 406 REAL captured lines:
0 parse errors, both branches confirmed — Yealink lines get
mac=80:5e:0c:b2:44:d3, the switch line passes through with no mac label.
2. Dashboard: identity by MAC, comparative panels, no invented thresholds.
- Variable is now label_values(mac) with allValue ".+" (not ".*") so "All"
matches only streams that HAVE a mac — excluding switches automatically.
- Removed the absolute colour thresholds. They were derived from a ~4-hour
sample of TWO atypically verbose handsets and were already wrong at 2
phones (actionable errors read 537 against a red line of 800; scheduler
timeouts 193 against yellow at 150). They were fleet-wide SUMS, so they
scale with phone count rather than with health. The fleet onboarded today,
so no representative 24h baseline exists to replace them with — panels are
comparative (worst offenders) until a week of real data exists. The one
surviving threshold is "phones failing provisioning" >= 1, which is a
qualitative fault rather than a magnitude.
- Stats now count AFFECTED DEVICES, not fleet-wide line totals.
- "Phones reporting" -> "Phones heard from", documented as NOT a health
count: most of the fleet runs at Yealink default log level 3 (Error) and
is silent unless something breaks. A silent phone is a healthy phone.
- Registration/DHCP panels labelled level-6-only: they key on info/notice
lines that level-3 phones never send, so an absent phone there means
"quiet", not "unregistered".
- Added "loudest phones by log volume" to surface handsets left at level 6
(two of them are ~80% of all volume).
- refresh 1m -> 5m.
Verified by firing all 14 panel queries CONCURRENTLY against a sink loaded
with replayed real data: 14/14 return data, 0 rejected. Last time these were
verified sequentially, which is what let the 429 bug ship.
Note: data predating the MAC rollout has no mac label and won't appear.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Symptom: the Yealink dashboard showed data, then panels went empty. Not data
loss — Loki has every line (per-device counts only ever grew; no ingestion
errors, no rejections, compactor deleted nothing). Loki was rejecting the
dashboard's own queries with HTTP 429 "too many outstanding requests" for ~45%
of requests (53 rejected vs 64 OK in 30 min), so on each refresh a random
subset of panels won the race and the rest blanked.
Cause: split_queries_by_interval defaulted to 30m, so each 24h panel query was
split into 48 sub-queries. 14 panels refreshing together = ~672 sub-queries
against a scheduler queue of 100, on a 2-core host.
Reproduced deterministically: 14 identical queries fired sequentially = 14/14
OK; the same 14 fired concurrently = 5/14 OK, 9 rejected.
Fix:
- split_queries_by_interval: 24h — 48 sub-queries become 1, and it matches the
schema_config index period. This fleet produces ~35k lines/day; splitting
buys nothing at this scale.
- query_scheduler.max_outstanding_requests_per_tenant: 2048 (was 100) — headroom
so one refresh cannot overflow the queue.
- max_query_parallelism: 8 (was 32) — realistic for 2 cores.
Validated on a disposable loki:2.9.8 container with its own data dir before
commit: starts clean, and /config confirms split_queries_by_interval=1d,
max_query_parallelism=8, max_outstanding_requests_per_tenant=2048.
Note for future dashboard work: the panel queries were individually verified
against live Loki before the dashboard shipped, but sequentially — which never
exercised the concurrency a dashboard actually creates. Verify dashboards by
loading them, not by looping their queries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phones are ~99% of this Loki's log volume and nothing surfaced them. Adds a
file-provisioned dashboard (uid yealink-phones, folder "Network") plus the
provider config and the two grafana bind mounts it needs.
Built against a 24h baseline of real handset traffic; all 14 panel queries were
executed against live Loki before commit. Three findings shaped it, each of
which contradicts the obvious reading of the data:
- severity=emergency is NOT an emergency. All 24 emergency lines are the phone
printing its own log-level table at boot ("sys log :type=1,E=3,W=4,N=5,I=6,D=7")
— Yealink emits its logging config at level 0. Alerting on it would be 100%
false positive. Filtered out.
- "LSYS<3+error> rtpcap get len not enough" is 467 of 830 error lines (56%) — a
noise floor, not a fault. Excluded from every "actionable" panel.
- The hostname label cannot identify a device: its values are Yealink subsystems
(sua, GUI, sys, cfg, ipp, dev, WEB, ATP), because the phones put the subsystem
in the RFC3164 HOSTNAME field. Everything keys on source_ip. Yealink lines are
separated from switch/router traffic on the same Loki by the |~ "<[0-7][+]"
module marker (26,450 Yealink vs 33 non-Yealink lines in the baseline).
Thresholds are seeded from measured 24h counts: check passwd err 21 (auth),
data_task schedule time out 117 (handset scheduler overrunning its 30s
threshold — the best "phone is unwell" proxy), tftp/provisioning failures 3,
Register: update server 175 (SIP beat, 120s period), DHCP lease 9.
Caveat: only two handsets report today, so thresholds are baselined on a
2-phone sample and want revisiting once the fleet is onboarded.
Dashboards are file-provisioned, so UI edits are not persisted — change the
JSON here and redeploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator requirement: logs must persist at least one year. Applied
live (data-preserving restart, verified prior data survived) and
here for the next redeploy to stay in sync.
Confirmed via packet capture: a Yealink phone and (per RouterOS's known
default behavior) the fleet's routers send classic BSD-style syslog,
not RFC5424. Alloy's loki.source.syslog defaults to RFC5424-only and
was silently dropping every message ("expecting a version value in the
range 1-999"). syslog_format = "rfc3164" fixes it, but that argument
only exists from Alloy v1.5.0 onward (confirmed empirically against
v1.4.3, which fails config load) — bump the pin accordingly. v1.5.0 is
an adjacent minor release, not the kind of large version jump that hit
the Loki/Grafana :latest kernel-incompatibility bug fixed earlier.
:latest images (loki 3.7.3/go1.26.4, presumably grafana/alloy too) reset
every TCP connection on this host — kernel is 5.16.7 (Fedora 35, EOL
since 2023), too old for whatever the newer Go toolchains need.
Confirmed via a disposable grafana/loki:2.9.8 test container: same
config, same host, works cleanly. Pin all three images to older stable
releases instead of chasing this again on every redeploy.
Also fixes an unrelated config bug: instance_addr belongs directly
under `common:`, not nested under `common.ring:` — the misplaced field
was silently ignored, which didn't help but wasn't the actual cause.
Alloy receives syslog on 514/udp and 1514/tcp from network devices,
forwards to a local Loki instance; Grafana provisioned with Loki as
default datasource. Compose bind-mounts config from
/mnt/containers/logging/... on the host (staged separately) since
Portainer's GitOps pull only fetches the compose file, not the repo's
other files.
pipistrello
