Pin alexanderzobnin-zabbix-app 6.5.0 for synchronous startup installation on Grafana 13.1.0, keeping the plugin reproducible across Portainer redeploys.
Reconstruct Zabbix dashboard 1 as a provisioned Grafana dashboard with eight site rows, 27 full-width traffic/storage graphs, a one-hour default range, and a computer-selectable failed-logon panel. Traffic errors and discards use a separate packets-per-second right axis.
Validated the compose model with Docker Compose 5.3.0, parsed the dashboard JSON, confirmed the Zabbix datasource against API 7.0.24, and ran all 28 panel queries: 263 frames returned with no query errors. Five panels are blank because their source Zabbix items have no retained history.
Verbose Yealink traces show LDAP binds fail before authentication because the provisioned short hostname SERVERPDC cannot be resolved. The failures started before the separate AD unauthenticated-bind hardening, so credentials and that security change are not the cause.\n\nInclude the explicit domain-resolution signature in current-state, trend, top-phone, and operational-log queries. Update panel guidance to direct investigation toward a resolvable LDAP FQDN or stable IP.\n\nThe added LogQL alternative was executed successfully against production Loki 3.7.2.
The Yealink dashboard treated nearly every level-3 line as actionable, making routine boot and firmware chatter look like fleet-wide failures.\n\nReplace generic error counts with recent LDAP, provisioning, and internal-database signals. Add dedicated operational trend and log panels, and relabel raw severity as diagnostic-only while retaining web-password, registration, and DHCP context.\n\nValidated all 17 LogQL targets directly against Loki 3.7.2. In the inspected 24-hour sample, provisioning had settled, LDAP failures remained active, and one handset reported a malformed internal database.
Two changes driven by the fleet syslog rollout (~4 -> 106 senders) and the
operator enabling Yealink's prepend-MAC on the autoprovisioned handsets.
1. Alloy: lift the MAC into a real `mac` label.
UDP syslog offers only three possible identities: sender IP, RFC3164
HOSTNAME, message body.
- source_ip is DHCP (observed lease 1200s) so it is NOT stable: a re-IP
looks like a new device, and a recycled IP silently inherits another
phone's history.
- HOSTNAME is useless: Yealink puts its subsystem there (sua/GUI/cfg/sys),
and after prepend-MAC the bracketed MAC lands in that slot and is rejected
as a hostname, so the label is simply absent on new lines.
So the body is the only place identity exists. A loki.process regex stage
extracts it. Non-Yealink senders (the D-Link switches) don't match and pass
through with no mac label. Cardinality is safe: mac is 1:1 with a device and
does not multiply against source_ip.
Verified on a disposable alloy+loki rig by replaying 406 REAL captured lines:
0 parse errors, both branches confirmed — Yealink lines get
mac=80:5e:0c:b2:44:d3, the switch line passes through with no mac label.
2. Dashboard: identity by MAC, comparative panels, no invented thresholds.
- Variable is now label_values(mac) with allValue ".+" (not ".*") so "All"
matches only streams that HAVE a mac — excluding switches automatically.
- Removed the absolute colour thresholds. They were derived from a ~4-hour
sample of TWO atypically verbose handsets and were already wrong at 2
phones (actionable errors read 537 against a red line of 800; scheduler
timeouts 193 against yellow at 150). They were fleet-wide SUMS, so they
scale with phone count rather than with health. The fleet onboarded today,
so no representative 24h baseline exists to replace them with — panels are
comparative (worst offenders) until a week of real data exists. The one
surviving threshold is "phones failing provisioning" >= 1, which is a
qualitative fault rather than a magnitude.
- Stats now count AFFECTED DEVICES, not fleet-wide line totals.
- "Phones reporting" -> "Phones heard from", documented as NOT a health
count: most of the fleet runs at Yealink default log level 3 (Error) and
is silent unless something breaks. A silent phone is a healthy phone.
- Registration/DHCP panels labelled level-6-only: they key on info/notice
lines that level-3 phones never send, so an absent phone there means
"quiet", not "unregistered".
- Added "loudest phones by log volume" to surface handsets left at level 6
(two of them are ~80% of all volume).
- refresh 1m -> 5m.
Verified by firing all 14 panel queries CONCURRENTLY against a sink loaded
with replayed real data: 14/14 return data, 0 rejected. Last time these were
verified sequentially, which is what let the 429 bug ship.
Note: data predating the MAC rollout has no mac label and won't appear.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Symptom: the Yealink dashboard showed data, then panels went empty. Not data
loss — Loki has every line (per-device counts only ever grew; no ingestion
errors, no rejections, compactor deleted nothing). Loki was rejecting the
dashboard's own queries with HTTP 429 "too many outstanding requests" for ~45%
of requests (53 rejected vs 64 OK in 30 min), so on each refresh a random
subset of panels won the race and the rest blanked.
Cause: split_queries_by_interval defaulted to 30m, so each 24h panel query was
split into 48 sub-queries. 14 panels refreshing together = ~672 sub-queries
against a scheduler queue of 100, on a 2-core host.
Reproduced deterministically: 14 identical queries fired sequentially = 14/14
OK; the same 14 fired concurrently = 5/14 OK, 9 rejected.
Fix:
- split_queries_by_interval: 24h — 48 sub-queries become 1, and it matches the
schema_config index period. This fleet produces ~35k lines/day; splitting
buys nothing at this scale.
- query_scheduler.max_outstanding_requests_per_tenant: 2048 (was 100) — headroom
so one refresh cannot overflow the queue.
- max_query_parallelism: 8 (was 32) — realistic for 2 cores.
Validated on a disposable loki:2.9.8 container with its own data dir before
commit: starts clean, and /config confirms split_queries_by_interval=1d,
max_query_parallelism=8, max_outstanding_requests_per_tenant=2048.
Note for future dashboard work: the panel queries were individually verified
against live Loki before the dashboard shipped, but sequentially — which never
exercised the concurrency a dashboard actually creates. Verify dashboards by
loading them, not by looping their queries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phones are ~99% of this Loki's log volume and nothing surfaced them. Adds a
file-provisioned dashboard (uid yealink-phones, folder "Network") plus the
provider config and the two grafana bind mounts it needs.
Built against a 24h baseline of real handset traffic; all 14 panel queries were
executed against live Loki before commit. Three findings shaped it, each of
which contradicts the obvious reading of the data:
- severity=emergency is NOT an emergency. All 24 emergency lines are the phone
printing its own log-level table at boot ("sys log :type=1,E=3,W=4,N=5,I=6,D=7")
— Yealink emits its logging config at level 0. Alerting on it would be 100%
false positive. Filtered out.
- "LSYS<3+error> rtpcap get len not enough" is 467 of 830 error lines (56%) — a
noise floor, not a fault. Excluded from every "actionable" panel.
- The hostname label cannot identify a device: its values are Yealink subsystems
(sua, GUI, sys, cfg, ipp, dev, WEB, ATP), because the phones put the subsystem
in the RFC3164 HOSTNAME field. Everything keys on source_ip. Yealink lines are
separated from switch/router traffic on the same Loki by the |~ "<[0-7][+]"
module marker (26,450 Yealink vs 33 non-Yealink lines in the baseline).
Thresholds are seeded from measured 24h counts: check passwd err 21 (auth),
data_task schedule time out 117 (handset scheduler overrunning its 30s
threshold — the best "phone is unwell" proxy), tftp/provisioning failures 3,
Register: update server 175 (SIP beat, 120s period), DHCP lease 9.
Caveat: only two handsets report today, so thresholds are baselined on a
2-phone sample and want revisiting once the fleet is onboarded.
Dashboards are file-provisioned, so UI edits are not persisted — change the
JSON here and redeploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator requirement: logs must persist at least one year. Applied
live (data-preserving restart, verified prior data survived) and
here for the next redeploy to stay in sync.
Confirmed via packet capture: a Yealink phone and (per RouterOS's known
default behavior) the fleet's routers send classic BSD-style syslog,
not RFC5424. Alloy's loki.source.syslog defaults to RFC5424-only and
was silently dropping every message ("expecting a version value in the
range 1-999"). syslog_format = "rfc3164" fixes it, but that argument
only exists from Alloy v1.5.0 onward (confirmed empirically against
v1.4.3, which fails config load) — bump the pin accordingly. v1.5.0 is
an adjacent minor release, not the kind of large version jump that hit
the Loki/Grafana :latest kernel-incompatibility bug fixed earlier.
:latest images (loki 3.7.3/go1.26.4, presumably grafana/alloy too) reset
every TCP connection on this host — kernel is 5.16.7 (Fedora 35, EOL
since 2023), too old for whatever the newer Go toolchains need.
Confirmed via a disposable grafana/loki:2.9.8 test container: same
config, same host, works cleanly. Pin all three images to older stable
releases instead of chasing this again on every redeploy.
Also fixes an unrelated config bug: instance_addr belongs directly
under `common:`, not nested under `common.ring:` — the misplaced field
was silently ignored, which didn't help but wasn't the actual cause.
Alloy receives syslog on 514/udp and 1514/tcp from network devices,
forwards to a local Loki instance; Grafana provisioned with Loki as
default datasource. Compose bind-mounts config from
/mnt/containers/logging/... on the host (staged separately) since
Portainer's GitOps pull only fetches the compose file, not the repo's
other files.
pipistrello
