Pin alexanderzobnin-zabbix-app 6.5.0 for synchronous startup installation on Grafana 13.1.0, keeping the plugin reproducible across Portainer redeploys.
Reconstruct Zabbix dashboard 1 as a provisioned Grafana dashboard with eight site rows, 27 full-width traffic/storage graphs, a one-hour default range, and a computer-selectable failed-logon panel. Traffic errors and discards use a separate packets-per-second right axis.
Validated the compose model with Docker Compose 5.3.0, parsed the dashboard JSON, confirmed the Zabbix datasource against API 7.0.24, and ran all 28 panel queries: 263 frames returned with no query errors. Five panels are blank because their source Zabbix items have no retained history.
Phones are ~99% of this Loki's log volume and nothing surfaced them. Adds a
file-provisioned dashboard (uid yealink-phones, folder "Network") plus the
provider config and the two grafana bind mounts it needs.
Built against a 24h baseline of real handset traffic; all 14 panel queries were
executed against live Loki before commit. Three findings shaped it, each of
which contradicts the obvious reading of the data:
- severity=emergency is NOT an emergency. All 24 emergency lines are the phone
printing its own log-level table at boot ("sys log :type=1,E=3,W=4,N=5,I=6,D=7")
— Yealink emits its logging config at level 0. Alerting on it would be 100%
false positive. Filtered out.
- "LSYS<3+error> rtpcap get len not enough" is 467 of 830 error lines (56%) — a
noise floor, not a fault. Excluded from every "actionable" panel.
- The hostname label cannot identify a device: its values are Yealink subsystems
(sua, GUI, sys, cfg, ipp, dev, WEB, ATP), because the phones put the subsystem
in the RFC3164 HOSTNAME field. Everything keys on source_ip. Yealink lines are
separated from switch/router traffic on the same Loki by the |~ "<[0-7][+]"
module marker (26,450 Yealink vs 33 non-Yealink lines in the baseline).
Thresholds are seeded from measured 24h counts: check passwd err 21 (auth),
data_task schedule time out 117 (handset scheduler overrunning its 30s
threshold — the best "phone is unwell" proxy), tftp/provisioning failures 3,
Register: update server 175 (SIP beat, 120s period), DHCP lease 9.
Caveat: only two handsets report today, so thresholds are baselined on a
2-phone sample and want revisiting once the fleet is onboarded.
Dashboards are file-provisioned, so UI edits are not persisted — change the
JSON here and redeploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Confirmed via packet capture: a Yealink phone and (per RouterOS's known
default behavior) the fleet's routers send classic BSD-style syslog,
not RFC5424. Alloy's loki.source.syslog defaults to RFC5424-only and
was silently dropping every message ("expecting a version value in the
range 1-999"). syslog_format = "rfc3164" fixes it, but that argument
only exists from Alloy v1.5.0 onward (confirmed empirically against
v1.4.3, which fails config load) — bump the pin accordingly. v1.5.0 is
an adjacent minor release, not the kind of large version jump that hit
the Loki/Grafana :latest kernel-incompatibility bug fixed earlier.
:latest images (loki 3.7.3/go1.26.4, presumably grafana/alloy too) reset
every TCP connection on this host — kernel is 5.16.7 (Fedora 35, EOL
since 2023), too old for whatever the newer Go toolchains need.
Confirmed via a disposable grafana/loki:2.9.8 test container: same
config, same host, works cleanly. Pin all three images to older stable
releases instead of chasing this again on every redeploy.
Also fixes an unrelated config bug: instance_addr belongs directly
under `common:`, not nested under `common.ring:` — the misplaced field
was silently ignored, which didn't help but wasn't the actual cause.
Alloy receives syslog on 514/udp and 1514/tcp from network devices,
forwards to a local Loki instance; Grafana provisioned with Loki as
default datasource. Compose bind-mounts config from
/mnt/containers/logging/... on the host (staged separately) since
Portainer's GitOps pull only fetches the compose file, not the repo's
other files.
pipistrello
