pipistrelloandClaude Opus 4.8
13b1887a16
Phones are ~99% of this Loki's log volume and nothing surfaced them. Adds a
file-provisioned dashboard (uid yealink-phones, folder "Network") plus the
provider config and the two grafana bind mounts it needs.
Built against a 24h baseline of real handset traffic; all 14 panel queries were
executed against live Loki before commit. Three findings shaped it, each of
which contradicts the obvious reading of the data:
- severity=emergency is NOT an emergency. All 24 emergency lines are the phone
printing its own log-level table at boot ("sys log :type=1,E=3,W=4,N=5,I=6,D=7")
— Yealink emits its logging config at level 0. Alerting on it would be 100%
false positive. Filtered out.
- "LSYS<3+error> rtpcap get len not enough" is 467 of 830 error lines (56%) — a
noise floor, not a fault. Excluded from every "actionable" panel.
- The hostname label cannot identify a device: its values are Yealink subsystems
(sua, GUI, sys, cfg, ipp, dev, WEB, ATP), because the phones put the subsystem
in the RFC3164 HOSTNAME field. Everything keys on source_ip. Yealink lines are
separated from switch/router traffic on the same Loki by the |~ "<[0-7][+]"
module marker (26,450 Yealink vs 33 non-Yealink lines in the baseline).
Thresholds are seeded from measured 24h counts: check passwd err 21 (auth),
data_task schedule time out 117 (handset scheduler overrunning its 30s
threshold — the best "phone is unwell" proxy), tftp/provisioning failures 3,
Register: update server 175 (SIP beat, 120s period), DHCP lease 9.
Caveat: only two handsets report today, so thresholds are baselined on a
2-phone sample and want revisiting once the fleet is onboarded.
Dashboards are file-provisioned, so UI edits are not persisted — change the
JSON here and redeploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
69 lines
2.4 KiB
YAML
69 lines
2.4 KiB
YAML
version: '3.8'
|
|
|
|
services:
|
|
loki:
|
|
# Pinned, not :latest — this host's kernel (5.16.7, Fedora 35 from 2022) resets
|
|
# every TCP connection to containers built with newer Go toolchains
|
|
# (confirmed: grafana/loki:latest, built with go1.26.4, fails; 2.9.8 works).
|
|
image: grafana/loki:2.9.8
|
|
container_name: loki
|
|
command: -config.file=/etc/loki/config.yaml
|
|
restart: unless-stopped
|
|
volumes:
|
|
- /mnt/containers/logging/loki-config/config.yaml:/etc/loki/config.yaml:Z
|
|
- /mnt/containers/logging/loki-data:/loki:Z
|
|
networks:
|
|
- logging-nw
|
|
|
|
alloy:
|
|
# Pinned for the same reason as loki — see comment there. v1.5.0
|
|
# specifically (not v1.4.3) because it's the first release with
|
|
# syslog_format = "rfc3164" support, needed below.
|
|
image: grafana/alloy:v1.5.0
|
|
container_name: alloy
|
|
command:
|
|
- run
|
|
- --server.http.listen-addr=0.0.0.0:12345
|
|
- --storage.path=/var/lib/alloy/data
|
|
- /etc/alloy/config.alloy
|
|
restart: unless-stopped
|
|
depends_on:
|
|
- loki
|
|
volumes:
|
|
- /mnt/containers/logging/alloy-config/config.alloy:/etc/alloy/config.alloy:Z
|
|
- /mnt/containers/logging/alloy-data:/var/lib/alloy/data:Z
|
|
ports:
|
|
# syslog intake for routers/switches/phones (UDP, RFC3164/5424)
|
|
- '514:514/udp'
|
|
# optional TCP syslog for devices that don't do UDP
|
|
- '1514:1514/tcp'
|
|
networks:
|
|
- logging-nw
|
|
|
|
grafana:
|
|
# Pinned for the same reason as loki — see comment there.
|
|
image: grafana/grafana:10.4.2
|
|
container_name: grafana
|
|
restart: unless-stopped
|
|
depends_on:
|
|
- loki
|
|
environment:
|
|
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}
|
|
- GF_USERS_ALLOW_SIGN_UP=false
|
|
volumes:
|
|
- /mnt/containers/logging/grafana-data:/var/lib/grafana:Z
|
|
- /mnt/containers/logging/grafana-config/provisioning/datasources:/etc/grafana/provisioning/datasources:Z
|
|
# Dashboard provisioning: the provider config below points at the dashboards
|
|
# dir; both are file-provisioned, so UI edits are not persisted — edit the
|
|
# JSON in git and redeploy.
|
|
- /mnt/containers/logging/grafana-config/provisioning/dashboards:/etc/grafana/provisioning/dashboards:Z
|
|
- /mnt/containers/logging/grafana-config/dashboards:/var/lib/grafana/dashboards:Z
|
|
networks:
|
|
- logging-nw
|
|
- reverseproxy-nw
|
|
|
|
networks:
|
|
logging-nw:
|
|
reverseproxy-nw:
|
|
external: true
|