Files
pipistrelloandClaude Opus 4.8 f438f3845c Key Yealink phones by MAC, and stop pretending fleet-wide sums are thresholds
Two changes driven by the fleet syslog rollout (~4 -> 106 senders) and the
operator enabling Yealink's prepend-MAC on the autoprovisioned handsets.

1. Alloy: lift the MAC into a real `mac` label.

   UDP syslog offers only three possible identities: sender IP, RFC3164
   HOSTNAME, message body.
   - source_ip is DHCP (observed lease 1200s) so it is NOT stable: a re-IP
     looks like a new device, and a recycled IP silently inherits another
     phone's history.
   - HOSTNAME is useless: Yealink puts its subsystem there (sua/GUI/cfg/sys),
     and after prepend-MAC the bracketed MAC lands in that slot and is rejected
     as a hostname, so the label is simply absent on new lines.
   So the body is the only place identity exists. A loki.process regex stage
   extracts it. Non-Yealink senders (the D-Link switches) don't match and pass
   through with no mac label. Cardinality is safe: mac is 1:1 with a device and
   does not multiply against source_ip.

   Verified on a disposable alloy+loki rig by replaying 406 REAL captured lines:
   0 parse errors, both branches confirmed — Yealink lines get
   mac=80:5e:0c:b2:44:d3, the switch line passes through with no mac label.

2. Dashboard: identity by MAC, comparative panels, no invented thresholds.

   - Variable is now label_values(mac) with allValue ".+" (not ".*") so "All"
     matches only streams that HAVE a mac — excluding switches automatically.
   - Removed the absolute colour thresholds. They were derived from a ~4-hour
     sample of TWO atypically verbose handsets and were already wrong at 2
     phones (actionable errors read 537 against a red line of 800; scheduler
     timeouts 193 against yellow at 150). They were fleet-wide SUMS, so they
     scale with phone count rather than with health. The fleet onboarded today,
     so no representative 24h baseline exists to replace them with — panels are
     comparative (worst offenders) until a week of real data exists. The one
     surviving threshold is "phones failing provisioning" >= 1, which is a
     qualitative fault rather than a magnitude.
   - Stats now count AFFECTED DEVICES, not fleet-wide line totals.
   - "Phones reporting" -> "Phones heard from", documented as NOT a health
     count: most of the fleet runs at Yealink default log level 3 (Error) and
     is silent unless something breaks. A silent phone is a healthy phone.
   - Registration/DHCP panels labelled level-6-only: they key on info/notice
     lines that level-3 phones never send, so an absent phone there means
     "quiet", not "unregistered".
   - Added "loudest phones by log volume" to surface handsets left at level 6
     (two of them are ~80% of all volume).
   - refresh 1m -> 5m.

   Verified by firing all 14 panel queries CONCURRENTLY against a sink loaded
   with replayed real data: 14/14 return data, 0 rejected. Last time these were
   verified sequentially, which is what let the 429 bug ship.

Note: data predating the MAC rollout has no mac label and won't appear.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 19:55:09 +03:00

107 lines
3.2 KiB
Plaintext

// Receives syslog from routers, switches, and Yealink phones on this host's
// LAN interface, tags each message with useful labels, and ships it to the
// local Loki instance.
//
// Point network devices at this host's IP, port 514/udp (or 1514/tcp for
// devices that only speak TCP framing).
loki.relabel "syslog" {
forward_to = []
rule {
source_labels = ["__syslog_message_hostname"]
target_label = "hostname"
}
rule {
source_labels = ["__syslog_message_severity"]
target_label = "severity"
}
rule {
source_labels = ["__syslog_message_facility"]
target_label = "facility"
}
rule {
source_labels = ["__syslog_connection_ip_address"]
target_label = "source_ip"
}
}
// syslog_format is "rfc3164" (legacy BSD syslog, no version field) rather
// than the component's rfc5424 default — confirmed by packet capture that
// both the Yealink phones and RouterOS routers send rfc3164 in practice.
// A listener can only parse one format; if a device ever sends genuine
// rfc5424, it needs its own listener on a different port.
// Lifts the Yealink device MAC out of the message body into a real `mac` label.
//
// Why this is needed: UDP syslog gives us only three possible identities — the
// sender IP, the RFC3164 HOSTNAME field, and the message body.
// * source_ip is DHCP (observed lease time: 1200s), so it is NOT stable. A
// phone that re-IPs becomes a "new device", and one that inherits a
// recycled IP silently inherits another phone's history.
// * HOSTNAME is useless here: Yealink puts its internal *subsystem* there
// (sua, GUI, cfg, sys...). After enabling prepend-MAC the bracketed MAC
// occupies that slot and is rejected as a hostname, so the label is simply
// absent on new lines.
// So the body is the only place the device identity exists. Yealink's
// `static.syslog.prepend_mac_address.enable` puts it at the head of every line:
// [80:5e:0c:b2:44:d3] sua [824.1146]: FSM <6+info > [255] free nist ressource
//
// Non-Yealink senders (e.g. the D-Link switches) don't match the regex and are
// passed through untouched with no `mac` label — that is intentional.
//
// Cardinality is safe: mac is 1:1 with a device, so it does not multiply
// against source_ip (~100 phones -> ~100 values, and the pair is stable).
loki.process "extract_mac" {
forward_to = [loki.write.default.receiver]
stage.regex {
expression = "^\\[(?P<mac>(?:[0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2})\\]"
}
stage.labels {
values = {
mac = "mac",
}
}
}
loki.source.syslog "network_devices_udp" {
listener {
address = "0.0.0.0:514"
protocol = "udp"
syslog_format = "rfc3164"
labels = {
job = "syslog",
transport = "udp",
}
}
relabel_rules = loki.relabel.syslog.rules
forward_to = [loki.process.extract_mac.receiver]
}
loki.source.syslog "network_devices_tcp" {
listener {
address = "0.0.0.0:1514"
protocol = "tcp"
syslog_format = "rfc3164"
labels = {
job = "syslog",
transport = "tcp",
}
}
relabel_rules = loki.relabel.syslog.rules
forward_to = [loki.process.extract_mac.receiver]
}
loki.write "default" {
endpoint {
url = "http://loki:3100/loki/api/v1/push"
}
}