Files
docker-infrastructure/logging/grafana-dashboards/yealink-phones.json
T
pipistrello a4aa0e90fe grafana: expose phone LDAP DNS root cause
Verbose Yealink traces show LDAP binds fail before authentication because the provisioned short hostname SERVERPDC cannot be resolved. The failures started before the separate AD unauthenticated-bind hardening, so credentials and that security change are not the cause.\n\nInclude the explicit domain-resolution signature in current-state, trend, top-phone, and operational-log queries. Update panel guidance to direct investigation toward a resolvable LDAP FQDN or stable IP.\n\nThe added LogQL alternative was executed successfully against production Loki 3.7.2.
2026-07-17 14:25:30 +03:00

464 lines
21 KiB
JSON

{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": { "type": "grafana", "uid": "-- Grafana --" },
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"links": [],
"liveNow": false,
"panels": [
{
"type": "text",
"title": "How to read this dashboard",
"gridPos": { "h": 5, "w": 24, "x": 0, "y": 0 },
"id": 100,
"options": {
"mode": "markdown",
"content": "**Identity is the phone's MAC**, lifted out of the message body into a `mac` label by Alloy (Yealink `static.syslog.prepend_mac_address.enable`). Do **not** use `source_ip` as identity — phones are on DHCP with a 1200 s lease, so a re-IP looks like a new device and a recycled IP would inherit another phone's history. Only devices sending a MAC appear here, so switches sharing this Loki are excluded automatically. *Data from before the MAC rollout has no `mac` label and will not show.*\n\n**Yealink's `error` severity is diagnostic verbosity, not a fleet health verdict.** Firmware-internal messages such as missing optional DHCP attributes, GUI event targets, RTP capture reads, scheduler overruns near the built-in 30 s threshold, kernel register dumps, and provisioning-engine state account for most red lines. Use the narrow **Operational signals** panels first; use raw severity only when investigating one handset.\n\n**The top status cards are intentionally recent.** LDAP and internal-database cards cover the last 15 minutes; provisioning covers the last hour. This prevents a completed fleet rollout or reboot from looking like a current outage for the rest of a 24 h dashboard range. LDAP failures affect directory/search and caller-name lookup, not SIP calling itself.\n\n**A silent phone is not proven healthy or offline.** Most of the fleet runs at log level 3 and only sends selected errors. Registration and DHCP evidence at the bottom is available only from verbose level-6 phones. `severity=emergency` is also excluded from the severity chart because Yealink uses it while printing its boot-time log-level table, not to report an emergency."
}
},
{
"type": "stat",
"title": "Phones heard from",
"description": "Distinct MACs that sent at least one line in range. NOT a health count — level-3 phones stay silent unless something breaks, so this is expected to sit well below the fleet size.",
"gridPos": { "h": 4, "w": 4, "x": 0, "y": 5 },
"id": 1,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "count(count by (mac) (count_over_time({job=\"syslog\", mac=~\"$phone\"} [$__range])))",
"queryType": "instant"
}
],
"options": {
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"colorMode": "none",
"graphMode": "none",
"textMode": "auto"
},
"fieldConfig": { "defaults": { "unit": "short", "color": { "mode": "fixed", "fixedColor": "text" } }, "overrides": [] }
},
{
"type": "stat",
"title": "Log lines",
"description": "Total lines from MAC-identified phones in range.",
"gridPos": { "h": 4, "w": 4, "x": 4, "y": 5 },
"id": 2,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "sum(count_over_time({job=\"syslog\", mac=~\"$phone\"} [$__range]))",
"queryType": "instant"
}
],
"options": {
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"colorMode": "none",
"graphMode": "none",
"textMode": "auto"
},
"fieldConfig": { "defaults": { "unit": "short", "color": { "mode": "fixed", "fixedColor": "text" } }, "overrides": [] }
},
{
"type": "stat",
"title": "Active LDAP failures (15m)",
"description": "Distinct phones that could not resolve or contact the LDAP directory in the last 15 minutes. Verbose evidence identifies short-name DNS resolution of SERVERPDC as the cause. This can break directory/caller-name lookup, but not SIP calling itself.",
"gridPos": { "h": 4, "w": 4, "x": 8, "y": 5 },
"id": 3,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "count(count by (mac) (count_over_time({job=\"syslog\", mac=~\"$phone\"} |~ \"Domain name resolution failed|ldap_sasl_bind_s|Can't contact LDAP server\" [15m])))",
"queryType": "instant"
}
],
"options": {
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
},
"fieldConfig": { "defaults": { "unit": "short", "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 1 }] } }, "overrides": [] }
},
{
"type": "stat",
"title": "Recent provisioning failures (1h)",
"description": "Distinct phones with TFTP/provisioning download failures in the last hour. A short burst during reboot or fleet rollout is expected; a value that remains non-zero deserves investigation.",
"gridPos": { "h": 4, "w": 4, "x": 12, "y": 5 },
"id": 4,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "count(count by (mac) (count_over_time({job=\"syslog\", mac=~\"$phone\"} |~ \"tftp to file failed|no config Provisioning Server|DURL<3[+]error\" [1h])))",
"queryType": "instant"
}
],
"options": {
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
},
"fieldConfig": {
"defaults": {
"unit": "short",
"color": { "mode": "thresholds" },
"thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 1 }] }
},
"overrides": []
}
},
{
"type": "stat",
"title": "Web password rejects",
"description": "Phones logging CUIT 'check passwd err' in the selected range. This is a rejected login to the handset web/UI, not SIP authentication; occasional entries are usually operator mistakes or background probing.",
"gridPos": { "h": 4, "w": 4, "x": 16, "y": 5 },
"id": 5,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "count(count by (mac) (count_over_time({job=\"syslog\", mac=~\"$phone\"} |~ \"check passwd err\" [$__range])))",
"queryType": "instant"
}
],
"options": {
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"colorMode": "none",
"graphMode": "none",
"textMode": "auto"
},
"fieldConfig": { "defaults": { "unit": "short", "color": { "mode": "fixed", "fixedColor": "text" } }, "overrides": [] }
},
{
"type": "stat",
"title": "Internal DB faults (15m)",
"description": "Phones reporting a malformed internal database or related GUI virtual-machine failures in the last 15 minutes. Unlike ordinary firmware chatter, a persistent value can affect local phone applications and should be checked.",
"gridPos": { "h": 4, "w": 4, "x": 20, "y": 5 },
"id": 6,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "count(count by (mac) (count_over_time({job=\"syslog\", mac=~\"$phone\"} |~ \"database disk image is malformed|Null Virtual Machine pointer|Invalid scalar query\" [15m])))",
"queryType": "instant"
}
],
"options": {
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"colorMode": "value",
"graphMode": "none",
"textMode": "auto"
},
"fieldConfig": { "defaults": { "unit": "short", "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 1 }] } }, "overrides": [] }
},
{
"type": "bargauge",
"title": "Phones by operational-signal events",
"description": "Top phones for the narrow signals this dashboard can interpret: LDAP DNS/connectivity, internal database faults, provisioning/download failures, and SIP DNS timeouts. Keyed on MAC so DHCP changes do not split identity.",
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 9 },
"id": 7,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "topk(10, sum by (mac) (count_over_time({job=\"syslog\", mac=~\"$phone\"} |~ \"Domain name resolution failed|ldap_sasl_bind_s|Can't contact LDAP server|database disk image is malformed|Null Virtual Machine pointer|Invalid scalar query|tftp to file failed|no config Provisioning Server|DURL<3[+]error|Timeout while contacting DNS servers\" [$__range])))",
"queryType": "instant",
"legendFormat": "{{mac}}"
}
],
"options": {
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"displayMode": "gradient",
"orientation": "horizontal",
"showUnfilled": true
},
"fieldConfig": { "defaults": { "unit": "short", "color": { "mode": "continuous-GrYlRd" } }, "overrides": [] }
},
{
"type": "bargauge",
"title": "Loudest phones by log volume",
"description": "Identifies handsets left at log level 6 (Informational). Yealink's default is level 3 (Error); a phone near the top here is verbose by configuration, not necessarily unhealthy — but it dominates disk and query cost.",
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 9 },
"id": 8,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "topk(10, sum by (mac) (count_over_time({job=\"syslog\", mac=~\"$phone\"} [$__range])))",
"queryType": "instant",
"legendFormat": "{{mac}}"
}
],
"options": {
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"displayMode": "gradient",
"orientation": "horizontal",
"showUnfilled": true
},
"fieldConfig": { "defaults": { "unit": "short", "color": { "mode": "continuous-BlPu" } }, "overrides": [] }
},
{
"type": "timeseries",
"title": "Operational signals over time",
"description": "Narrow, interpretable signals separated from firmware error-severity noise. LDAP short-name DNS failure affects directory lookup; internal DB affects local apps; provisioning should settle after boot; SIP DNS timeouts can prevent registration.",
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 18 },
"id": 9,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "sum(count_over_time({job=\"syslog\", mac=~\"$phone\"} |~ \"Domain name resolution failed|ldap_sasl_bind_s|Can't contact LDAP server\" [$__interval]))",
"queryType": "range",
"legendFormat": "LDAP directory"
},
{
"refId": "B",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "sum(count_over_time({job=\"syslog\", mac=~\"$phone\"} |~ \"database disk image is malformed|Null Virtual Machine pointer|Invalid scalar query\" [$__interval]))",
"queryType": "range",
"legendFormat": "Internal DB / GUI"
},
{
"refId": "C",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "sum(count_over_time({job=\"syslog\", mac=~\"$phone\"} |~ \"tftp to file failed|no config Provisioning Server|DURL<3[+]error\" [$__interval]))",
"queryType": "range",
"legendFormat": "Provisioning / download"
},
{
"refId": "D",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "sum(count_over_time({job=\"syslog\", mac=~\"$phone\"} |~ \"Timeout while contacting DNS servers\" [$__interval]))",
"queryType": "range",
"legendFormat": "SIP DNS timeout"
}
],
"options": {
"legend": { "displayMode": "table", "placement": "right", "showLegend": true, "calcs": ["sum"] },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": { "drawStyle": "bars", "lineWidth": 0, "fillOpacity": 80, "stacking": { "mode": "normal", "group": "A" }, "showPoints": "never" }
},
"overrides": []
}
},
{
"type": "timeseries",
"title": "Diagnostic severity volume (not health)",
"description": "Shows what the firmware labeled by severity. Do not interpret the red area as failed phones: observed error-level traffic is dominated by benign internal diagnostics. Emergency is excluded because Yealink uses it for a boot-time log-level table.",
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 18 },
"id": 10,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "sum by (severity) (count_over_time({job=\"syslog\", mac=~\"$phone\", severity!=\"emergency\"} [$__interval]))",
"queryType": "range",
"legendFormat": "{{severity}}"
}
],
"options": {
"legend": { "displayMode": "list", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": { "drawStyle": "bars", "lineWidth": 0, "fillOpacity": 70, "stacking": { "mode": "normal", "group": "A" }, "showPoints": "never" }
},
"overrides": [
{ "matcher": { "id": "byName", "options": "error" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "red" } }] },
{ "matcher": { "id": "byName", "options": "warning" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "orange" } }] },
{ "matcher": { "id": "byName", "options": "notice" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "blue" } }] },
{ "matcher": { "id": "byName", "options": "informational" }, "properties": [{ "id": "color", "value": { "mode": "fixed", "fixedColor": "green" } }] }
]
}
},
{
"type": "logs",
"title": "Operational signals — investigate these first",
"description": "LDAP DNS/connectivity, internal database faults, provisioning/download failures, and SIP DNS timeouts. LDAP affects directory features but not calls; verbose phones identify SERVERPDC short-name resolution as the current LDAP cause. Labels show stable MAC and current source IP.",
"gridPos": { "h": 10, "w": 24, "x": 0, "y": 26 },
"id": 11,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "{job=\"syslog\", mac=~\"$phone\"} |~ \"Domain name resolution failed|ldap_sasl_bind_s|Can't contact LDAP server|database disk image is malformed|Null Virtual Machine pointer|Invalid scalar query|tftp to file failed|no config Provisioning Server|DURL<3[+]error|Timeout while contacting DNS servers\"",
"queryType": "range"
}
],
"options": {
"showTime": true,
"showLabels": true,
"showCommonLabels": false,
"wrapLogMessage": true,
"enableLogDetails": true,
"dedupStrategy": "none",
"sortOrder": "Descending"
}
},
{
"type": "logs",
"title": "Raw firmware error severity (diagnostic only)",
"description": "For per-phone troubleshooting after the operational panels. These lines are not all actionable: Yealink marks substantial internal/boot chatter as error. Only the very high-volume RTP capture signature is removed to keep the panel usable.",
"gridPos": { "h": 10, "w": 24, "x": 0, "y": 36 },
"id": 12,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "{job=\"syslog\", mac=~\"$phone\", severity=\"error\"} !~ \"rtpcap get len not enough\"",
"queryType": "range"
}
],
"options": {
"showTime": true,
"showLabels": true,
"showCommonLabels": false,
"wrapLogMessage": true,
"enableLogDetails": true,
"dedupStrategy": "exact",
"sortOrder": "Descending"
}
},
{
"type": "timeseries",
"title": "SIP registration refresh (level-6 phones only)",
"description": "REG 'Register: update server' fires at info level, so ONLY handsets left at log level 6 appear here. A gap for such a phone means it lost registration to the PBX (192.168.0.8); an absent phone means it is at level 3, NOT that it is unregistered.",
"gridPos": { "h": 7, "w": 12, "x": 0, "y": 46 },
"id": 13,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "sum by (mac) (count_over_time({job=\"syslog\", mac=~\"$phone\"} |~ \"Register: update server\" [$__interval]))",
"queryType": "range",
"legendFormat": "{{mac}}"
}
],
"options": {
"legend": { "displayMode": "list", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi" }
},
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": { "drawStyle": "points", "pointSize": 5, "lineWidth": 1, "fillOpacity": 0, "showPoints": "always" }
},
"overrides": []
}
},
{
"type": "logs",
"title": "DHCP lease events (level-6 phones only)",
"description": "Lease acquisition/renewal as the phone sees it — notice level, so the same caveat as the registration panel. Useful when a handset is 'connected but dead'. Observed lease time is 1200 s, which is exactly why source_ip is not a safe identity.",
"gridPos": { "h": 7, "w": 12, "x": 12, "y": 46 },
"id": 14,
"datasource": { "type": "loki", "uid": "${datasource}" },
"targets": [
{
"refId": "A",
"datasource": { "type": "loki", "uid": "${datasource}" },
"expr": "{job=\"syslog\", mac=~\"$phone\"} |~ \"lease of .* obtained|lease renew\"",
"queryType": "range"
}
],
"options": {
"showTime": true,
"showLabels": true,
"showCommonLabels": false,
"wrapLogMessage": true,
"enableLogDetails": true,
"dedupStrategy": "none",
"sortOrder": "Descending"
}
}
],
"refresh": "5m",
"schemaVersion": 39,
"tags": ["yealink", "phones", "syslog", "02-tekhnohim"],
"templating": {
"list": [
{
"name": "datasource",
"label": "Datasource",
"type": "datasource",
"query": "loki",
"current": {},
"hide": 0,
"refresh": 1,
"regex": "",
"skipUrlSync": false
},
{
"name": "phone",
"label": "Phone (MAC)",
"description": "Device identity. allValue is '.+' (not '.*') so that 'All' matches only streams that HAVE a mac label — which excludes switches and other non-Yealink senders sharing this Loki.",
"type": "query",
"datasource": { "type": "loki", "uid": "${datasource}" },
"definition": "label_values({job=\"syslog\"}, mac)",
"query": {
"label": "mac",
"refId": "LokiVariableQueryEditor-VariableQuery",
"stream": "{job=\"syslog\"}",
"type": 1
},
"current": { "selected": true, "text": ["All"], "value": ["$__all"] },
"includeAll": true,
"multi": true,
"allValue": ".+",
"hide": 0,
"refresh": 2,
"sort": 1,
"skipUrlSync": false
}
]
},
"time": { "from": "now-24h", "to": "now" },
"timepicker": {},
"timezone": "browser",
"title": "Yealink IP phones — syslog health",
"uid": "yealink-phones",
"version": 4,
"weekStart": ""
}