diff --git a/logging/README.md b/logging/README.md index 649c70a..75eac32 100644 --- a/logging/README.md +++ b/logging/README.md @@ -5,18 +5,24 @@ part of the manual `mktxp-stack`: - `loki` stores syslog in the persistent host path `/mnt/containers/logging/loki-data`; -- `alloy` replaces the `syslog-ng` -> `promtail` relay; -- `grafana` provides the existing MikroTik Loki dashboard. +- `syslog-ng` accepts and normalizes the fleet's mixed/raw syslog formats; +- `alloy` replaces Promtail and sends normalized RFC5424 to Loki; +- `grafana` provides the MikroTik Loki dashboard and a Telemt fleet dashboard + backed by the internal DEPOT Zabbix server. -The retired stack's `mktxp`, Prometheus, Promtail, and syslog-ng services are -not part of this stack. +The retired stack's `mktxp`, Prometheus, and Promtail services are not part of +this stack. syslog-ng remains because real DEPOT RouterOS traffic includes raw +messages that Alloy's strict RFC3164 parser rejects. ## Portainer settings - Repository: `https://git.it-depot.ru/depot/docker-infrastructure` - Branch: `master` - Compose path: `logging/docker-compose.yaml` -- Required environment variable: `GRAFANA_ADMIN_PASSWORD` +- Required environment variables: + - `GRAFANA_ADMIN_PASSWORD` + - `ZABBIX_API_TOKEN` — token for a read-only Zabbix API user; it is passed + into Grafana's provisioned datasource and must not be committed. Do not deploy the new stack before completing the cutover below: the current containers still own names `loki` and `grafana` and host ports 3000, 3100, 514, @@ -33,12 +39,14 @@ Before deployment, the repository files must be copied to: ├── alloy-data/ ├── grafana-config/ │ ├── dashboards/mikrotik-loki-logs.json +│ ├── dashboards/telemt-proxy-fleet.json │ └── provisioning/ │ ├── dashboards/dashboards.yaml -│ └── datasources/loki.yaml +│ └── datasources/{loki,zabbix}.yaml ├── grafana-data/ ├── loki-config/config.yaml -└── loki-data/ +├── loki-data/ +└── syslog-ng-config/syslog-ng.conf ``` SELinux is enforcing. The compose mounts use `:Z`; do not remove those flags. @@ -47,10 +55,11 @@ Loki `10001:10001`, Grafana `472:0`, and Alloy `0:0`. ## Cutover -The live Loki history is inside the old container at `/tmp/loki`. A stopped, -consistent rescue copy from 2026-07-30 is pre-seeded in the new host path, but -it must be refreshed from the stopped live container immediately before the -old stack is removed: +The live Loki history is inside the old container at `/tmp/loki`. Do not merge +a newer WAL over an older rescue WAL: the resulting non-contiguous segment +sequence makes Loki fail with `segments are not sequential`. Instead, move the +seed aside and copy the stopped container's complete data directory as one +generation: ```bash cd /mnt/containers/mktxp/mktxp-stack @@ -59,8 +68,10 @@ cd /mnt/containers/mktxp/mktxp-stack curl -fsS http://127.0.0.1:3100/ready docker compose stop -# The stopped Loki container still contains the newest WAL/chunks. -docker cp -a loki:/tmp/loki/. /mnt/containers/logging/loki-data/ +# Preserve the seed separately; never overlay WAL generations. +mv /mnt/containers/logging/loki-data \ + /mnt/containers/logging/loki-data-rescue-seed-2026-07-30 +docker cp -a loki:/tmp/loki /mnt/containers/logging/loki-data chown -R 10001:10001 /mnt/containers/logging/loki-data # Removes containers and the old network, but NOT the old named Grafana and @@ -98,6 +109,45 @@ In Grafana, the provisioned **Network / Mikrotik Loki Logs** dashboard should list existing `routerboard` values and show both pre-cutover and post-cutover entries. +## Telemt / Zabbix monitoring + +The Zabbix server is `10.0.0.11` internally. The DEPOT router at `10.0.0.1` +publishes TCP `185.108.4.158:10051` only for the three proxy-host egress +addresses, allowing Zabbix Agent 2 active checks without exposing the trapper +to the whole internet. + +Create the following Zabbix hosts with these exact technical **Host name** +values (visible names may be friendlier): + +| Client | Zabbix Host name | +|---|---| +| 00-xserver | `00-xserver-fedora-xxx` | +| 11-nevafud | `11-nevafud-fedora` | +| 19-nutrilent | `19-nutrilent-docker` | + +Import `zabbix-template-telemt-active.yaml`, link **Telemt proxy by active +agent** to all three hosts, and ensure each host is in the `Linux servers` +group. Do not add a passive-agent interface: these hosts are behind NAT and +send active checks to `185.108.4.158`. + +The matching host-side collector is in `zabbix-agent/`. It enters only the +Telemt container network namespace for the control API, discards links and +secrets, caches the active-IP list and Prometheus values, and exposes +read-only Agent 2 keys. The active-IP list retains one day of history; the +numeric count and other counters retain 31 days. + +After redeploying Grafana with a valid `ZABBIX_API_TOKEN`, verify: + +```bash +docker exec grafana grafana cli plugins ls +curl -fsS http://127.0.0.1:3000/api/health +``` + +The provisioned **Network / Telegram proxy fleet** dashboard selects one of +the three exact host names above and mirrors the Telemt section of the +client-02 dashboard: current IP count, active/total connections, IP list, +traffic, and uptime. + ## Rollback Remove the new Portainer stack, then recreate the old stack with its Loki data diff --git a/logging/alloy-config.alloy b/logging/alloy-config.alloy index b5817d2..4d607ca 100644 --- a/logging/alloy-config.alloy +++ b/logging/alloy-config.alloy @@ -1,7 +1,9 @@ -// Direct replacement for the old syslog-ng -> Promtail -> Loki path. +// Replaces Promtail in the old syslog-ng -> Promtail -> Loki path. // -// Existing RouterOS senders target 185.108.4.158:514/udp and use classic -// BSD/RFC3164 framing. TCP/601 is retained for RFC5424-capable senders. +// DEPOT's fleet is not uniformly parseable as RFC3164: some RouterOS senders +// emit raw messages without a PRI/header. syslog-ng therefore keeps listening +// on the public stack ports and normalizes every message to RFC5424 before +// forwarding it here on the private stack network. loki.relabel "syslog" { forward_to = [] @@ -28,29 +30,14 @@ loki.relabel "syslog" { } } -loki.source.syslog "network_devices_udp" { +loki.source.syslog "normalized_syslog" { listener { - address = "0.0.0.0:514" - protocol = "udp" - syslog_format = "rfc3164" - labels = { - job = "syslog", - transport = "udp", - } - } - - relabel_rules = loki.relabel.syslog.rules - forward_to = [loki.write.default.receiver] -} - -loki.source.syslog "network_devices_tcp" { - listener { - address = "0.0.0.0:601" + address = "0.0.0.0:1514" protocol = "tcp" syslog_format = "rfc5424" labels = { job = "syslog", - transport = "tcp", + transport = "syslog-ng", } } diff --git a/logging/docker-compose.yaml b/logging/docker-compose.yaml index 681118e..2441625 100644 --- a/logging/docker-compose.yaml +++ b/logging/docker-compose.yaml @@ -16,8 +16,29 @@ services: networks: - logging-nw + syslog-ng: + # DEPOT's RouterOS fleet includes senders that omit the PRI/header Alloy's + # strict parser requires. Keep the proven normalizer, pinned by digest, + # and relay canonical RFC5424 to Alloy. + image: balabit/syslog-ng@sha256:ce85e1af17daf888c72267b19f7f104bf329044257564b784ff8c00ce4fc01ef + container_name: syslog-ng + command: -edv + restart: unless-stopped + depends_on: + - alloy + volumes: + - /mnt/containers/logging/syslog-ng-config/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf:ro,Z + ports: + # Existing fleet target: 185.108.4.158:514/udp. + - "514:514/udp" + # Preserve the old stack's TCP syslog endpoint. + - "601:601/tcp" + networks: + - logging-nw + alloy: - # Replaces the syslog-ng -> Promtail relay with one direct syslog receiver. + # Replaces Promtail. syslog-ng normalizes the mixed/raw fleet input into + # RFC5424 before sending it to Alloy on the private stack network. image: grafana/alloy:v1.16.1 container_name: alloy command: @@ -31,11 +52,6 @@ services: volumes: - /mnt/containers/logging/alloy-config/config.alloy:/etc/alloy/config.alloy:ro,Z - /mnt/containers/logging/alloy-data:/var/lib/alloy/data:Z - ports: - # Existing fleet target: 185.108.4.158:514/udp. - - "514:514/udp" - # Preserve the old stack's TCP syslog endpoint. - - "601:601/tcp" networks: - logging-nw @@ -48,6 +64,11 @@ services: environment: GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD} GF_USERS_ALLOW_SIGN_UP: "false" + # Install the same Zabbix datasource version used by client 02. + GF_PLUGINS_PREINSTALL_SYNC: alexanderzobnin-zabbix-app@6.5.0 + # Used only by the provisioned Zabbix datasource; set as a Portainer + # secret/environment value and never commit the token. + ZABBIX_API_TOKEN: ${ZABBIX_API_TOKEN} volumes: - /mnt/containers/logging/grafana-data:/var/lib/grafana:Z - /mnt/containers/logging/grafana-config/provisioning/datasources:/etc/grafana/provisioning/datasources:ro,Z diff --git a/logging/grafana-dashboards/telemt-proxy-fleet.json b/logging/grafana-dashboards/telemt-proxy-fleet.json new file mode 100644 index 0000000..b3352f9 --- /dev/null +++ b/logging/grafana-dashboards/telemt-proxy-fleet.json @@ -0,0 +1,836 @@ +{ + "uid": "telemt-proxy-fleet", + "title": "Telegram proxy fleet", + "tags": [ + "proxy", + "telemt", + "zabbix", + "fleet" + ], + "schemaVersion": 39, + "version": 1, + "editable": true, + "refresh": "30s", + "time": { + "from": "now-6h", + "to": "now" + }, + "timezone": "", + "templating": { + "list": [ + { + "current": { + "selected": true, + "text": "00-xserver-fedora-xxx", + "value": "00-xserver-fedora-xxx" + }, + "hide": 0, + "includeAll": false, + "label": "Proxy host", + "multi": false, + "name": "proxy_host", + "options": [ + { + "selected": true, + "text": "00-xserver-fedora-xxx", + "value": "00-xserver-fedora-xxx" + }, + { + "selected": false, + "text": "11-nevafud-fedora", + "value": "11-nevafud-fedora" + }, + { + "selected": false, + "text": "19-nutrilent-docker", + "value": "19-nutrilent-docker" + } + ], + "query": "00-xserver-fedora-xxx,11-nevafud-fedora,19-nutrilent-docker", + "queryValue": "", + "skipUrlSync": false, + "type": "custom" + } + ] + }, + "annotations": { + "list": [] + }, + "panels": [ + { + "type": "row", + "title": "Telegram MTProto (:1080) — Telemt", + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "panels": [] + }, + { + "id": 20, + "type": "stat", + "title": "Уникальных пользователей сейчас (IP)", + "description": "Текущее число уникальных активных клиентских IP для Telemt user secret `tekhnohim`. Это наиболее близкая доступная метрика к числу подключённых пользователей: несколько клиентов за одним NAT считаются как один IP, а один пользователь с нескольких внешних IP — как несколько.", + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "gridPos": { + "h": 5, + "w": 10, + "x": 0, + "y": 1 + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "background", + "graphMode": "area", + "textMode": "value_and_name", + "justifyMode": "center", + "wideLayout": true + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "unit": "short", + "decimals": 0, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "dark-gray", + "value": null + }, + { + "color": "green", + "value": 1 + } + ] + } + }, + "overrides": [] + }, + "targets": [ + { + "application": { + "filter": "" + }, + "countTriggers": true, + "countTriggersBy": "", + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "evaltype": "0", + "functions": [], + "group": { + "filter": "Linux servers" + }, + "host": { + "filter": "$proxy_host" + }, + "item": { + "filter": "/^Telemt: Active IP address count$/" + }, + "itemTag": { + "filter": "" + }, + "macro": { + "filter": "" + }, + "minSeverity": 3, + "mode": 0, + "options": { + "count": false, + "disableDataAlignment": false, + "showDisabledItems": false, + "skipEmptyValues": false, + "useTrends": "default", + "useZabbixValueMapping": false + }, + "proxy": { + "filter": "" + }, + "queryType": "0", + "refId": "A", + "resultFormat": "time_series", + "schema": 12, + "table": { + "skipEmptyValues": false + }, + "tags": { + "filter": "" + }, + "textFilter": "", + "trigger": { + "filter": "" + } + } + ] + }, + { + "id": 21, + "type": "stat", + "title": "Активных соединений", + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "gridPos": { + "h": 5, + "w": 7, + "x": 10, + "y": 1 + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "value", + "graphMode": "area", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "blue" + }, + "unit": "short", + "decimals": 0 + }, + "overrides": [] + }, + "targets": [ + { + "application": { + "filter": "" + }, + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "functions": [], + "group": { + "filter": "Linux servers" + }, + "host": { + "filter": "$proxy_host" + }, + "item": { + "filter": "/^Telemt: Active connections$/" + }, + "itemTag": { + "filter": "" + }, + "mode": 0, + "options": { + "count": false, + "disableDataAlignment": false, + "showDisabledItems": false, + "skipEmptyValues": false, + "useTrends": "default", + "useZabbixValueMapping": false + }, + "queryType": "0", + "refId": "A", + "resultFormat": "time_series", + "schema": 12 + } + ] + }, + { + "id": 22, + "type": "stat", + "title": "Всего подключений", + "description": "Накопительный счётчик подключений user secret `tekhnohim` с момента запуска Telemt.", + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "gridPos": { + "h": 5, + "w": 7, + "x": 17, + "y": 1 + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "value", + "graphMode": "area", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "purple" + }, + "unit": "short", + "decimals": 0 + }, + "overrides": [] + }, + "targets": [ + { + "application": { + "filter": "" + }, + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "functions": [], + "group": { + "filter": "Linux servers" + }, + "host": { + "filter": "$proxy_host" + }, + "item": { + "filter": "/^Telemt: Total connections$/" + }, + "itemTag": { + "filter": "" + }, + "mode": 0, + "options": { + "count": false, + "disableDataAlignment": false, + "showDisabledItems": false, + "skipEmptyValues": false, + "useTrends": "default", + "useZabbixValueMapping": false + }, + "queryType": "0", + "refId": "A", + "resultFormat": "time_series", + "schema": 12 + } + ] + }, + { + "id": 28, + "type": "table", + "title": "Текущие активные IP-адреса Telemt", + "description": "Последний снимок активных IP-адресов из Zabbix item telemt.active_ips.list. Обновляется каждые 30 секунд.", + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "gridPos": { + "h": 6, + "w": 24, + "x": 0, + "y": 6 + }, + "fieldConfig": { + "defaults": { + "custom": { + "align": "left", + "filterable": false + } + }, + "overrides": [] + }, + "options": { + "cellHeight": "auto", + "footer": { + "countRows": false, + "fields": "", + "reducer": [ + "sum" + ], + "show": false + }, + "showHeader": true, + "sortBy": [] + }, + "transformations": [ + { + "id": "organize", + "options": { + "excludeByName": { + "Host": true, + "Item": true, + "Key": true + }, + "renameByName": { + "Last value": "Активные подключения (пользователь | IP)" + } + } + } + ], + "targets": [ + { + "application": { + "filter": "" + }, + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "functions": [], + "group": { + "filter": "Linux servers" + }, + "host": { + "filter": "$proxy_host" + }, + "item": { + "filter": "/^Telemt: Active IP addresses$/" + }, + "itemTag": { + "filter": "" + }, + "mode": 0, + "options": { + "count": false, + "disableDataAlignment": false, + "showDisabledItems": false, + "skipEmptyValues": true, + "useTrends": "default", + "useZabbixValueMapping": false + }, + "queryType": "2", + "refId": "A", + "resultFormat": "table", + "schema": 12, + "textFilter": "", + "useCaptureGroups": false + } + ] + }, + { + "id": 23, + "type": "timeseries", + "title": "Пользователи и соединения", + "description": "Текущее число уникальных активных IP и параллельных Telemt-соединений.", + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 12 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "stepAfter", + "lineWidth": 2, + "pointSize": 3, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + } + }, + "unit": "short", + "decimals": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "application": { + "filter": "" + }, + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "functions": [], + "group": { + "filter": "Linux servers" + }, + "host": { + "filter": "$proxy_host" + }, + "item": { + "filter": "/^Telemt: (Active IP address count|Active connections)$/" + }, + "itemTag": { + "filter": "" + }, + "mode": 0, + "options": { + "count": false, + "disableDataAlignment": false, + "showDisabledItems": false, + "skipEmptyValues": false, + "useTrends": "default", + "useZabbixValueMapping": false + }, + "queryType": "0", + "refId": "A", + "resultFormat": "time_series", + "schema": 12 + } + ] + }, + { + "id": 24, + "type": "timeseries", + "title": "Трафик Telemt (накопительно)", + "description": "Накопительные байтовые счётчики Telemt по user secret `tekhnohim`; сбрасываются при перезапуске сервиса.", + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 12 + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "group": "A", + "mode": "none" + } + }, + "unit": "bytes" + }, + "overrides": [] + }, + "options": { + "legend": { + "calcs": [ + "lastNotNull", + "max" + ], + "displayMode": "table", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "application": { + "filter": "" + }, + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "functions": [], + "group": { + "filter": "Linux servers" + }, + "host": { + "filter": "$proxy_host" + }, + "item": { + "filter": "/^Telemt: Bytes (received from|sent to) clients$/" + }, + "itemTag": { + "filter": "" + }, + "mode": 0, + "options": { + "count": false, + "disableDataAlignment": false, + "showDisabledItems": false, + "skipEmptyValues": false, + "useTrends": "default", + "useZabbixValueMapping": false + }, + "queryType": "0", + "refId": "A", + "resultFormat": "time_series", + "schema": 12 + } + ] + }, + { + "id": 25, + "type": "stat", + "title": "Аптайм Telemt", + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "gridPos": { + "h": 4, + "w": 8, + "x": 0, + "y": 20 + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "value", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "green" + }, + "unit": "s" + }, + "overrides": [] + }, + "targets": [ + { + "application": { + "filter": "" + }, + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "functions": [], + "group": { + "filter": "Linux servers" + }, + "host": { + "filter": "$proxy_host" + }, + "item": { + "filter": "/^Telemt: Uptime$/" + }, + "itemTag": { + "filter": "" + }, + "mode": 0, + "options": { + "count": false, + "disableDataAlignment": false, + "showDisabledItems": false, + "skipEmptyValues": false, + "useTrends": "default", + "useZabbixValueMapping": false + }, + "queryType": "0", + "refId": "A", + "resultFormat": "time_series", + "schema": 12 + } + ] + }, + { + "id": 26, + "type": "stat", + "title": "Получено от клиентов", + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "gridPos": { + "h": 4, + "w": 8, + "x": 8, + "y": 20 + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "value", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "semi-dark-blue" + }, + "unit": "bytes" + }, + "overrides": [] + }, + "targets": [ + { + "application": { + "filter": "" + }, + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "functions": [], + "group": { + "filter": "Linux servers" + }, + "host": { + "filter": "$proxy_host" + }, + "item": { + "filter": "/^Telemt: Bytes received from clients$/" + }, + "itemTag": { + "filter": "" + }, + "mode": 0, + "options": { + "count": false, + "disableDataAlignment": false, + "showDisabledItems": false, + "skipEmptyValues": false, + "useTrends": "default", + "useZabbixValueMapping": false + }, + "queryType": "0", + "refId": "A", + "resultFormat": "time_series", + "schema": 12 + } + ] + }, + { + "id": 27, + "type": "stat", + "title": "Отправлено клиентам", + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "gridPos": { + "h": 4, + "w": 8, + "x": 16, + "y": 20 + }, + "options": { + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "colorMode": "value", + "graphMode": "none", + "textMode": "value_and_name", + "justifyMode": "center" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "fixed", + "fixedColor": "semi-dark-purple" + }, + "unit": "bytes" + }, + "overrides": [] + }, + "targets": [ + { + "application": { + "filter": "" + }, + "datasource": { + "type": "alexanderzobnin-zabbix-datasource", + "uid": "zabbix-depot" + }, + "functions": [], + "group": { + "filter": "Linux servers" + }, + "host": { + "filter": "$proxy_host" + }, + "item": { + "filter": "/^Telemt: Bytes sent to clients$/" + }, + "itemTag": { + "filter": "" + }, + "mode": 0, + "options": { + "count": false, + "disableDataAlignment": false, + "showDisabledItems": false, + "skipEmptyValues": false, + "useTrends": "default", + "useZabbixValueMapping": false + }, + "queryType": "0", + "refId": "A", + "resultFormat": "time_series", + "schema": 12 + } + ] + } + ] +} diff --git a/logging/grafana-provisioning/datasources/zabbix.yaml b/logging/grafana-provisioning/datasources/zabbix.yaml new file mode 100644 index 0000000..b61d88c --- /dev/null +++ b/logging/grafana-provisioning/datasources/zabbix.yaml @@ -0,0 +1,22 @@ +apiVersion: 1 + +datasources: + - name: Zabbix + uid: zabbix-depot + type: alexanderzobnin-zabbix-datasource + access: proxy + url: http://10.0.0.11/api_jsonrpc.php + editable: false + jsonData: + authType: token + cacheTTL: 1h + dbConnectionEnable: false + disableDataAlignment: false + disableReadOnlyUsersAck: true + queryTimeout: 60 + timeout: 30 + trends: true + trendsFrom: 7d + trendsRange: 4d + secureJsonData: + apiToken: ${ZABBIX_API_TOKEN} diff --git a/logging/syslog-ng.conf b/logging/syslog-ng.conf new file mode 100644 index 0000000..e383708 --- /dev/null +++ b/logging/syslog-ng.conf @@ -0,0 +1,30 @@ +@version: 3.29 +@include "scl.conf" + +source s_local { + internal(); +}; + +# default-network-drivers accepts the mixed/raw formats used by the DEPOT +# fleet on the standard UDP/TCP syslog listeners. +source s_network { + default-network-drivers(); +}; + +# Alloy's syslog receiver accepts one explicit format per listener. Normalize +# every accepted input to RFC5424 before relaying it over the private network. +template t_alloy_rfc5424 { + template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} - ${MSG}\n"); +}; + +destination d_alloy { + syslog("alloy" transport("tcp") port("1514") + template(t_alloy_rfc5424) + ); +}; + +log { + source(s_local); + source(s_network); + destination(d_alloy); +}; diff --git a/logging/zabbix-agent/telemt.conf b/logging/zabbix-agent/telemt.conf new file mode 100644 index 0000000..e9f31a5 --- /dev/null +++ b/logging/zabbix-agent/telemt.conf @@ -0,0 +1,8 @@ +# Cached by zabbix-telemt-collector.service; contains no proxy links or secrets. +UserParameter=telemt.active_ips.list,cat /var/lib/zabbix/telemt-active-ips.txt +UserParameter=telemt.active_ips.count,/usr/local/libexec/zabbix-telemt-value active_ips_count +UserParameter=telemt.connections.current,/usr/local/libexec/zabbix-telemt-value connections_current +UserParameter=telemt.connections.total,/usr/local/libexec/zabbix-telemt-value connections_total +UserParameter=telemt.bytes.received,/usr/local/libexec/zabbix-telemt-value bytes_received +UserParameter=telemt.bytes.sent,/usr/local/libexec/zabbix-telemt-value bytes_sent +UserParameter=telemt.uptime,/usr/local/libexec/zabbix-telemt-value uptime diff --git a/logging/zabbix-agent/zabbix-telemt-collector b/logging/zabbix-agent/zabbix-telemt-collector new file mode 100755 index 0000000..9d2bb75 --- /dev/null +++ b/logging/zabbix-agent/zabbix-telemt-collector @@ -0,0 +1,65 @@ +#!/usr/bin/bash +set -euo pipefail + +container_name="ts-lan-telemt" +output_dir="/var/lib/zabbix" +json_target="${output_dir}/telemt-active-ips.json" +text_target="${output_dir}/telemt-active-ips.txt" +metrics_target="${output_dir}/telemt-metrics.prom" + +install -d -o root -g zabbix -m 0750 "${output_dir}" + +container_pid="$(docker inspect --format '{{.State.Pid}}' "${container_name}")" +if [[ -z "${container_pid}" || "${container_pid}" == "0" ]]; then + exit 1 +fi + +host_ip="$(ip -4 route get 1.1.1.1 | awk '{for (i=1; i<=NF; i++) if ($i == "src") {print $(i+1); exit}}')" +if [[ -z "${host_ip}" ]]; then + exit 1 +fi + +json_tmp="$(mktemp "${output_dir}/.telemt-active-ips.json.XXXXXX")" +text_tmp="$(mktemp "${output_dir}/.telemt-active-ips.txt.XXXXXX")" +metrics_tmp="$(mktemp "${output_dir}/.telemt-metrics.prom.XXXXXX")" +trap 'rm -f "${json_tmp}" "${text_tmp}" "${metrics_tmp}"' EXIT + +# The API response also contains proxy links and secrets. Filter it in-memory +# and persist only the username and current active IP address list. +nsenter -t "${container_pid}" -n \ + curl --fail --silent --show-error --max-time 5 \ + http://127.0.0.1:9091/v1/users | + jq --compact-output '{ + users: [ + .data[] + | { + username, + active_unique_ips: (.active_unique_ips_list // []) + } + ] + }' > "${json_tmp}" + +{ + active_count="$(jq '[.users[].active_unique_ips[]] | unique | length' "${json_tmp}")" + printf 'Active IP addresses: %s\n' "${active_count}" + jq --raw-output ' + [ + .users[] as $user + | $user.active_unique_ips[] + | "\($user.username) | \(.)" + ] + | if length == 0 then ["No active IP addresses"] else . end + | .[] + ' "${json_tmp}" +} > "${text_tmp}" + +# Cache Prometheus output locally. Zabbix active checks read the cache, so the +# central server does not need a route to each private Docker host. +curl --fail --silent --show-error --max-time 5 \ + "http://${host_ip}:9092/metrics" > "${metrics_tmp}" + +chown root:zabbix "${json_tmp}" "${text_tmp}" "${metrics_tmp}" +chmod 0640 "${json_tmp}" "${text_tmp}" "${metrics_tmp}" +mv -f "${json_tmp}" "${json_target}" +mv -f "${text_tmp}" "${text_target}" +mv -f "${metrics_tmp}" "${metrics_target}" diff --git a/logging/zabbix-agent/zabbix-telemt-collector.service b/logging/zabbix-agent/zabbix-telemt-collector.service new file mode 100644 index 0000000..297603e --- /dev/null +++ b/logging/zabbix-agent/zabbix-telemt-collector.service @@ -0,0 +1,12 @@ +[Unit] +Description=Collect sanitized Telemt data for Zabbix +After=docker.service +Requires=docker.service + +[Service] +Type=oneshot +ExecStart=/usr/local/libexec/zabbix-telemt-collector +User=root +Group=root +PrivateTmp=true +NoNewPrivileges=true diff --git a/logging/zabbix-agent/zabbix-telemt-collector.timer b/logging/zabbix-agent/zabbix-telemt-collector.timer new file mode 100644 index 0000000..81252db --- /dev/null +++ b/logging/zabbix-agent/zabbix-telemt-collector.timer @@ -0,0 +1,12 @@ +[Unit] +Description=Refresh sanitized Telemt data for Zabbix + +[Timer] +OnBootSec=30s +OnUnitActiveSec=30s +AccuracySec=2s +Unit=zabbix-telemt-collector.service +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/logging/zabbix-agent/zabbix-telemt-value b/logging/zabbix-agent/zabbix-telemt-value new file mode 100755 index 0000000..d9c43fc --- /dev/null +++ b/logging/zabbix-agent/zabbix-telemt-value @@ -0,0 +1,30 @@ +#!/usr/bin/bash +set -euo pipefail + +json_file="/var/lib/zabbix/telemt-active-ips.json" +metrics_file="/var/lib/zabbix/telemt-metrics.prom" + +case "${1:-}" in + active_ips_count) + jq '[.users[].active_unique_ips[]] | unique | length' "${json_file}" + ;; + connections_current) + awk '$1 ~ /^telemt_user_connections_current\{/ {sum += $2} END {print sum + 0}' "${metrics_file}" + ;; + connections_total) + awk '$1 ~ /^telemt_user_connections_total\{/ {sum += $2} END {print sum + 0}' "${metrics_file}" + ;; + bytes_received) + awk '$1 ~ /^telemt_user_octets_from_client\{/ {sum += $2} END {printf "%.0f\n", sum + 0}' "${metrics_file}" + ;; + bytes_sent) + awk '$1 ~ /^telemt_user_octets_to_client\{/ {sum += $2} END {printf "%.0f\n", sum + 0}' "${metrics_file}" + ;; + uptime) + awk '$1 == "telemt_uptime_seconds" {printf "%.0f\n", $2; found=1} END {if (!found) exit 1}' "${metrics_file}" + ;; + *) + printf 'unsupported Telemt value: %s\n' "${1:-}" >&2 + exit 2 + ;; +esac diff --git a/logging/zabbix-template-telemt-active.yaml b/logging/zabbix-template-telemt-active.yaml new file mode 100644 index 0000000..a569969 --- /dev/null +++ b/logging/zabbix-template-telemt-active.yaml @@ -0,0 +1,94 @@ +zabbix_export: + version: '7.0' + template_groups: + - uuid: a571c0d144b14fd4a87a9d9b2aa9fcd6 + name: Templates/Applications + templates: + - uuid: 1d617f451f8640089f9ac33d62865e72 + template: 'Telemt proxy by active agent' + name: 'Telemt proxy by active agent' + description: | + Telemt metrics cached locally by zabbix-telemt-collector.timer. + Designed for hosts behind NAT; every item is a Zabbix agent active check. + groups: + - name: Templates/Applications + items: + - uuid: e383ca45b2c047519f30665999cfc02e + name: 'Telemt: Active IP address count' + type: ZABBIX_ACTIVE + key: telemt.active_ips.count + delay: 30s + history: 31d + trends: 365d + value_type: UNSIGNED + tags: + - tag: component + value: telemt + - uuid: 5a29849443d5473884246d049a2dd79f + name: 'Telemt: Active IP addresses' + type: ZABBIX_ACTIVE + key: telemt.active_ips.list + delay: 30s + history: 1d + value_type: TEXT + tags: + - tag: component + value: telemt + - uuid: 465288678a084f8c84f56c2cffc0dcda + name: 'Telemt: Active connections' + type: ZABBIX_ACTIVE + key: telemt.connections.current + delay: 30s + history: 31d + trends: 365d + value_type: UNSIGNED + tags: + - tag: component + value: telemt + - uuid: dddf99d04abb459f8890eff46b948c4b + name: 'Telemt: Total connections' + type: ZABBIX_ACTIVE + key: telemt.connections.total + delay: 30s + history: 31d + trends: 365d + value_type: UNSIGNED + tags: + - tag: component + value: telemt + - uuid: 03e3a3671d744efaa65d4328d8d4035f + name: 'Telemt: Bytes received from clients' + type: ZABBIX_ACTIVE + key: telemt.bytes.received + delay: 30s + history: 31d + trends: 365d + value_type: UNSIGNED + units: B + tags: + - tag: component + value: telemt + - uuid: 97679bf1d06b47b0acf8f33f1253d114 + name: 'Telemt: Bytes sent to clients' + type: ZABBIX_ACTIVE + key: telemt.bytes.sent + delay: 30s + history: 31d + trends: 365d + value_type: UNSIGNED + units: B + tags: + - tag: component + value: telemt + - uuid: b554b83a21ca46638c753e803870661f + name: 'Telemt: Uptime' + type: ZABBIX_ACTIVE + key: telemt.uptime + delay: 30s + history: 31d + trends: 365d + value_type: UNSIGNED + units: uptime + tags: + - tag: component + value: telemt