depot/docker-infrastructure
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

DEPOT logging: add Telemt fleet monitoring via Zabbix

Browse Source 
Install and persist Grafana Zabbix plugin 6.5.0, provision the internal Zabbix datasource, and add a fleet dashboard modeled on client 02 Telemt panels.

Add an active-agent Zabbix 7 template plus a 30-second sanitized collector for active IP count/list, connections, traffic, and uptime. Active checks avoid inbound polling of client-private Docker hosts.

The exact Zabbix host names and required ZABBIX_API_TOKEN Portainer variable are documented. Compose validation passed on 10.0.0.6; the live plugin registered successfully under Grafana 13.1.0.
This commit is contained in:
pipistrello
2026-07-31 11:55:55 +03:00
parent 961f0112de
commit 1fb6899b1f
12 changed files with 1207 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
logging/README.md
+63 -13
View File
@@ -5,18 +5,24 @@ part of the manual `mktxp-stack`:
- `loki` stores syslog in the persistent host path
`/mnt/containers/logging/loki-data`;
- `alloy` replaces the `syslog-ng` -> `promtail` relay;
- `grafana` provides the existing MikroTik Loki dashboard.
- `syslog-ng` accepts and normalizes the fleet's mixed/raw syslog formats;
- `alloy` replaces Promtail and sends normalized RFC5424 to Loki;
- `grafana` provides the MikroTik Loki dashboard and a Telemt fleet dashboard
backed by the internal DEPOT Zabbix server.
The retired stack's `mktxp`, Prometheus, Promtail, and syslog-ng services are
not part of this stack.
The retired stack's `mktxp`, Prometheus, and Promtail services are not part of
this stack. syslog-ng remains because real DEPOT RouterOS traffic includes raw
messages that Alloy's strict RFC3164 parser rejects.
## Portainer settings
- Repository: `https://git.it-depot.ru/depot/docker-infrastructure`
- Branch: `master`
- Compose path: `logging/docker-compose.yaml`
- Required environment variable: `GRAFANA_ADMIN_PASSWORD`
- Required environment variables:
- `GRAFANA_ADMIN_PASSWORD`
- `ZABBIX_API_TOKEN` — token for a read-only Zabbix API user; it is passed
into Grafana's provisioned datasource and must not be committed.
Do not deploy the new stack before completing the cutover below: the current
containers still own names `loki` and `grafana` and host ports 3000, 3100, 514,
@@ -33,12 +39,14 @@ Before deployment, the repository files must be copied to:
├── alloy-data/
├── grafana-config/
│ ├── dashboards/mikrotik-loki-logs.json
│ ├── dashboards/telemt-proxy-fleet.json
│ └── provisioning/
│ ├── dashboards/dashboards.yaml
│ └── datasources/loki.yaml
│ └── datasources/{loki,zabbix}.yaml
├── grafana-data/
├── loki-config/config.yaml
└── loki-data/
├── loki-data/
└── syslog-ng-config/syslog-ng.conf
```
SELinux is enforcing. The compose mounts use `:Z`; do not remove those flags.
@@ -47,10 +55,11 @@ Loki `10001:10001`, Grafana `472:0`, and Alloy `0:0`.
## Cutover
The live Loki history is inside the old container at `/tmp/loki`. A stopped,
consistent rescue copy from 2026-07-30 is pre-seeded in the new host path, but
it must be refreshed from the stopped live container immediately before the
old stack is removed:
The live Loki history is inside the old container at `/tmp/loki`. Do not merge
a newer WAL over an older rescue WAL: the resulting non-contiguous segment
sequence makes Loki fail with `segments are not sequential`. Instead, move the
seed aside and copy the stopped container's complete data directory as one
generation:
```bash
cd /mnt/containers/mktxp/mktxp-stack
@@ -59,8 +68,10 @@ cd /mnt/containers/mktxp/mktxp-stack
curl -fsS http://127.0.0.1:3100/ready
docker compose stop
# The stopped Loki container still contains the newest WAL/chunks.
docker cp -a loki:/tmp/loki/. /mnt/containers/logging/loki-data/
# Preserve the seed separately; never overlay WAL generations.
mv /mnt/containers/logging/loki-data \
/mnt/containers/logging/loki-data-rescue-seed-2026-07-30
docker cp -a loki:/tmp/loki /mnt/containers/logging/loki-data
chown -R 10001:10001 /mnt/containers/logging/loki-data
# Removes containers and the old network, but NOT the old named Grafana and
@@ -98,6 +109,45 @@ In Grafana, the provisioned **Network / Mikrotik Loki Logs** dashboard should
list existing `routerboard` values and show both pre-cutover and post-cutover
entries.
## Telemt / Zabbix monitoring
The Zabbix server is `10.0.0.11` internally. The DEPOT router at `10.0.0.1`
publishes TCP `185.108.4.158:10051` only for the three proxy-host egress
addresses, allowing Zabbix Agent 2 active checks without exposing the trapper
to the whole internet.
Create the following Zabbix hosts with these exact technical **Host name**
values (visible names may be friendlier):
| Client | Zabbix Host name |
|---|---|
| 00-xserver | `00-xserver-fedora-xxx` |
| 11-nevafud | `11-nevafud-fedora` |
| 19-nutrilent | `19-nutrilent-docker` |
Import `zabbix-template-telemt-active.yaml`, link **Telemt proxy by active
agent** to all three hosts, and ensure each host is in the `Linux servers`
group. Do not add a passive-agent interface: these hosts are behind NAT and
send active checks to `185.108.4.158`.
The matching host-side collector is in `zabbix-agent/`. It enters only the
Telemt container network namespace for the control API, discards links and
secrets, caches the active-IP list and Prometheus values, and exposes
read-only Agent 2 keys. The active-IP list retains one day of history; the
numeric count and other counters retain 31 days.
After redeploying Grafana with a valid `ZABBIX_API_TOKEN`, verify:
```bash
docker exec grafana grafana cli plugins ls
curl -fsS http://127.0.0.1:3000/api/health
```
The provisioned **Network / Telegram proxy fleet** dashboard selects one of
the three exact host names above and mirrors the Telemt section of the
client-02 dashboard: current IP count, active/total connections, IP list,
traffic, and uptime.
## Rollback
Remove the new Portainer stack, then recreate the old stack with its Loki data
logging/alloy-config.alloy
+8 -21
View File
@@ -1,7 +1,9 @@
// Direct replacement for the old syslog-ng -> Promtail -> Loki path.
// Replaces Promtail in the old syslog-ng -> Promtail -> Loki path.
//
// Existing RouterOS senders target 185.108.4.158:514/udp and use classic
// BSD/RFC3164 framing. TCP/601 is retained for RFC5424-capable senders.
// DEPOT's fleet is not uniformly parseable as RFC3164: some RouterOS senders
// emit raw messages without a PRI/header. syslog-ng therefore keeps listening
// on the public stack ports and normalizes every message to RFC5424 before
// forwarding it here on the private stack network.
loki.relabel "syslog" {
forward_to = []
@@ -28,29 +30,14 @@ loki.relabel "syslog" {
}
}
loki.source.syslog "network_devices_udp" {
loki.source.syslog "normalized_syslog" {
listener {
address = "0.0.0.0:514"
protocol = "udp"
syslog_format = "rfc3164"
labels = {
job = "syslog",
transport = "udp",
}
}
relabel_rules = loki.relabel.syslog.rules
forward_to = [loki.write.default.receiver]
}
loki.source.syslog "network_devices_tcp" {
listener {
address = "0.0.0.0:601"
address = "0.0.0.0:1514"
protocol = "tcp"
syslog_format = "rfc5424"
labels = {
job = "syslog",
transport = "tcp",
transport = "syslog-ng",
}
}
logging/docker-compose.yaml
+27 -6
View File
@@ -16,8 +16,29 @@ services:
networks:
- logging-nw
syslog-ng:
# DEPOT's RouterOS fleet includes senders that omit the PRI/header Alloy's
# strict parser requires. Keep the proven normalizer, pinned by digest,
# and relay canonical RFC5424 to Alloy.
image: balabit/syslog-ng@sha256:ce85e1af17daf888c72267b19f7f104bf329044257564b784ff8c00ce4fc01ef
container_name: syslog-ng
command: -edv
restart: unless-stopped
depends_on:
- alloy
volumes:
- /mnt/containers/logging/syslog-ng-config/syslog-ng.conf:/etc/syslog-ng/syslog-ng.conf:ro,Z
ports:
# Existing fleet target: 185.108.4.158:514/udp.
- "514:514/udp"
# Preserve the old stack's TCP syslog endpoint.
- "601:601/tcp"
networks:
- logging-nw
alloy:
# Replaces the syslog-ng -> Promtail relay with one direct syslog receiver.
# Replaces Promtail. syslog-ng normalizes the mixed/raw fleet input into
# RFC5424 before sending it to Alloy on the private stack network.
image: grafana/alloy:v1.16.1
container_name: alloy
command:
@@ -31,11 +52,6 @@ services:
volumes:
- /mnt/containers/logging/alloy-config/config.alloy:/etc/alloy/config.alloy:ro,Z
- /mnt/containers/logging/alloy-data:/var/lib/alloy/data:Z
ports:
# Existing fleet target: 185.108.4.158:514/udp.
- "514:514/udp"
# Preserve the old stack's TCP syslog endpoint.
- "601:601/tcp"
networks:
- logging-nw
@@ -48,6 +64,11 @@ services:
environment:
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
GF_USERS_ALLOW_SIGN_UP: "false"
# Install the same Zabbix datasource version used by client 02.
GF_PLUGINS_PREINSTALL_SYNC: alexanderzobnin-zabbix-app@6.5.0
# Used only by the provisioned Zabbix datasource; set as a Portainer
# secret/environment value and never commit the token.
ZABBIX_API_TOKEN: ${ZABBIX_API_TOKEN}
volumes:
- /mnt/containers/logging/grafana-data:/var/lib/grafana:Z
- /mnt/containers/logging/grafana-config/provisioning/datasources:/etc/grafana/provisioning/datasources:ro,Z
logging/grafana-dashboards/telemt-proxy-fleet.json
+836
View File
@@ -0,0 +1,836 @@
{
"uid": "telemt-proxy-fleet",
"title": "Telegram proxy fleet",
"tags": [
"proxy",
"telemt",
"zabbix",
"fleet"
],
"schemaVersion": 39,
"version": 1,
"editable": true,
"refresh": "30s",
"time": {
"from": "now-6h",
"to": "now"
},
"timezone": "",
"templating": {
"list": [
{
"current": {
"selected": true,
"text": "00-xserver-fedora-xxx",
"value": "00-xserver-fedora-xxx"
},
"hide": 0,
"includeAll": false,
"label": "Proxy host",
"multi": false,
"name": "proxy_host",
"options": [
{
"selected": true,
"text": "00-xserver-fedora-xxx",
"value": "00-xserver-fedora-xxx"
},
{
"selected": false,
"text": "11-nevafud-fedora",
"value": "11-nevafud-fedora"
},
{
"selected": false,
"text": "19-nutrilent-docker",
"value": "19-nutrilent-docker"
}
],
"query": "00-xserver-fedora-xxx,11-nevafud-fedora,19-nutrilent-docker",
"queryValue": "",
"skipUrlSync": false,
"type": "custom"
}
]
},
"annotations": {
"list": []
},
"panels": [
{
"type": "row",
"title": "Telegram MTProto (:1080) — Telemt",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"panels": []
},
{
"id": 20,
"type": "stat",
"title": "Уникальных пользователей сейчас (IP)",
"description": "Текущее число уникальных активных клиентских IP для Telemt user secret `tekhnohim`. Это наиболее близкая доступная метрика к числу подключённых пользователей: несколько клиентов за одним NAT считаются как один IP, а один пользователь с нескольких внешних IP — как несколько.",
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"gridPos": {
"h": 5,
"w": 10,
"x": 0,
"y": 1
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "background",
"graphMode": "area",
"textMode": "value_and_name",
"justifyMode": "center",
"wideLayout": true
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"unit": "short",
"decimals": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "dark-gray",
"value": null
},
{
"color": "green",
"value": 1
}
]
}
},
"overrides": []
},
"targets": [
{
"application": {
"filter": ""
},
"countTriggers": true,
"countTriggersBy": "",
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"evaltype": "0",
"functions": [],
"group": {
"filter": "Linux servers"
},
"host": {
"filter": "$proxy_host"
},
"item": {
"filter": "/^Telemt: Active IP address count$/"
},
"itemTag": {
"filter": ""
},
"macro": {
"filter": ""
},
"minSeverity": 3,
"mode": 0,
"options": {
"count": false,
"disableDataAlignment": false,
"showDisabledItems": false,
"skipEmptyValues": false,
"useTrends": "default",
"useZabbixValueMapping": false
},
"proxy": {
"filter": ""
},
"queryType": "0",
"refId": "A",
"resultFormat": "time_series",
"schema": 12,
"table": {
"skipEmptyValues": false
},
"tags": {
"filter": ""
},
"textFilter": "",
"trigger": {
"filter": ""
}
}
]
},
{
"id": 21,
"type": "stat",
"title": "Активных соединений",
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"gridPos": {
"h": 5,
"w": 7,
"x": 10,
"y": 1
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"textMode": "value_and_name",
"justifyMode": "center"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "fixed",
"fixedColor": "blue"
},
"unit": "short",
"decimals": 0
},
"overrides": []
},
"targets": [
{
"application": {
"filter": ""
},
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"functions": [],
"group": {
"filter": "Linux servers"
},
"host": {
"filter": "$proxy_host"
},
"item": {
"filter": "/^Telemt: Active connections$/"
},
"itemTag": {
"filter": ""
},
"mode": 0,
"options": {
"count": false,
"disableDataAlignment": false,
"showDisabledItems": false,
"skipEmptyValues": false,
"useTrends": "default",
"useZabbixValueMapping": false
},
"queryType": "0",
"refId": "A",
"resultFormat": "time_series",
"schema": 12
}
]
},
{
"id": 22,
"type": "stat",
"title": "Всего подключений",
"description": "Накопительный счётчик подключений user secret `tekhnohim` с момента запуска Telemt.",
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"gridPos": {
"h": 5,
"w": 7,
"x": 17,
"y": 1
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "area",
"textMode": "value_and_name",
"justifyMode": "center"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "fixed",
"fixedColor": "purple"
},
"unit": "short",
"decimals": 0
},
"overrides": []
},
"targets": [
{
"application": {
"filter": ""
},
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"functions": [],
"group": {
"filter": "Linux servers"
},
"host": {
"filter": "$proxy_host"
},
"item": {
"filter": "/^Telemt: Total connections$/"
},
"itemTag": {
"filter": ""
},
"mode": 0,
"options": {
"count": false,
"disableDataAlignment": false,
"showDisabledItems": false,
"skipEmptyValues": false,
"useTrends": "default",
"useZabbixValueMapping": false
},
"queryType": "0",
"refId": "A",
"resultFormat": "time_series",
"schema": 12
}
]
},
{
"id": 28,
"type": "table",
"title": "Текущие активные IP-адреса Telemt",
"description": "Последний снимок активных IP-адресов из Zabbix item telemt.active_ips.list. Обновляется каждые 30 секунд.",
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"gridPos": {
"h": 6,
"w": 24,
"x": 0,
"y": 6
},
"fieldConfig": {
"defaults": {
"custom": {
"align": "left",
"filterable": false
}
},
"overrides": []
},
"options": {
"cellHeight": "auto",
"footer": {
"countRows": false,
"fields": "",
"reducer": [
"sum"
],
"show": false
},
"showHeader": true,
"sortBy": []
},
"transformations": [
{
"id": "organize",
"options": {
"excludeByName": {
"Host": true,
"Item": true,
"Key": true
},
"renameByName": {
"Last value": "Активные подключения (пользователь | IP)"
}
}
}
],
"targets": [
{
"application": {
"filter": ""
},
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"functions": [],
"group": {
"filter": "Linux servers"
},
"host": {
"filter": "$proxy_host"
},
"item": {
"filter": "/^Telemt: Active IP addresses$/"
},
"itemTag": {
"filter": ""
},
"mode": 0,
"options": {
"count": false,
"disableDataAlignment": false,
"showDisabledItems": false,
"skipEmptyValues": true,
"useTrends": "default",
"useZabbixValueMapping": false
},
"queryType": "2",
"refId": "A",
"resultFormat": "table",
"schema": 12,
"textFilter": "",
"useCaptureGroups": false
}
]
},
{
"id": 23,
"type": "timeseries",
"title": "Пользователи и соединения",
"description": "Текущее число уникальных активных IP и параллельных Telemt-соединений.",
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 12
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"fillOpacity": 15,
"lineInterpolation": "stepAfter",
"lineWidth": 2,
"pointSize": 3,
"showPoints": "never",
"spanNulls": true,
"stacking": {
"group": "A",
"mode": "none"
}
},
"unit": "short",
"decimals": 0
},
"overrides": []
},
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"application": {
"filter": ""
},
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"functions": [],
"group": {
"filter": "Linux servers"
},
"host": {
"filter": "$proxy_host"
},
"item": {
"filter": "/^Telemt: (Active IP address count|Active connections)$/"
},
"itemTag": {
"filter": ""
},
"mode": 0,
"options": {
"count": false,
"disableDataAlignment": false,
"showDisabledItems": false,
"skipEmptyValues": false,
"useTrends": "default",
"useZabbixValueMapping": false
},
"queryType": "0",
"refId": "A",
"resultFormat": "time_series",
"schema": 12
}
]
},
{
"id": 24,
"type": "timeseries",
"title": "Трафик Telemt (накопительно)",
"description": "Накопительные байтовые счётчики Telemt по user secret `tekhnohim`; сбрасываются при перезапуске сервиса.",
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 12
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"drawStyle": "line",
"fillOpacity": 15,
"lineInterpolation": "linear",
"lineWidth": 2,
"pointSize": 3,
"showPoints": "never",
"spanNulls": true,
"stacking": {
"group": "A",
"mode": "none"
}
},
"unit": "bytes"
},
"overrides": []
},
"options": {
"legend": {
"calcs": [
"lastNotNull",
"max"
],
"displayMode": "table",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"application": {
"filter": ""
},
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"functions": [],
"group": {
"filter": "Linux servers"
},
"host": {
"filter": "$proxy_host"
},
"item": {
"filter": "/^Telemt: Bytes (received from|sent to) clients$/"
},
"itemTag": {
"filter": ""
},
"mode": 0,
"options": {
"count": false,
"disableDataAlignment": false,
"showDisabledItems": false,
"skipEmptyValues": false,
"useTrends": "default",
"useZabbixValueMapping": false
},
"queryType": "0",
"refId": "A",
"resultFormat": "time_series",
"schema": 12
}
]
},
{
"id": 25,
"type": "stat",
"title": "Аптайм Telemt",
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"gridPos": {
"h": 4,
"w": 8,
"x": 0,
"y": 20
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "value_and_name",
"justifyMode": "center"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "fixed",
"fixedColor": "green"
},
"unit": "s"
},
"overrides": []
},
"targets": [
{
"application": {
"filter": ""
},
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"functions": [],
"group": {
"filter": "Linux servers"
},
"host": {
"filter": "$proxy_host"
},
"item": {
"filter": "/^Telemt: Uptime$/"
},
"itemTag": {
"filter": ""
},
"mode": 0,
"options": {
"count": false,
"disableDataAlignment": false,
"showDisabledItems": false,
"skipEmptyValues": false,
"useTrends": "default",
"useZabbixValueMapping": false
},
"queryType": "0",
"refId": "A",
"resultFormat": "time_series",
"schema": 12
}
]
},
{
"id": 26,
"type": "stat",
"title": "Получено от клиентов",
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"gridPos": {
"h": 4,
"w": 8,
"x": 8,
"y": 20
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "value_and_name",
"justifyMode": "center"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "fixed",
"fixedColor": "semi-dark-blue"
},
"unit": "bytes"
},
"overrides": []
},
"targets": [
{
"application": {
"filter": ""
},
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"functions": [],
"group": {
"filter": "Linux servers"
},
"host": {
"filter": "$proxy_host"
},
"item": {
"filter": "/^Telemt: Bytes received from clients$/"
},
"itemTag": {
"filter": ""
},
"mode": 0,
"options": {
"count": false,
"disableDataAlignment": false,
"showDisabledItems": false,
"skipEmptyValues": false,
"useTrends": "default",
"useZabbixValueMapping": false
},
"queryType": "0",
"refId": "A",
"resultFormat": "time_series",
"schema": 12
}
]
},
{
"id": 27,
"type": "stat",
"title": "Отправлено клиентам",
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"gridPos": {
"h": 4,
"w": 8,
"x": 16,
"y": 20
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"colorMode": "value",
"graphMode": "none",
"textMode": "value_and_name",
"justifyMode": "center"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "fixed",
"fixedColor": "semi-dark-purple"
},
"unit": "bytes"
},
"overrides": []
},
"targets": [
{
"application": {
"filter": ""
},
"datasource": {
"type": "alexanderzobnin-zabbix-datasource",
"uid": "zabbix-depot"
},
"functions": [],
"group": {
"filter": "Linux servers"
},
"host": {
"filter": "$proxy_host"
},
"item": {
"filter": "/^Telemt: Bytes sent to clients$/"
},
"itemTag": {
"filter": ""
},
"mode": 0,
"options": {
"count": false,
"disableDataAlignment": false,
"showDisabledItems": false,
"skipEmptyValues": false,
"useTrends": "default",
"useZabbixValueMapping": false
},
"queryType": "0",
"refId": "A",
"resultFormat": "time_series",
"schema": 12
}
]
}
]
}
logging/grafana-provisioning/datasources/zabbix.yaml
+22
View File
@@ -0,0 +1,22 @@
apiVersion: 1
datasources:
- name: Zabbix
uid: zabbix-depot
type: alexanderzobnin-zabbix-datasource
access: proxy
url: http://10.0.0.11/api_jsonrpc.php
editable: false
jsonData:
authType: token
cacheTTL: 1h
dbConnectionEnable: false
disableDataAlignment: false
disableReadOnlyUsersAck: true
queryTimeout: 60
timeout: 30
trends: true
trendsFrom: 7d
trendsRange: 4d
secureJsonData:
apiToken: ${ZABBIX_API_TOKEN}
logging/syslog-ng.conf
+30
View File
@@ -0,0 +1,30 @@
@version: 3.29
@include "scl.conf"
source s_local {
internal();
};
# default-network-drivers accepts the mixed/raw formats used by the DEPOT
# fleet on the standard UDP/TCP syslog listeners.
source s_network {
default-network-drivers();
};
# Alloy's syslog receiver accepts one explicit format per listener. Normalize
# every accepted input to RFC5424 before relaying it over the private network.
template t_alloy_rfc5424 {
template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} - ${MSG}\n");
};
destination d_alloy {
syslog("alloy" transport("tcp") port("1514")
template(t_alloy_rfc5424)
);
};
log {
source(s_local);
source(s_network);
destination(d_alloy);
};
logging/zabbix-agent/telemt.conf
+8
View File
@@ -0,0 +1,8 @@
# Cached by zabbix-telemt-collector.service; contains no proxy links or secrets.
UserParameter=telemt.active_ips.list,cat /var/lib/zabbix/telemt-active-ips.txt
UserParameter=telemt.active_ips.count,/usr/local/libexec/zabbix-telemt-value active_ips_count
UserParameter=telemt.connections.current,/usr/local/libexec/zabbix-telemt-value connections_current
UserParameter=telemt.connections.total,/usr/local/libexec/zabbix-telemt-value connections_total
UserParameter=telemt.bytes.received,/usr/local/libexec/zabbix-telemt-value bytes_received
UserParameter=telemt.bytes.sent,/usr/local/libexec/zabbix-telemt-value bytes_sent
UserParameter=telemt.uptime,/usr/local/libexec/zabbix-telemt-value uptime
logging/zabbix-agent/zabbix-telemt-collector
Executable
+65
View File
@@ -0,0 +1,65 @@
#!/usr/bin/bash
set -euo pipefail
container_name="ts-lan-telemt"
output_dir="/var/lib/zabbix"
json_target="${output_dir}/telemt-active-ips.json"
text_target="${output_dir}/telemt-active-ips.txt"
metrics_target="${output_dir}/telemt-metrics.prom"
install -d -o root -g zabbix -m 0750 "${output_dir}"
container_pid="$(docker inspect --format '{{.State.Pid}}' "${container_name}")"
if [[ -z "${container_pid}" || "${container_pid}" == "0" ]]; then
exit 1
fi
host_ip="$(ip -4 route get 1.1.1.1 | awk '{for (i=1; i<=NF; i++) if ($i == "src") {print $(i+1); exit}}')"
if [[ -z "${host_ip}" ]]; then
exit 1
fi
json_tmp="$(mktemp "${output_dir}/.telemt-active-ips.json.XXXXXX")"
text_tmp="$(mktemp "${output_dir}/.telemt-active-ips.txt.XXXXXX")"
metrics_tmp="$(mktemp "${output_dir}/.telemt-metrics.prom.XXXXXX")"
trap 'rm -f "${json_tmp}" "${text_tmp}" "${metrics_tmp}"' EXIT
# The API response also contains proxy links and secrets. Filter it in-memory
# and persist only the username and current active IP address list.
nsenter -t "${container_pid}" -n \
curl --fail --silent --show-error --max-time 5 \
http://127.0.0.1:9091/v1/users |
jq --compact-output '{
users: [
.data[]
| {
username,
active_unique_ips: (.active_unique_ips_list // [])
}
]
}' > "${json_tmp}"
{
active_count="$(jq '[.users[].active_unique_ips[]] | unique | length' "${json_tmp}")"
printf 'Active IP addresses: %s\n' "${active_count}"
jq --raw-output '
[
.users[] as $user
| $user.active_unique_ips[]
| "\($user.username) | \(.)"
]
| if length == 0 then ["No active IP addresses"] else . end
| .[]
' "${json_tmp}"
} > "${text_tmp}"
# Cache Prometheus output locally. Zabbix active checks read the cache, so the
# central server does not need a route to each private Docker host.
curl --fail --silent --show-error --max-time 5 \
"http://${host_ip}:9092/metrics" > "${metrics_tmp}"
chown root:zabbix "${json_tmp}" "${text_tmp}" "${metrics_tmp}"
chmod 0640 "${json_tmp}" "${text_tmp}" "${metrics_tmp}"
mv -f "${json_tmp}" "${json_target}"
mv -f "${text_tmp}" "${text_target}"
mv -f "${metrics_tmp}" "${metrics_target}"
logging/zabbix-agent/zabbix-telemt-collector.service
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=Collect sanitized Telemt data for Zabbix
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
ExecStart=/usr/local/libexec/zabbix-telemt-collector
User=root
Group=root
PrivateTmp=true
NoNewPrivileges=true
logging/zabbix-agent/zabbix-telemt-collector.timer
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=Refresh sanitized Telemt data for Zabbix
[Timer]
OnBootSec=30s
OnUnitActiveSec=30s
AccuracySec=2s
Unit=zabbix-telemt-collector.service
Persistent=true
[Install]
WantedBy=timers.target
logging/zabbix-agent/zabbix-telemt-value
Executable
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/bash
set -euo pipefail
json_file="/var/lib/zabbix/telemt-active-ips.json"
metrics_file="/var/lib/zabbix/telemt-metrics.prom"
case "${1:-}" in
active_ips_count)
jq '[.users[].active_unique_ips[]] | unique | length' "${json_file}"
;;
connections_current)
awk '$1 ~ /^telemt_user_connections_current\{/ {sum += $2} END {print sum + 0}' "${metrics_file}"
;;
connections_total)
awk '$1 ~ /^telemt_user_connections_total\{/ {sum += $2} END {print sum + 0}' "${metrics_file}"
;;
bytes_received)
awk '$1 ~ /^telemt_user_octets_from_client\{/ {sum += $2} END {printf "%.0f\n", sum + 0}' "${metrics_file}"
;;
bytes_sent)
awk '$1 ~ /^telemt_user_octets_to_client\{/ {sum += $2} END {printf "%.0f\n", sum + 0}' "${metrics_file}"
;;
uptime)
awk '$1 == "telemt_uptime_seconds" {printf "%.0f\n", $2; found=1} END {if (!found) exit 1}' "${metrics_file}"
;;
*)
printf 'unsupported Telemt value: %s\n' "${1:-}" >&2
exit 2
;;
esac
logging/zabbix-template-telemt-active.yaml
+94
View File
@@ -0,0 +1,94 @@
zabbix_export:
version: '7.0'
template_groups:
- uuid: a571c0d144b14fd4a87a9d9b2aa9fcd6
name: Templates/Applications
templates:
- uuid: 1d617f451f8640089f9ac33d62865e72
template: 'Telemt proxy by active agent'
name: 'Telemt proxy by active agent'
description: |
Telemt metrics cached locally by zabbix-telemt-collector.timer.
Designed for hosts behind NAT; every item is a Zabbix agent active check.
groups:
- name: Templates/Applications
items:
- uuid: e383ca45b2c047519f30665999cfc02e
name: 'Telemt: Active IP address count'
type: ZABBIX_ACTIVE
key: telemt.active_ips.count
delay: 30s
history: 31d
trends: 365d
value_type: UNSIGNED
tags:
- tag: component
value: telemt
- uuid: 5a29849443d5473884246d049a2dd79f
name: 'Telemt: Active IP addresses'
type: ZABBIX_ACTIVE
key: telemt.active_ips.list
delay: 30s
history: 1d
value_type: TEXT
tags:
- tag: component
value: telemt
- uuid: 465288678a084f8c84f56c2cffc0dcda
name: 'Telemt: Active connections'
type: ZABBIX_ACTIVE
key: telemt.connections.current
delay: 30s
history: 31d
trends: 365d
value_type: UNSIGNED
tags:
- tag: component
value: telemt
- uuid: dddf99d04abb459f8890eff46b948c4b
name: 'Telemt: Total connections'
type: ZABBIX_ACTIVE
key: telemt.connections.total
delay: 30s
history: 31d
trends: 365d
value_type: UNSIGNED
tags:
- tag: component
value: telemt
- uuid: 03e3a3671d744efaa65d4328d8d4035f
name: 'Telemt: Bytes received from clients'
type: ZABBIX_ACTIVE
key: telemt.bytes.received
delay: 30s
history: 31d
trends: 365d
value_type: UNSIGNED
units: B
tags:
- tag: component
value: telemt
- uuid: 97679bf1d06b47b0acf8f33f1253d114
name: 'Telemt: Bytes sent to clients'
type: ZABBIX_ACTIVE
key: telemt.bytes.sent
delay: 30s
history: 31d
trends: 365d
value_type: UNSIGNED
units: B
tags:
- tag: component
value: telemt
- uuid: b554b83a21ca46638c753e803870661f
name: 'Telemt: Uptime'
type: ZABBIX_ACTIVE
key: telemt.uptime
delay: 30s
history: 31d
trends: 365d
value_type: UNSIGNED
units: uptime
tags:
- tag: component
value: telemt