docker-infrastructure/logging

Internal logging stack

Portainer Git stack for outline (10.0.0.6). It replaces only the logging part of the manual mktxp-stack:

• loki stores syslog in the persistent host path /mnt/containers/logging/loki-data;
• syslog-ng accepts and normalizes the fleet's mixed/raw syslog formats;
• alloy replaces Promtail and sends normalized RFC5424 to Loki;
• grafana provides the MikroTik Loki dashboard and a Telemt fleet dashboard backed by the internal DEPOT Zabbix server.

The retired stack's mktxp, Prometheus, and Promtail services are not part of this stack. syslog-ng remains because real DEPOT RouterOS traffic includes raw messages that Alloy's strict RFC3164 parser rejects.

Portainer settings

• Repository: https://git.it-depot.ru/depot/docker-infrastructure
• Branch: master
• Compose path: logging/docker-compose.yaml
• Required environment variables:
  • GRAFANA_ADMIN_PASSWORD
  • ZABBIX_API_TOKEN — token for a read-only Zabbix API user; it is passed into Grafana's provisioned datasource and must not be committed.

Do not deploy the new stack before completing the cutover below: the current containers still own names loki and grafana and host ports 3000, 3100, 514, and 601.

Host files

The Git-backed compose file intentionally bind-mounts stable host paths. Before deployment, the repository files must be copied to:

/mnt/containers/logging/
├── alloy-config/config.alloy
├── alloy-data/
├── grafana-config/
│   ├── dashboards/mikrotik-loki-logs.json
│   ├── dashboards/telemt-proxy-fleet.json
│   └── provisioning/
│       ├── dashboards/dashboards.yaml
│       └── datasources/{loki,zabbix}.yaml
├── grafana-data/
├── loki-config/config.yaml
├── loki-data/
└── syslog-ng-config/syslog-ng.conf

SELinux is enforcing. The compose mounts use :Z; do not remove those flags. The persistent data directories should be owned by the container users: Loki 10001:10001, Grafana 472:0, and Alloy 0:0.

Cutover

The live Loki history is inside the old container at /tmp/loki. Do not merge a newer WAL over an older rescue WAL: the resulting non-contiguous segment sequence makes Loki fail with segments are not sequential. Instead, move the seed aside and copy the stopped container's complete data directory as one generation:

cd /mnt/containers/mktxp/mktxp-stack

# Record a last pre-cutover query and stop all writers cleanly.
curl -fsS http://127.0.0.1:3100/ready
docker compose stop

# Preserve the seed separately; never overlay WAL generations.
mv /mnt/containers/logging/loki-data \
  /mnt/containers/logging/loki-data-rescue-seed-2026-07-30
docker cp -a loki:/tmp/loki /mnt/containers/logging/loki-data
chown -R 10001:10001 /mnt/containers/logging/loki-data

# Removes containers and the old network, but NOT the old named Grafana and
# Prometheus volumes. Do not add --volumes.
docker compose down

Now deploy the Portainer Git stack. Expected interruption is the time between docker compose stop and the new stack becoming ready.

Verification

docker ps --filter name=loki --filter name=alloy --filter name=grafana
curl -fsS http://127.0.0.1:3100/ready
curl -fsS http://127.0.0.1:3000/api/health
ss -lnup | grep ':514'
ss -lntp | grep -E ':(601|3000|3100)\b'

Confirm both history and new ingestion:

START=$(date -u -d '24 hours ago' +%s%N)
curl -fsSG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="syslog"}' \
  --data-urlencode "start=$START" \
  --data-urlencode 'limit=1'

logger --server 127.0.0.1 --udp --port 514 \
  --rfc3164 --tag depot-cutover-test 'alloy ingestion test'

In Grafana, the provisioned Network / Mikrotik Loki Logs dashboard should list existing routerboard values and show both pre-cutover and post-cutover entries.

Telemt / Zabbix monitoring

The Zabbix server is 10.0.0.11 internally. The DEPOT router at 10.0.0.1 publishes TCP 185.108.4.158:10051 only for the three proxy-host egress addresses, allowing Zabbix Agent 2 active checks without exposing the trapper to the whole internet.

Create the following Zabbix hosts with these exact technical Host name values (visible names may be friendlier):

Client	Zabbix Host name
00-xserver	00-xserver-fedora-xxx
11-nevafud	11-nevafud-fedora
19-nutrilent	19-nutrilent-docker

Import zabbix-template-telemt-active.yaml, link Telemt proxy by active agent to all three hosts, and ensure each host is in the Linux servers group. Do not add a passive-agent interface: these hosts are behind NAT and send active checks to 185.108.4.158.

The matching host-side collector is in zabbix-agent/. It enters only the Telemt container network namespace for the control API, discards links and secrets, caches the active-IP list and Prometheus values, and exposes read-only Agent 2 keys. The active-IP list retains one day of history; the numeric count and other counters retain 31 days.

After redeploying Grafana with a valid ZABBIX_API_TOKEN, verify:

docker exec grafana grafana cli plugins ls
curl -fsS http://127.0.0.1:3000/api/health

The provisioned Network / Telegram proxy fleet dashboard selects one of the three exact host names above and mirrors the Telemt section of the client-02 dashboard: current IP count, active/total connections, IP list, traffic, and uptime.

Rollback

Remove the new Portainer stack, then recreate the old stack with its Loki data bound back into /tmp/loki:

cd /mnt/containers/mktxp/mktxp-stack
cp docker-compose.yaml docker-compose.rollback.yaml

Add this second volume to the loki service in docker-compose.rollback.yaml:

      - /mnt/containers/logging/loki-data:/tmp/loki:Z

Then run:

docker compose -f docker-compose.rollback.yaml up -d
curl -fsS http://127.0.0.1:3100/ready

Do not restore the old Loki without that bind mount: its original writable layer is deleted when the old stack is taken down.