docker-infrastructure/logging/README.md

Internal logging stack

Portainer Git stack for outline (10.0.0.6). It replaces only the logging part of the manual mktxp-stack:

• loki stores syslog in the persistent host path /mnt/containers/logging/loki-data;
• syslog-ng accepts and normalizes the fleet's mixed/raw syslog formats;
• alloy replaces Promtail and sends normalized RFC5424 to Loki;
• grafana provides the MikroTik Loki dashboard and a Telemt fleet dashboard backed by the internal DEPOT Zabbix server.

The retired stack's mktxp, Prometheus, and Promtail services are not part of this stack. syslog-ng remains because real DEPOT RouterOS traffic includes raw messages that Alloy's strict RFC3164 parser rejects.

Portainer settings

• Repository: https://git.it-depot.ru/depot/docker-infrastructure
• Branch: master
• Compose path: logging/docker-compose.yaml
• Required environment variables:
  • GRAFANA_ADMIN_PASSWORD
  • ZABBIX_API_TOKEN — token for a read-only Zabbix API user; it is passed into Grafana's provisioned datasource and must not be committed.

Do not deploy the new stack before completing the cutover below: the current containers still own names loki and grafana and host ports 3000, 3100, 514, and 601.

Host files

The Git-backed compose file intentionally bind-mounts stable host paths. Before deployment, the repository files must be copied to:

/mnt/containers/logging/
├── alloy-config/config.alloy
├── alloy-data/
├── grafana-config/
│   ├── dashboards/mikrotik-loki-logs.json
│   ├── dashboards/telemt-proxy-fleet.json
│   └── provisioning/
│       ├── dashboards/dashboards.yaml
│       └── datasources/{loki,zabbix}.yaml
├── grafana-data/
├── loki-config/config.yaml
├── loki-data/
└── syslog-ng-config/syslog-ng.conf

SELinux is enforcing. The compose mounts use :Z; do not remove those flags. The persistent data directories should be owned by the container users: Loki 10001:10001, Grafana 472:0, and Alloy 0:0.

Cutover

The live Loki history is inside the old container at /tmp/loki. Do not merge a newer WAL over an older rescue WAL: the resulting non-contiguous segment sequence makes Loki fail with segments are not sequential. Instead, move the seed aside and copy the stopped container's complete data directory as one generation:

cd /mnt/containers/mktxp/mktxp-stack

# Record a last pre-cutover query and stop all writers cleanly.
curl -fsS http://127.0.0.1:3100/ready
docker compose stop

# Preserve the seed separately; never overlay WAL generations.
mv /mnt/containers/logging/loki-data \
  /mnt/containers/logging/loki-data-rescue-seed-2026-07-30
docker cp -a loki:/tmp/loki /mnt/containers/logging/loki-data
chown -R 10001:10001 /mnt/containers/logging/loki-data

# Removes containers and the old network, but NOT the old named Grafana and
# Prometheus volumes. Do not add --volumes.
docker compose down

Now deploy the Portainer Git stack. Expected interruption is the time between docker compose stop and the new stack becoming ready.

Verification

docker ps --filter name=loki --filter name=alloy --filter name=grafana
curl -fsS http://127.0.0.1:3100/ready
curl -fsS http://127.0.0.1:3000/api/health
ss -lnup | grep ':514'
ss -lntp | grep -E ':(601|3000|3100)\b'

Confirm both history and new ingestion:

START=$(date -u -d '24 hours ago' +%s%N)
curl -fsSG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="syslog"}' \
  --data-urlencode "start=$START" \
  --data-urlencode 'limit=1'

logger --server 127.0.0.1 --udp --port 514 \
  --rfc3164 --tag depot-cutover-test 'alloy ingestion test'

In Grafana, the provisioned Network / Mikrotik Loki Logs dashboard should list existing routerboard values and show both pre-cutover and post-cutover entries.

Telemt / Zabbix monitoring

The Zabbix server is 10.0.0.11 internally. The DEPOT router at 10.0.0.1 publishes TCP 185.108.4.158:10051 only for the four proxy-host egress addresses, allowing Zabbix Agent 2 active checks without exposing the trapper to the whole internet.

Create the following Zabbix hosts. The technical Host name must match the Agent 2 Hostname; Grafana selects the Visible name shown below.

Client	Zabbix Host name	Visible name
00-xserver	00-xserver-fedora-xxx	00-xserver-fedora-xxx
02-tekhnohim	Fedora Kirochnaya	02-tekhnohim-docker
11-nevafud	11-nevafud-fedora	11-nevafud-fedora
19-nutrilent	19-nutrilent-docker	19-nutrilent-docker

Import zabbix-template-telemt-active.yaml, link Telemt proxy by active agent to all four hosts, and ensure each host is in the Linux servers group. All four send active checks to 185.108.4.158. Client 02 should have no DEPOT passive-agent interface; its existing local Zabbix server continues to perform the host's passive Linux checks independently.

The matching host-side collector is in zabbix-agent/. It enters only the Telemt container network namespace for the control API, discards links and secrets, caches the active-IP list and Prometheus values, and exposes read-only Agent 2 keys. The active-IP list retains one day of history; the numeric count and other counters retain 31 days.

After redeploying Grafana with a valid ZABBIX_API_TOKEN, verify:

docker exec grafana grafana cli plugins ls
curl -fsS http://127.0.0.1:3000/api/health

The provisioned Network / Telegram proxy fleet dashboard selects one of the four visible host names above and mirrors the Telemt section of the client-02 dashboard: current IP count, active/total connections, IP list, traffic, and uptime.

Rollback

Remove the new Portainer stack, then recreate the old stack with its Loki data bound back into /tmp/loki:

cd /mnt/containers/mktxp/mktxp-stack
cp docker-compose.yaml docker-compose.rollback.yaml

Add this second volume to the loki service in docker-compose.rollback.yaml:

      - /mnt/containers/logging/loki-data:/tmp/loki:Z

Then run:

docker compose -f docker-compose.rollback.yaml up -d
curl -fsS http://127.0.0.1:3100/ready

Do not restore the old Loki without that bind mount: its original writable layer is deleted when the old stack is taken down.