depot/docker-infrastructure
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
Files
f0889261b2fcc2649e8a8d28115e3b67a4dcb955
docker-infrastructure/logging/README.md
T
pipistrello f0889261b2 logging: make host-bound GitOps files reproducibly stageable 
The Portainer stack consumes its compose file from Git but bind-mounts runtime configuration from stable paths under /mnt/containers/logging. The stable host copy of docker-compose.yaml had consequently fallen behind the deployed Git revision even though the running stack and dashboard files were current.\n\nAdd a root-run staging/check helper that copies and byte-verifies Compose, Loki, Alloy, syslog-ng, Grafana provisioning, and both dashboards without touching credentials or persistent data. Document the workflow and the final source-restricted passive Agent 2 interfaces for all four Telemt hosts.\n\nValidated with bash -n and git diff --check. Unrelated .DS_Store changes remain untracked/uncommitted.
2026-07-31 14:41:22 +03:00

6.5 KiB
Raw Blame History

Internal logging stack

Portainer Git stack for outline (10.0.0.6). It replaces only the logging part of the manual mktxp-stack:

  • loki stores syslog in the persistent host path /mnt/containers/logging/loki-data;
  • syslog-ng accepts and normalizes the fleet's mixed/raw syslog formats;
  • alloy replaces Promtail and sends normalized RFC5424 to Loki;
  • grafana provides the MikroTik Loki dashboard and a Telemt fleet dashboard backed by the internal DEPOT Zabbix server.

The retired stack's mktxp, Prometheus, and Promtail services are not part of this stack. syslog-ng remains because real DEPOT RouterOS traffic includes raw messages that Alloy's strict RFC3164 parser rejects.

Portainer settings

  • Repository: https://git.it-depot.ru/depot/docker-infrastructure
  • Branch: master
  • Compose path: logging/docker-compose.yaml
  • Required environment variables:
    • GRAFANA_ADMIN_PASSWORD
    • ZABBIX_API_TOKEN — token for a read-only Zabbix API user; it is passed into Grafana's provisioned datasource and must not be committed.

Do not deploy the new stack before completing the cutover below: the current containers still own names loki and grafana and host ports 3000, 3100, 514, and 601.

Host files

The Git-backed compose file intentionally bind-mounts stable host paths. Before deployment, stage and verify every tracked runtime file with:

sudo logging/stage-host-files.sh
sudo logging/stage-host-files.sh --check

The script copies only non-secret configuration and dashboard files. Portainer still supplies GRAFANA_ADMIN_PASSWORD and ZABBIX_API_TOKEN out of band. The resulting layout is:

/mnt/containers/logging/
├── alloy-config/config.alloy
├── alloy-data/
├── grafana-config/
│   ├── dashboards/mikrotik-loki-logs.json
│   ├── dashboards/telemt-proxy-fleet.json
│   └── provisioning/
│       ├── dashboards/dashboards.yaml
│       └── datasources/{loki,zabbix}.yaml
├── grafana-data/
├── loki-config/config.yaml
├── loki-data/
└── syslog-ng-config/syslog-ng.conf

SELinux is enforcing. The compose mounts use :Z; do not remove those flags. The persistent data directories should be owned by the container users: Loki 10001:10001, Grafana 472:0, and Alloy 0:0.

Cutover

The live Loki history is inside the old container at /tmp/loki. Do not merge a newer WAL over an older rescue WAL: the resulting non-contiguous segment sequence makes Loki fail with segments are not sequential. Instead, move the seed aside and copy the stopped container's complete data directory as one generation:

cd /mnt/containers/mktxp/mktxp-stack

# Record a last pre-cutover query and stop all writers cleanly.
curl -fsS http://127.0.0.1:3100/ready
docker compose stop

# Preserve the seed separately; never overlay WAL generations.
mv /mnt/containers/logging/loki-data \
  /mnt/containers/logging/loki-data-rescue-seed-2026-07-30
docker cp -a loki:/tmp/loki /mnt/containers/logging/loki-data
chown -R 10001:10001 /mnt/containers/logging/loki-data

# Removes containers and the old network, but NOT the old named Grafana and
# Prometheus volumes. Do not add --volumes.
docker compose down

Now deploy the Portainer Git stack. Expected interruption is the time between docker compose stop and the new stack becoming ready.

Verification

docker ps --filter name=loki --filter name=alloy --filter name=grafana
curl -fsS http://127.0.0.1:3100/ready
curl -fsS http://127.0.0.1:3000/api/health
ss -lnup | grep ':514'
ss -lntp | grep -E ':(601|3000|3100)\b'

Confirm both history and new ingestion:

START=$(date -u -d '24 hours ago' +%s%N)
curl -fsSG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="syslog"}' \
  --data-urlencode "start=$START" \
  --data-urlencode 'limit=1'

logger --server 127.0.0.1 --udp --port 514 \
  --rfc3164 --tag depot-cutover-test 'alloy ingestion test'

In Grafana, the provisioned Network / Mikrotik Loki Logs dashboard should list existing routerboard values and show both pre-cutover and post-cutover entries.

Telemt / Zabbix monitoring

The Zabbix server is 10.0.0.11 internally. The DEPOT router at 10.0.0.1 publishes TCP 185.108.4.158:10051 only for the four proxy-host egress addresses, allowing Zabbix Agent 2 active checks without exposing the trapper to the whole internet.

Create the following Zabbix hosts. The technical Host name must match the Agent 2 Hostname; Grafana selects the Visible name shown below.

Client Zabbix Host name Visible name
00-xserver 00-xserver-fedora-xxx 00-xserver-fedora-xxx
02-tekhnohim Fedora Kirochnaya 02-tekhnohim-docker
11-nevafud 11-nevafud-fedora 11-nevafud-fedora
19-nutrilent 19-nutrilent-docker 19-nutrilent-docker

Import zabbix-template-telemt-active.yaml, link Telemt proxy by active agent to all four hosts, and ensure each host is in the Linux servers group. All four send active checks to 185.108.4.158 and have source-restricted passive Agent 2 interfaces on TCP 10050. Client 02's existing local Zabbix server continues to perform its original passive Linux checks independently.

The matching host-side collector is in zabbix-agent/. It enters only the Telemt container network namespace for the control API, discards links and secrets, caches the active-IP list and Prometheus values, and exposes read-only Agent 2 keys. The active-IP list retains one day of history; the numeric count and other counters retain 31 days.

After redeploying Grafana with a valid ZABBIX_API_TOKEN, verify:

docker exec grafana grafana cli plugins ls
curl -fsS http://127.0.0.1:3000/api/health

The provisioned Network / Telegram proxy fleet dashboard selects one of the four visible host names above and mirrors the Telemt section of the client-02 dashboard: current IP count, active/total connections, IP list, traffic, and uptime.

Rollback

Remove the new Portainer stack, then recreate the old stack with its Loki data bound back into /tmp/loki:

cd /mnt/containers/mktxp/mktxp-stack
cp docker-compose.yaml docker-compose.rollback.yaml

Add this second volume to the loki service in docker-compose.rollback.yaml:

      - /mnt/containers/logging/loki-data:/tmp/loki:Z

Then run:

docker compose -f docker-compose.rollback.yaml up -d
curl -fsS http://127.0.0.1:3100/ready

Do not restore the old Loki without that bind mount: its original writable layer is deleted when the old stack is taken down.

Reference in New Issue View Git Blame Copy Permalink