Internal logging stack

Portainer Git stack for outline (10.0.0.6). It replaces only the logging part of the manual mktxp-stack:

loki stores syslog in the persistent host path /mnt/containers/logging/loki-data;
alloy replaces the syslog-ng -> promtail relay;
grafana provides the existing MikroTik Loki dashboard.

The retired stack's mktxp, Prometheus, Promtail, and syslog-ng services are not part of this stack.

Portainer settings

Repository: https://git.it-depot.ru/depot/docker-infrastructure
Branch: master
Compose path: logging/docker-compose.yaml
Required environment variable: GRAFANA_ADMIN_PASSWORD

Do not deploy the new stack before completing the cutover below: the current containers still own names loki and grafana and host ports 3000, 3100, 514, and 601.

Host files

The Git-backed compose file intentionally bind-mounts stable host paths. Before deployment, the repository files must be copied to:

/mnt/containers/logging/
├── alloy-config/config.alloy
├── alloy-data/
├── grafana-config/
│   ├── dashboards/mikrotik-loki-logs.json
│   └── provisioning/
│       ├── dashboards/dashboards.yaml
│       └── datasources/loki.yaml
├── grafana-data/
├── loki-config/config.yaml
└── loki-data/

SELinux is enforcing. The compose mounts use :Z; do not remove those flags. The persistent data directories should be owned by the container users: Loki 10001:10001, Grafana 472:0, and Alloy 0:0.

Cutover

The live Loki history is inside the old container at /tmp/loki. A stopped, consistent rescue copy from 2026-07-30 is pre-seeded in the new host path, but it must be refreshed from the stopped live container immediately before the old stack is removed:

cd /mnt/containers/mktxp/mktxp-stack

# Record a last pre-cutover query and stop all writers cleanly.
curl -fsS http://127.0.0.1:3100/ready
docker compose stop

# The stopped Loki container still contains the newest WAL/chunks.
docker cp -a loki:/tmp/loki/. /mnt/containers/logging/loki-data/
chown -R 10001:10001 /mnt/containers/logging/loki-data

# Removes containers and the old network, but NOT the old named Grafana and
# Prometheus volumes. Do not add --volumes.
docker compose down

Now deploy the Portainer Git stack. Expected interruption is the time between docker compose stop and the new stack becoming ready.

Verification

docker ps --filter name=loki --filter name=alloy --filter name=grafana
curl -fsS http://127.0.0.1:3100/ready
curl -fsS http://127.0.0.1:3000/api/health
ss -lnup | grep ':514'
ss -lntp | grep -E ':(601|3000|3100)\b'

Confirm both history and new ingestion:

START=$(date -u -d '24 hours ago' +%s%N)
curl -fsSG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="syslog"}' \
  --data-urlencode "start=$START" \
  --data-urlencode 'limit=1'

logger --server 127.0.0.1 --udp --port 514 \
  --rfc3164 --tag depot-cutover-test 'alloy ingestion test'

In Grafana, the provisioned Network / Mikrotik Loki Logs dashboard should list existing routerboard values and show both pre-cutover and post-cutover entries.

Rollback

Remove the new Portainer stack, then recreate the old stack with its Loki data bound back into /tmp/loki:

cd /mnt/containers/mktxp/mktxp-stack
cp docker-compose.yaml docker-compose.rollback.yaml

Add this second volume to the loki service in docker-compose.rollback.yaml:

      - /mnt/containers/logging/loki-data:/tmp/loki:Z

Then run:

docker compose -f docker-compose.rollback.yaml up -d
curl -fsS http://127.0.0.1:3100/ready

Do not restore the old Loki without that bind mount: its original writable layer is deleted when the old stack is taken down.

3.8 KiB Raw Blame History