00-depot: add persistent Loki Alloy Grafana stack

2026-07-31 11:10:47 +03:00
parent 7ac58cb1ea
commit 961f0112de
7 changed files with 878 additions and 0 deletions
@@ -0,0 +1,126 @@
+# Internal logging stack
+
+Portainer Git stack for `outline` (`10.0.0.6`). It replaces only the logging
+part of the manual `mktxp-stack`:
+
+- `loki` stores syslog in the persistent host path
+  `/mnt/containers/logging/loki-data`;
+- `alloy` replaces the `syslog-ng` -> `promtail` relay;
+- `grafana` provides the existing MikroTik Loki dashboard.
+
+The retired stack's `mktxp`, Prometheus, Promtail, and syslog-ng services are
+not part of this stack.
+
+## Portainer settings
+
+- Repository: `https://git.it-depot.ru/depot/docker-infrastructure`
+- Branch: `master`
+- Compose path: `logging/docker-compose.yaml`
+- Required environment variable: `GRAFANA_ADMIN_PASSWORD`
+
+Do not deploy the new stack before completing the cutover below: the current
+containers still own names `loki` and `grafana` and host ports 3000, 3100, 514,
+and 601.
+
+## Host files
+
+The Git-backed compose file intentionally bind-mounts stable host paths.
+Before deployment, the repository files must be copied to:
+
+```text
+/mnt/containers/logging/
+├── alloy-config/config.alloy
+├── alloy-data/
+├── grafana-config/
+│   ├── dashboards/mikrotik-loki-logs.json
+│   └── provisioning/
+│       ├── dashboards/dashboards.yaml
+│       └── datasources/loki.yaml
+├── grafana-data/
+├── loki-config/config.yaml
+└── loki-data/
+```
+
+SELinux is enforcing. The compose mounts use `:Z`; do not remove those flags.
+The persistent data directories should be owned by the container users:
+Loki `10001:10001`, Grafana `472:0`, and Alloy `0:0`.
+
+## Cutover
+
+The live Loki history is inside the old container at `/tmp/loki`. A stopped,
+consistent rescue copy from 2026-07-30 is pre-seeded in the new host path, but
+it must be refreshed from the stopped live container immediately before the
+old stack is removed:
+
+```bash
+cd /mnt/containers/mktxp/mktxp-stack
+
+# Record a last pre-cutover query and stop all writers cleanly.
+curl -fsS http://127.0.0.1:3100/ready
+docker compose stop
+
+# The stopped Loki container still contains the newest WAL/chunks.
+docker cp -a loki:/tmp/loki/. /mnt/containers/logging/loki-data/
+chown -R 10001:10001 /mnt/containers/logging/loki-data
+
+# Removes containers and the old network, but NOT the old named Grafana and
+# Prometheus volumes. Do not add --volumes.
+docker compose down
+```
+
+Now deploy the Portainer Git stack. Expected interruption is the time between
+`docker compose stop` and the new stack becoming ready.
+
+## Verification
+
+```bash
+docker ps --filter name=loki --filter name=alloy --filter name=grafana
+curl -fsS http://127.0.0.1:3100/ready
+curl -fsS http://127.0.0.1:3000/api/health
+ss -lnup | grep ':514'
+ss -lntp | grep -E ':(601|3000|3100)\b'
+```
+
+Confirm both history and new ingestion:
+
+```bash
+START=$(date -u -d '24 hours ago' +%s%N)
+curl -fsSG http://127.0.0.1:3100/loki/api/v1/query_range \
+  --data-urlencode 'query={job="syslog"}' \
+  --data-urlencode "start=$START" \
+  --data-urlencode 'limit=1'
+
+logger --server 127.0.0.1 --udp --port 514 \
+  --rfc3164 --tag depot-cutover-test 'alloy ingestion test'
+```
+
+In Grafana, the provisioned **Network / Mikrotik Loki Logs** dashboard should
+list existing `routerboard` values and show both pre-cutover and post-cutover
+entries.
+
+## Rollback
+
+Remove the new Portainer stack, then recreate the old stack with its Loki data
+bound back into `/tmp/loki`:
+
+```bash
+cd /mnt/containers/mktxp/mktxp-stack
+cp docker-compose.yaml docker-compose.rollback.yaml
+```
+
+Add this second volume to the `loki` service in
+`docker-compose.rollback.yaml`:
+
+```yaml
+      - /mnt/containers/logging/loki-data:/tmp/loki:Z
+```
+
+Then run:
+
+```bash
+docker compose -f docker-compose.rollback.yaml up -d
+curl -fsS http://127.0.0.1:3100/ready
+```
+
+Do not restore the old Loki without that bind mount: its original writable
+layer is deleted when the old stack is taken down.