127 lines
3.8 KiB
Markdown
127 lines
3.8 KiB
Markdown
# Internal logging stack
|
|
|
|
Portainer Git stack for `outline` (`10.0.0.6`). It replaces only the logging
|
|
part of the manual `mktxp-stack`:
|
|
|
|
- `loki` stores syslog in the persistent host path
|
|
`/mnt/containers/logging/loki-data`;
|
|
- `alloy` replaces the `syslog-ng` -> `promtail` relay;
|
|
- `grafana` provides the existing MikroTik Loki dashboard.
|
|
|
|
The retired stack's `mktxp`, Prometheus, Promtail, and syslog-ng services are
|
|
not part of this stack.
|
|
|
|
## Portainer settings
|
|
|
|
- Repository: `https://git.it-depot.ru/depot/docker-infrastructure`
|
|
- Branch: `master`
|
|
- Compose path: `logging/docker-compose.yaml`
|
|
- Required environment variable: `GRAFANA_ADMIN_PASSWORD`
|
|
|
|
Do not deploy the new stack before completing the cutover below: the current
|
|
containers still own names `loki` and `grafana` and host ports 3000, 3100, 514,
|
|
and 601.
|
|
|
|
## Host files
|
|
|
|
The Git-backed compose file intentionally bind-mounts stable host paths.
|
|
Before deployment, the repository files must be copied to:
|
|
|
|
```text
|
|
/mnt/containers/logging/
|
|
├── alloy-config/config.alloy
|
|
├── alloy-data/
|
|
├── grafana-config/
|
|
│ ├── dashboards/mikrotik-loki-logs.json
|
|
│ └── provisioning/
|
|
│ ├── dashboards/dashboards.yaml
|
|
│ └── datasources/loki.yaml
|
|
├── grafana-data/
|
|
├── loki-config/config.yaml
|
|
└── loki-data/
|
|
```
|
|
|
|
SELinux is enforcing. The compose mounts use `:Z`; do not remove those flags.
|
|
The persistent data directories should be owned by the container users:
|
|
Loki `10001:10001`, Grafana `472:0`, and Alloy `0:0`.
|
|
|
|
## Cutover
|
|
|
|
The live Loki history is inside the old container at `/tmp/loki`. A stopped,
|
|
consistent rescue copy from 2026-07-30 is pre-seeded in the new host path, but
|
|
it must be refreshed from the stopped live container immediately before the
|
|
old stack is removed:
|
|
|
|
```bash
|
|
cd /mnt/containers/mktxp/mktxp-stack
|
|
|
|
# Record a last pre-cutover query and stop all writers cleanly.
|
|
curl -fsS http://127.0.0.1:3100/ready
|
|
docker compose stop
|
|
|
|
# The stopped Loki container still contains the newest WAL/chunks.
|
|
docker cp -a loki:/tmp/loki/. /mnt/containers/logging/loki-data/
|
|
chown -R 10001:10001 /mnt/containers/logging/loki-data
|
|
|
|
# Removes containers and the old network, but NOT the old named Grafana and
|
|
# Prometheus volumes. Do not add --volumes.
|
|
docker compose down
|
|
```
|
|
|
|
Now deploy the Portainer Git stack. Expected interruption is the time between
|
|
`docker compose stop` and the new stack becoming ready.
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
docker ps --filter name=loki --filter name=alloy --filter name=grafana
|
|
curl -fsS http://127.0.0.1:3100/ready
|
|
curl -fsS http://127.0.0.1:3000/api/health
|
|
ss -lnup | grep ':514'
|
|
ss -lntp | grep -E ':(601|3000|3100)\b'
|
|
```
|
|
|
|
Confirm both history and new ingestion:
|
|
|
|
```bash
|
|
START=$(date -u -d '24 hours ago' +%s%N)
|
|
curl -fsSG http://127.0.0.1:3100/loki/api/v1/query_range \
|
|
--data-urlencode 'query={job="syslog"}' \
|
|
--data-urlencode "start=$START" \
|
|
--data-urlencode 'limit=1'
|
|
|
|
logger --server 127.0.0.1 --udp --port 514 \
|
|
--rfc3164 --tag depot-cutover-test 'alloy ingestion test'
|
|
```
|
|
|
|
In Grafana, the provisioned **Network / Mikrotik Loki Logs** dashboard should
|
|
list existing `routerboard` values and show both pre-cutover and post-cutover
|
|
entries.
|
|
|
|
## Rollback
|
|
|
|
Remove the new Portainer stack, then recreate the old stack with its Loki data
|
|
bound back into `/tmp/loki`:
|
|
|
|
```bash
|
|
cd /mnt/containers/mktxp/mktxp-stack
|
|
cp docker-compose.yaml docker-compose.rollback.yaml
|
|
```
|
|
|
|
Add this second volume to the `loki` service in
|
|
`docker-compose.rollback.yaml`:
|
|
|
|
```yaml
|
|
- /mnt/containers/logging/loki-data:/tmp/loki:Z
|
|
```
|
|
|
|
Then run:
|
|
|
|
```bash
|
|
docker compose -f docker-compose.rollback.yaml up -d
|
|
curl -fsS http://127.0.0.1:3100/ready
|
|
```
|
|
|
|
Do not restore the old Loki without that bind mount: its original writable
|
|
layer is deleted when the old stack is taken down.
|