# Internal logging stack

Portainer Git stack for `outline` (`10.0.0.6`). It replaces only the logging
part of the manual `mktxp-stack`:

- `loki` stores syslog in the persistent host path
  `/mnt/containers/logging/loki-data`;
- `alloy` replaces the `syslog-ng` -> `promtail` relay;
- `grafana` provides the existing MikroTik Loki dashboard.

The retired stack's `mktxp`, Prometheus, Promtail, and syslog-ng services are
not part of this stack.

## Portainer settings

- Repository: `https://git.it-depot.ru/depot/docker-infrastructure`
- Branch: `master`
- Compose path: `logging/docker-compose.yaml`
- Required environment variable: `GRAFANA_ADMIN_PASSWORD`

Do not deploy the new stack before completing the cutover below: the current
containers still own names `loki` and `grafana` and host ports 3000, 3100, 514,
and 601.

## Host files

The Git-backed compose file intentionally bind-mounts stable host paths.
Before deployment, the repository files must be copied to:

```text
/mnt/containers/logging/
├── alloy-config/config.alloy
├── alloy-data/
├── grafana-config/
│   ├── dashboards/mikrotik-loki-logs.json
│   └── provisioning/
│       ├── dashboards/dashboards.yaml
│       └── datasources/loki.yaml
├── grafana-data/
├── loki-config/config.yaml
└── loki-data/
```

SELinux is enforcing. The compose mounts use `:Z`; do not remove those flags.
The persistent data directories should be owned by the container users:
Loki `10001:10001`, Grafana `472:0`, and Alloy `0:0`.

## Cutover

The live Loki history is inside the old container at `/tmp/loki`. A stopped,
consistent rescue copy from 2026-07-30 is pre-seeded in the new host path, but
it must be refreshed from the stopped live container immediately before the
old stack is removed:

```bash
cd /mnt/containers/mktxp/mktxp-stack

# Record a last pre-cutover query and stop all writers cleanly.
curl -fsS http://127.0.0.1:3100/ready
docker compose stop

# The stopped Loki container still contains the newest WAL/chunks.
docker cp -a loki:/tmp/loki/. /mnt/containers/logging/loki-data/
chown -R 10001:10001 /mnt/containers/logging/loki-data

# Removes containers and the old network, but NOT the old named Grafana and
# Prometheus volumes. Do not add --volumes.
docker compose down
```

Now deploy the Portainer Git stack. Expected interruption is the time between
`docker compose stop` and the new stack becoming ready.

## Verification

```bash
docker ps --filter name=loki --filter name=alloy --filter name=grafana
curl -fsS http://127.0.0.1:3100/ready
curl -fsS http://127.0.0.1:3000/api/health
ss -lnup | grep ':514'
ss -lntp | grep -E ':(601|3000|3100)\b'
```

Confirm both history and new ingestion:

```bash
START=$(date -u -d '24 hours ago' +%s%N)
curl -fsSG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="syslog"}' \
  --data-urlencode "start=$START" \
  --data-urlencode 'limit=1'

logger --server 127.0.0.1 --udp --port 514 \
  --rfc3164 --tag depot-cutover-test 'alloy ingestion test'
```

In Grafana, the provisioned **Network / Mikrotik Loki Logs** dashboard should
list existing `routerboard` values and show both pre-cutover and post-cutover
entries.

## Rollback

Remove the new Portainer stack, then recreate the old stack with its Loki data
bound back into `/tmp/loki`:

```bash
cd /mnt/containers/mktxp/mktxp-stack
cp docker-compose.yaml docker-compose.rollback.yaml
```

Add this second volume to the `loki` service in
`docker-compose.rollback.yaml`:

```yaml
      - /mnt/containers/logging/loki-data:/tmp/loki:Z
```

Then run:

```bash
docker compose -f docker-compose.rollback.yaml up -d
curl -fsS http://127.0.0.1:3100/ready
```

Do not restore the old Loki without that bind mount: its original writable
layer is deleted when the old stack is taken down.