docker-infrastructure/logging/README.md

# Internal logging stack

Portainer Git stack for `outline` (`10.0.0.6`). It replaces only the logging
part of the manual `mktxp-stack`:

- `loki` stores syslog in the persistent host path
  `/mnt/containers/logging/loki-data`;
- `syslog-ng` accepts and normalizes the fleet's mixed/raw syslog formats;
- `alloy` replaces Promtail and sends normalized RFC5424 to Loki;
- `grafana` provides the MikroTik Loki dashboard and a Telemt fleet dashboard
  backed by the internal DEPOT Zabbix server.

The retired stack's `mktxp`, Prometheus, and Promtail services are not part of
this stack. syslog-ng remains because real DEPOT RouterOS traffic includes raw
messages that Alloy's strict RFC3164 parser rejects.

## Portainer settings

- Repository: `https://git.it-depot.ru/depot/docker-infrastructure`
- Branch: `master`
- Compose path: `logging/docker-compose.yaml`
- Required environment variables:
  - `GRAFANA_ADMIN_PASSWORD`
  - `ZABBIX_API_TOKEN` — token for a read-only Zabbix API user; it is passed
    into Grafana's provisioned datasource and must not be committed.

Do not deploy the new stack before completing the cutover below: the current
containers still own names `loki` and `grafana` and host ports 3000, 3100, 514,
and 601.

## Host files

The Git-backed compose file intentionally bind-mounts stable host paths.
Before deployment, the repository files must be copied to:

```text
/mnt/containers/logging/
├── alloy-config/config.alloy
├── alloy-data/
├── grafana-config/
│   ├── dashboards/mikrotik-loki-logs.json
│   ├── dashboards/telemt-proxy-fleet.json
│   └── provisioning/
│       ├── dashboards/dashboards.yaml
│       └── datasources/{loki,zabbix}.yaml
├── grafana-data/
├── loki-config/config.yaml
├── loki-data/
└── syslog-ng-config/syslog-ng.conf
```

SELinux is enforcing. The compose mounts use `:Z`; do not remove those flags.
The persistent data directories should be owned by the container users:
Loki `10001:10001`, Grafana `472:0`, and Alloy `0:0`.

## Cutover

The live Loki history is inside the old container at `/tmp/loki`. Do not merge
a newer WAL over an older rescue WAL: the resulting non-contiguous segment
sequence makes Loki fail with `segments are not sequential`. Instead, move the
seed aside and copy the stopped container's complete data directory as one
generation:

```bash
cd /mnt/containers/mktxp/mktxp-stack

# Record a last pre-cutover query and stop all writers cleanly.
curl -fsS http://127.0.0.1:3100/ready
docker compose stop

# Preserve the seed separately; never overlay WAL generations.
mv /mnt/containers/logging/loki-data \
  /mnt/containers/logging/loki-data-rescue-seed-2026-07-30
docker cp -a loki:/tmp/loki /mnt/containers/logging/loki-data
chown -R 10001:10001 /mnt/containers/logging/loki-data

# Removes containers and the old network, but NOT the old named Grafana and
# Prometheus volumes. Do not add --volumes.
docker compose down
```

Now deploy the Portainer Git stack. Expected interruption is the time between
`docker compose stop` and the new stack becoming ready.

## Verification

```bash
docker ps --filter name=loki --filter name=alloy --filter name=grafana
curl -fsS http://127.0.0.1:3100/ready
curl -fsS http://127.0.0.1:3000/api/health
ss -lnup | grep ':514'
ss -lntp | grep -E ':(601|3000|3100)\b'
```

Confirm both history and new ingestion:

```bash
START=$(date -u -d '24 hours ago' +%s%N)
curl -fsSG http://127.0.0.1:3100/loki/api/v1/query_range \
  --data-urlencode 'query={job="syslog"}' \
  --data-urlencode "start=$START" \
  --data-urlencode 'limit=1'

logger --server 127.0.0.1 --udp --port 514 \
  --rfc3164 --tag depot-cutover-test 'alloy ingestion test'
```

In Grafana, the provisioned **Network / Mikrotik Loki Logs** dashboard should
list existing `routerboard` values and show both pre-cutover and post-cutover
entries.

## Telemt / Zabbix monitoring

The Zabbix server is `10.0.0.11` internally. The DEPOT router at `10.0.0.1`
publishes TCP `185.108.4.158:10051` only for the three proxy-host egress
addresses, allowing Zabbix Agent 2 active checks without exposing the trapper
to the whole internet.

Create the following Zabbix hosts with these exact technical **Host name**
values (visible names may be friendlier):

| Client | Zabbix Host name |
|---|---|
| 00-xserver | `00-xserver-fedora-xxx` |
| 11-nevafud | `11-nevafud-fedora` |
| 19-nutrilent | `19-nutrilent-docker` |

Import `zabbix-template-telemt-active.yaml`, link **Telemt proxy by active
agent** to all three hosts, and ensure each host is in the `Linux servers`
group. Do not add a passive-agent interface: these hosts are behind NAT and
send active checks to `185.108.4.158`.

The matching host-side collector is in `zabbix-agent/`. It enters only the
Telemt container network namespace for the control API, discards links and
secrets, caches the active-IP list and Prometheus values, and exposes
read-only Agent 2 keys. The active-IP list retains one day of history; the
numeric count and other counters retain 31 days.

After redeploying Grafana with a valid `ZABBIX_API_TOKEN`, verify:

```bash
docker exec grafana grafana cli plugins ls
curl -fsS http://127.0.0.1:3000/api/health
```

The provisioned **Network / Telegram proxy fleet** dashboard selects one of
the three exact host names above and mirrors the Telemt section of the
client-02 dashboard: current IP count, active/total connections, IP list,
traffic, and uptime.

## Rollback

Remove the new Portainer stack, then recreate the old stack with its Loki data
bound back into `/tmp/loki`:

```bash
cd /mnt/containers/mktxp/mktxp-stack
cp docker-compose.yaml docker-compose.rollback.yaml
```

Add this second volume to the `loki` service in
`docker-compose.rollback.yaml`:

```yaml
      - /mnt/containers/logging/loki-data:/tmp/loki:Z
```

Then run:

```bash
docker compose -f docker-compose.rollback.yaml up -d
curl -fsS http://127.0.0.1:3100/ready
```

Do not restore the old Loki without that bind mount: its original writable
layer is deleted when the old stack is taken down.