pipistrello
f0889261b2
The Portainer stack consumes its compose file from Git but bind-mounts runtime configuration from stable paths under /mnt/containers/logging. The stable host copy of docker-compose.yaml had consequently fallen behind the deployed Git revision even though the running stack and dashboard files were current.\n\nAdd a root-run staging/check helper that copies and byte-verifies Compose, Loki, Alloy, syslog-ng, Grafana provisioning, and both dashboards without touching credentials or persistent data. Document the workflow and the final source-restricted passive Agent 2 interfaces for all four Telemt hosts.\n\nValidated with bash -n and git diff --check. Unrelated .DS_Store changes remain untracked/uncommitted.
188 lines
6.5 KiB
Markdown
188 lines
6.5 KiB
Markdown
# Internal logging stack
|
|
|
|
Portainer Git stack for `outline` (`10.0.0.6`). It replaces only the logging
|
|
part of the manual `mktxp-stack`:
|
|
|
|
- `loki` stores syslog in the persistent host path
|
|
`/mnt/containers/logging/loki-data`;
|
|
- `syslog-ng` accepts and normalizes the fleet's mixed/raw syslog formats;
|
|
- `alloy` replaces Promtail and sends normalized RFC5424 to Loki;
|
|
- `grafana` provides the MikroTik Loki dashboard and a Telemt fleet dashboard
|
|
backed by the internal DEPOT Zabbix server.
|
|
|
|
The retired stack's `mktxp`, Prometheus, and Promtail services are not part of
|
|
this stack. syslog-ng remains because real DEPOT RouterOS traffic includes raw
|
|
messages that Alloy's strict RFC3164 parser rejects.
|
|
|
|
## Portainer settings
|
|
|
|
- Repository: `https://git.it-depot.ru/depot/docker-infrastructure`
|
|
- Branch: `master`
|
|
- Compose path: `logging/docker-compose.yaml`
|
|
- Required environment variables:
|
|
- `GRAFANA_ADMIN_PASSWORD`
|
|
- `ZABBIX_API_TOKEN` — token for a read-only Zabbix API user; it is passed
|
|
into Grafana's provisioned datasource and must not be committed.
|
|
|
|
Do not deploy the new stack before completing the cutover below: the current
|
|
containers still own names `loki` and `grafana` and host ports 3000, 3100, 514,
|
|
and 601.
|
|
|
|
## Host files
|
|
|
|
The Git-backed compose file intentionally bind-mounts stable host paths.
|
|
Before deployment, stage and verify every tracked runtime file with:
|
|
|
|
```bash
|
|
sudo logging/stage-host-files.sh
|
|
sudo logging/stage-host-files.sh --check
|
|
```
|
|
|
|
The script copies only non-secret configuration and dashboard files. Portainer
|
|
still supplies `GRAFANA_ADMIN_PASSWORD` and `ZABBIX_API_TOKEN` out of band.
|
|
The resulting layout is:
|
|
|
|
```text
|
|
/mnt/containers/logging/
|
|
├── alloy-config/config.alloy
|
|
├── alloy-data/
|
|
├── grafana-config/
|
|
│ ├── dashboards/mikrotik-loki-logs.json
|
|
│ ├── dashboards/telemt-proxy-fleet.json
|
|
│ └── provisioning/
|
|
│ ├── dashboards/dashboards.yaml
|
|
│ └── datasources/{loki,zabbix}.yaml
|
|
├── grafana-data/
|
|
├── loki-config/config.yaml
|
|
├── loki-data/
|
|
└── syslog-ng-config/syslog-ng.conf
|
|
```
|
|
|
|
SELinux is enforcing. The compose mounts use `:Z`; do not remove those flags.
|
|
The persistent data directories should be owned by the container users:
|
|
Loki `10001:10001`, Grafana `472:0`, and Alloy `0:0`.
|
|
|
|
## Cutover
|
|
|
|
The live Loki history is inside the old container at `/tmp/loki`. Do not merge
|
|
a newer WAL over an older rescue WAL: the resulting non-contiguous segment
|
|
sequence makes Loki fail with `segments are not sequential`. Instead, move the
|
|
seed aside and copy the stopped container's complete data directory as one
|
|
generation:
|
|
|
|
```bash
|
|
cd /mnt/containers/mktxp/mktxp-stack
|
|
|
|
# Record a last pre-cutover query and stop all writers cleanly.
|
|
curl -fsS http://127.0.0.1:3100/ready
|
|
docker compose stop
|
|
|
|
# Preserve the seed separately; never overlay WAL generations.
|
|
mv /mnt/containers/logging/loki-data \
|
|
/mnt/containers/logging/loki-data-rescue-seed-2026-07-30
|
|
docker cp -a loki:/tmp/loki /mnt/containers/logging/loki-data
|
|
chown -R 10001:10001 /mnt/containers/logging/loki-data
|
|
|
|
# Removes containers and the old network, but NOT the old named Grafana and
|
|
# Prometheus volumes. Do not add --volumes.
|
|
docker compose down
|
|
```
|
|
|
|
Now deploy the Portainer Git stack. Expected interruption is the time between
|
|
`docker compose stop` and the new stack becoming ready.
|
|
|
|
## Verification
|
|
|
|
```bash
|
|
docker ps --filter name=loki --filter name=alloy --filter name=grafana
|
|
curl -fsS http://127.0.0.1:3100/ready
|
|
curl -fsS http://127.0.0.1:3000/api/health
|
|
ss -lnup | grep ':514'
|
|
ss -lntp | grep -E ':(601|3000|3100)\b'
|
|
```
|
|
|
|
Confirm both history and new ingestion:
|
|
|
|
```bash
|
|
START=$(date -u -d '24 hours ago' +%s%N)
|
|
curl -fsSG http://127.0.0.1:3100/loki/api/v1/query_range \
|
|
--data-urlencode 'query={job="syslog"}' \
|
|
--data-urlencode "start=$START" \
|
|
--data-urlencode 'limit=1'
|
|
|
|
logger --server 127.0.0.1 --udp --port 514 \
|
|
--rfc3164 --tag depot-cutover-test 'alloy ingestion test'
|
|
```
|
|
|
|
In Grafana, the provisioned **Network / Mikrotik Loki Logs** dashboard should
|
|
list existing `routerboard` values and show both pre-cutover and post-cutover
|
|
entries.
|
|
|
|
## Telemt / Zabbix monitoring
|
|
|
|
The Zabbix server is `10.0.0.11` internally. The DEPOT router at `10.0.0.1`
|
|
publishes TCP `185.108.4.158:10051` only for the four proxy-host egress
|
|
addresses, allowing Zabbix Agent 2 active checks without exposing the trapper
|
|
to the whole internet.
|
|
|
|
Create the following Zabbix hosts. The technical **Host name** must match the
|
|
Agent 2 `Hostname`; Grafana selects the **Visible name** shown below.
|
|
|
|
| Client | Zabbix Host name | Visible name |
|
|
|---|---|---|
|
|
| 00-xserver | `00-xserver-fedora-xxx` | `00-xserver-fedora-xxx` |
|
|
| 02-tekhnohim | `Fedora Kirochnaya` | `02-tekhnohim-docker` |
|
|
| 11-nevafud | `11-nevafud-fedora` | `11-nevafud-fedora` |
|
|
| 19-nutrilent | `19-nutrilent-docker` | `19-nutrilent-docker` |
|
|
|
|
Import `zabbix-template-telemt-active.yaml`, link **Telemt proxy by active
|
|
agent** to all four hosts, and ensure each host is in the `Linux servers`
|
|
group. All four send active checks to `185.108.4.158` and have source-restricted
|
|
passive Agent 2 interfaces on TCP `10050`. Client 02's existing local Zabbix
|
|
server continues to perform its original passive Linux checks independently.
|
|
|
|
The matching host-side collector is in `zabbix-agent/`. It enters only the
|
|
Telemt container network namespace for the control API, discards links and
|
|
secrets, caches the active-IP list and Prometheus values, and exposes
|
|
read-only Agent 2 keys. The active-IP list retains one day of history; the
|
|
numeric count and other counters retain 31 days.
|
|
|
|
After redeploying Grafana with a valid `ZABBIX_API_TOKEN`, verify:
|
|
|
|
```bash
|
|
docker exec grafana grafana cli plugins ls
|
|
curl -fsS http://127.0.0.1:3000/api/health
|
|
```
|
|
|
|
The provisioned **Network / Telegram proxy fleet** dashboard selects one of
|
|
the four visible host names above and mirrors the Telemt section of the
|
|
client-02 dashboard: current IP count, active/total connections, IP list,
|
|
traffic, and uptime.
|
|
|
|
## Rollback
|
|
|
|
Remove the new Portainer stack, then recreate the old stack with its Loki data
|
|
bound back into `/tmp/loki`:
|
|
|
|
```bash
|
|
cd /mnt/containers/mktxp/mktxp-stack
|
|
cp docker-compose.yaml docker-compose.rollback.yaml
|
|
```
|
|
|
|
Add this second volume to the `loki` service in
|
|
`docker-compose.rollback.yaml`:
|
|
|
|
```yaml
|
|
- /mnt/containers/logging/loki-data:/tmp/loki:Z
|
|
```
|
|
|
|
Then run:
|
|
|
|
```bash
|
|
docker compose -f docker-compose.rollback.yaml up -d
|
|
curl -fsS http://127.0.0.1:3100/ready
|
|
```
|
|
|
|
Do not restore the old Loki without that bind mount: its original writable
|
|
layer is deleted when the old stack is taken down.
|