Make legacy poller resilient to S3/DB outages
The poller crashed on every S3 or DB error (1162 restarts during the fsn1 object storage degradation on 2026-09-21). Now: - main loop catches errors per cycle, classifies them (S3 5xx/throttling, connection/timeouts, psycopg.OperationalError = transient; everything else logged with traceback) and retries with exponential backoff (5 s .. 5 min, reset after success); db_init runs until it succeeds once - explicit botocore config for the poller: connect/read timeouts, standard retries (total attempts), s3v4, path addressing; S3(cfg) without a config keeps the previous client for the Gundi path - DB connect timeout + TCP keepalives - one S3 client per cycle shared by all workers - a batch stops after a transient S3/DB error (image stays ERROR as before) - no busy loop when ready sets keep failing - [HEARTBEAT] log line with counters, heartbeat file + Docker healthcheck - tools/entrance_inventory.py: read-only listing of icu/entrance orphans - unit tests with simulated S3/DB failures Per-image transaction, status names/order, keys and paths are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -33,6 +33,45 @@ docker compose up -d --build
|
||||
docker compose logs -f
|
||||
```
|
||||
|
||||
### Stoerungen und Lebenszeichen (Poller)
|
||||
|
||||
Fehler eines Zyklus (S3 5xx/Throttling, Netz, DB weg) beenden den Poller nicht:
|
||||
er loggt eine Zeile `[CYCLE_FAIL] kind=transient ...` und versucht es mit
|
||||
exponentiellem Backoff erneut (`backoff_initial_seconds` bis `backoff_max_seconds`,
|
||||
Reset nach Erfolg, dann `[RECOVERED]`). Unerwartete Fehler (`kind=unexpected`,
|
||||
z. B. AccessDenied oder Programmierfehler) werden zusaetzlich mit Traceback geloggt.
|
||||
Ein Bild, bei dem S3/DB wegbricht, bleibt wie bisher auf `ERROR`; der Rest des
|
||||
Batches wartet auf den naechsten Zyklus.
|
||||
|
||||
Alle `heartbeat_seconds` erscheint `[HEARTBEAT]` mit Zaehlern (Zyklen, Bilder,
|
||||
Fehler, letzter Erfolg). Der Docker-Healthcheck prueft, ob die Hauptschleife
|
||||
vorankommt (`/tmp/melesicumover.heartbeat` juenger als 15 min):
|
||||
|
||||
```bash
|
||||
docker inspect -f '{{.State.Health.Status}} restarts={{.RestartCount}}' melesicumover
|
||||
docker exec melesicumover cat /tmp/melesicumover.heartbeat
|
||||
```
|
||||
|
||||
S3-Timeouts/-Retries stehen unter `app.s3_*` in `config.yaml`.
|
||||
|
||||
### Eingang pruefen (verwaiste Objekte)
|
||||
|
||||
```bash
|
||||
docker compose run --rm --no-deps melesicumover python -m tools.entrance_inventory --db
|
||||
```
|
||||
|
||||
Listet `icu/entrance/` nach Typ (vollstaendig, JPG ohne JSON, JSON ohne JPG,
|
||||
nur Kategorien, fremd) und Alter, optional mit `import_job`-Status. Nur lesend.
|
||||
|
||||
### Tests
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
python -m unittest discover -s tests -v # oder: python -m pytest tests
|
||||
```
|
||||
|
||||
Simulierte S3-/DB-Stoerungen fuer Backoff und Fehlerbehandlung; weder Netz noch DB noetig.
|
||||
|
||||
## Gundi-Push-Pfad (`gundi-receiver`)
|
||||
|
||||
ICU pusht Events ueber eine Gundi-v2-kompatible API. Der Service `gundi-receiver`
|
||||
|
||||
Reference in New Issue
Block a user