Make legacy poller resilient to S3/DB outages

The poller crashed on every S3 or DB error (1162 restarts during the
fsn1 object storage degradation on 2026-09-21). Now:

- main loop catches errors per cycle, classifies them (S3 5xx/throttling,
  connection/timeouts, psycopg.OperationalError = transient; everything
  else logged with traceback) and retries with exponential backoff
  (5 s .. 5 min, reset after success); db_init runs until it succeeds once
- explicit botocore config for the poller: connect/read timeouts, standard
  retries (total attempts), s3v4, path addressing; S3(cfg) without a
  config keeps the previous client for the Gundi path
- DB connect timeout + TCP keepalives
- one S3 client per cycle shared by all workers
- a batch stops after a transient S3/DB error (image stays ERROR as before)
- no busy loop when ready sets keep failing
- [HEARTBEAT] log line with counters, heartbeat file + Docker healthcheck
- tools/entrance_inventory.py: read-only listing of icu/entrance orphans
- unit tests with simulated S3/DB failures

Per-image transaction, status names/order, keys and paths are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
dom
2026-09-21 16:56:55 +02:00
co-authored by Claude Opus 5
parent 2fc8dea10c
commit 0f574cccde
6 changed files with 1023 additions and 18 deletions
+8
View File
@@ -10,6 +10,14 @@ services:
environment:
CONFIG_YAML: /app/config.yaml
DEBUG: "0"
# Lebenszeichen: die Hauptschleife schreibt die Heartbeat-Datei bei jedem Zyklus
# und jedem Bild. Laenger als 15 min unveraendert = Schleife haengt (Backoff max. 5 min).
healthcheck:
test: ["CMD", "python", "-c", "import os, sys, time; sys.exit(0 if time.time() - os.path.getmtime('/tmp/melesicumover.heartbeat') < 900 else 1)"]
interval: 60s
timeout: 10s
retries: 3
start_period: 120s
# Kein Bind-Mount des Quellcodes mehr (der Code steckt im Image) und
# standardmaessig KEIN offener Debug-Port.
#