Make legacy poller resilient to S3/DB outages
The poller crashed on every S3 or DB error (1162 restarts during the fsn1 object storage degradation on 2026-09-21). Now: - main loop catches errors per cycle, classifies them (S3 5xx/throttling, connection/timeouts, psycopg.OperationalError = transient; everything else logged with traceback) and retries with exponential backoff (5 s .. 5 min, reset after success); db_init runs until it succeeds once - explicit botocore config for the poller: connect/read timeouts, standard retries (total attempts), s3v4, path addressing; S3(cfg) without a config keeps the previous client for the Gundi path - DB connect timeout + TCP keepalives - one S3 client per cycle shared by all workers - a batch stops after a transient S3/DB error (image stays ERROR as before) - no busy loop when ready sets keep failing - [HEARTBEAT] log line with counters, heartbeat file + Docker healthcheck - tools/entrance_inventory.py: read-only listing of icu/entrance orphans - unit tests with simulated S3/DB failures Per-image transaction, status names/order, keys and paths are unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -39,6 +39,15 @@ app:
|
||||
enable_deermapper_api: false
|
||||
deermapper_api_timeout_seconds: 20
|
||||
deermapper_api_image_field: "image"
|
||||
# Robustheit (nur Poller): S3-Timeouts/Retries, Backoff bei Stoerungen, Lebenszeichen
|
||||
s3_connect_timeout_seconds: 10
|
||||
s3_read_timeout_seconds: 30
|
||||
s3_retry_mode: "standard" # standard | adaptive | legacy
|
||||
s3_max_attempts: 4 # inkl. erstem Versuch
|
||||
pg_connect_timeout_seconds: 10
|
||||
backoff_initial_seconds: 5 # 5 s, 10 s, 20 s ... nach fehlgeschlagenem Zyklus
|
||||
backoff_max_seconds: 300 # hoechstens 5 min; Reset nach erfolgreichem Zyklus
|
||||
heartbeat_seconds: 300 # [HEARTBEAT]-Logzeile mit Zaehlern
|
||||
|
||||
# Gundi-v2-Push-Empfaenger (Service gundi-receiver).
|
||||
# api key -> via ENV (.env): GUNDI_API_KEY (min. 24 Zeichen)
|
||||
|
||||
Reference in New Issue
Block a user