Make the Gundi receiver robust against S3/DB outages

- Upload request path uses its own S3 client with short timeouts (5s/30s,
  2 attempts) and no longer holds a DB row lock during the S3 call; S3/DB
  failures answer 503 + Retry-After instead of 500 after minutes of waiting.
- Transient S3/DB errors in ingest/OCR no longer count against max_attempts
  (previously ~15 min of outage turned images into FAILED); the sweep retries.
- The sweep no longer queues the same id twice (no pile-up during long outages).
- Backlog check logs [GUNDI][ALARM] for stuck/failed entries.
- tools/gundi_requeue.py resets FAILED/FAIL entries; unit tests for all of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dom
2026-10-05 16:40:48 +02:00
co-authored by Claude Opus 5.5
parent cb2566a537
commit 6d518b4542
7 changed files with 659 additions and 21 deletions
+6
View File
@@ -65,3 +65,9 @@ gundi:
sweep_seconds: 60
stale_after_seconds: 300
max_attempts: 3
# S3-Ausfall-Robustheit (siehe README): Upload-Request scheitert schnell mit 503 + Retry-After;
# S3-/DB-Stoerungen zaehlen nicht gegen max_attempts (nur Datenfehler).
upload_connect_timeout_seconds: 5
upload_read_timeout_seconds: 30
upload_max_attempts: 2 # inkl. erstem Versuch
alarm_after_seconds: 1800 # [GUNDI][ALARM], wenn Eintraege laenger haengen