The portal blocks images whose top ICU category is human or machine for non-admins. With the 0.5 threshold, 103 such images would have lost their detection and become visible. Those categories are now migrated when they were the image's top category, regardless of score.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FAILED/OCR FAIL rows never disappear on their own, so counting all of them
made the alarm fire every 5 minutes forever. Only stuck entries and
failures from the last 24 h trigger it now; older ones stay in the DB for
tools/gundi_requeue.py.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Upload request path uses its own S3 client with short timeouts (5s/30s,
2 attempts) and no longer holds a DB row lock during the S3 call; S3/DB
failures answer 503 + Retry-After instead of 500 after minutes of waiting.
- Transient S3/DB errors in ingest/OCR no longer count against max_attempts
(previously ~15 min of outage turned images into FAILED); the sweep retries.
- The sweep no longer queues the same id twice (no pile-up during long outages).
- Backlog check logs [GUNDI][ALARM] for stuck/failed entries.
- tools/gundi_requeue.py resets FAILED/FAIL entries; unit tests for all of it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The poller crashed on every S3 or DB error (1162 restarts during the
fsn1 object storage degradation on 2026-09-21). Now:
- main loop catches errors per cycle, classifies them (S3 5xx/throttling,
connection/timeouts, psycopg.OperationalError = transient; everything
else logged with traceback) and retries with exponential backoff
(5 s .. 5 min, reset after success); db_init runs until it succeeds once
- explicit botocore config for the poller: connect/read timeouts, standard
retries (total attempts), s3v4, path addressing; S3(cfg) without a
config keeps the previous client for the Gundi path
- DB connect timeout + TCP keepalives
- one S3 client per cycle shared by all workers
- a batch stops after a transient S3/DB error (image stays ERROR as before)
- no busy loop when ready sets keep failing
- [HEARTBEAT] log line with counters, heartbeat file + Docker healthcheck
- tools/entrance_inventory.py: read-only listing of icu/entrance orphans
- unit tests with simulated S3/DB failures
Per-image transaction, status names/order, keys and paths are unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>