Commit Graph
22 Commits
Author SHA1 Message Date
domandClaude Opus 5.5 39b22f0124 Keep top human/machine legacy categories below 0.5 for privacy blocking
The portal blocks images whose top ICU category is human or machine for non-admins. With the 0.5 threshold, 103 such images would have lost their detection and become visible. Those categories are now migrated when they were the image's top category, regardless of score.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 22:15:12 +02:00
domandClaude Opus 5.5 ed105fbff8 Prepare legacy migration for the portal cutover
- detection from legacy category only for scores >= 0.5, without ''/none
- category_label + view detection_labeled unify ICU and legacy labels
- schema/003 + migrate_legacy --dedupe removes Gundi twins of legacy images
  (same imei, same size, <= 10 min), archived in image_superseded
- migrate_legacy --plan: read-only preview of the migration
- Gundi S3 prefix icu/gundi-test/ -> icu/gundi/

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 21:01:17 +02:00
domandClaude Opus 5.5 e7e5b6fb87 Backlog alarm: only count new failures (last 24h)
FAILED/OCR FAIL rows never disappear on their own, so counting all of them
made the alarm fire every 5 minutes forever. Only stuck entries and
failures from the last 24 h trigger it now; older ones stay in the DB for
tools/gundi_requeue.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-07 20:04:46 +02:00
domandClaude Opus 5.5 6d518b4542 Make the Gundi receiver robust against S3/DB outages
- Upload request path uses its own S3 client with short timeouts (5s/30s,
  2 attempts) and no longer holds a DB row lock during the S3 call; S3/DB
  failures answer 503 + Retry-After instead of 500 after minutes of waiting.
- Transient S3/DB errors in ingest/OCR no longer count against max_attempts
  (previously ~15 min of outage turned images into FAILED); the sweep retries.
- The sweep no longer queues the same id twice (no pile-up during long outages).
- Backlog check logs [GUNDI][ALARM] for stuck/failed entries.
- tools/gundi_requeue.py resets FAILED/FAIL entries; unit tests for all of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 16:40:48 +02:00
domandClaude Opus 5.5 cb2566a537 Accept icu_cameratrap_rep as camera trap image event type
ICU sends "icu_cameratrap_rep" instead of the documented Gundi/EarthRanger
type "cameratrap_rep". Events of the unknown type were stored as IGNORED, so
the following attachment upload got 404 and no image was ingested. Treat both
names as image events.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 15:55:37 +02:00
domandClaude Opus 5 0f574cccde Make legacy poller resilient to S3/DB outages
The poller crashed on every S3 or DB error (1162 restarts during the
fsn1 object storage degradation on 2026-09-21). Now:

- main loop catches errors per cycle, classifies them (S3 5xx/throttling,
  connection/timeouts, psycopg.OperationalError = transient; everything
  else logged with traceback) and retries with exponential backoff
  (5 s .. 5 min, reset after success); db_init runs until it succeeds once
- explicit botocore config for the poller: connect/read timeouts, standard
  retries (total attempts), s3v4, path addressing; S3(cfg) without a
  config keeps the previous client for the Gundi path
- DB connect timeout + TCP keepalives
- one S3 client per cycle shared by all workers
- a batch stops after a transient S3/DB error (image stays ERROR as before)
- no busy loop when ready sets keep failing
- [HEARTBEAT] log line with counters, heartbeat file + Docker healthcheck
- tools/entrance_inventory.py: read-only listing of icu/entrance orphans
- unit tests with simulated S3/DB failures

Per-image transaction, status names/order, keys and paths are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-21 16:56:55 +02:00
domandClaude Opus 5 2fc8dea10c Add Gundi v2 push receiver for ICU and normalized data model
ICU now pushes camera events via a Gundi-v2-compatible API instead of
dropping file triplets into S3. The new gundi-receiver service accepts
them directly, so images are in the system within seconds instead of
waiting out the 90s age gate + poll interval. OCR of the burnt-in camera
timestamp stays mandatory (ICU only sends server times; OCR detects
drifting camera clocks) but now runs asynchronously after ingest.

The push path writes a normalized model that separates what the legacy
metadata table conflates: camera_status (status time series), image
(image properties) and detection (classifications), plus the
image_with_status view as the portal's migration target. A repeatable,
batched migration splits existing metadata/category rows into it.

The legacy poller is unchanged and keeps writing metadata/category until
the portal is switched over.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 13:54:21 +00:00
domandClaude Sonnet 4.6 94bfc60fc4 Fix db_init for non-owner DB users
ALTER TABLE, CREATE INDEX and CREATE OR REPLACE FUNCTION all require table/function
ownership in PostgreSQL. Replace IF NOT EXISTS DDL with explicit existence checks so
db_init works for unprivileged application users where the schema was created by a
different role (e.g. postgres).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-14 08:58:46 +00:00
domandClaude Opus 4.8 88ef0d6943 Remove hardcoded credentials, harden deployment, optimize OCR
Secrets (S3 keys, PG password, DeerMapper API key) were committed in
config.yaml and .env and remain in git history. This removes them from
the tracked tree and moves all secrets to env injection.

Security:
- config.yaml: drop all credentials, keep only non-secret app tunables
- untrack .env, add .env.example template; .gitignore excludes .env
- main.py: tolerant config lookups + fail-fast validation for missing secrets
- docker-compose: env_file injection, no full-repo bind mount, debug port off
- Dockerfile: bake config into image, run as non-root user

Efficiency:
- OCR: run the second (expensive) tesseract pass only when the first
  is unparsable; identical fallback behavior

Docs:
- README with operation + security notes
- MIGRATION.md runbook: secret rotation, server cutover, decommission,
  git history purge

Note: the leaked secrets are compromised and MUST be rotated; removing
them from the tree is not sufficient. See MIGRATION.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 21:35:04 +02:00
dom 2e9ee3e014 Enhance configuration and processing with DeerMapper API integration, adjustable parameters, and improved error handling 2026-02-15 22:39:20 +00:00
dom 2294962d57 Add job completion check and cleanup for duplicate entrance sets 2026-02-14 12:17:06 +00:00
dom ddfb281be3 Remove Shiny app and associated Docker configuration 2026-02-09 11:08:13 +00:00
dom b35394c569 Add environment configuration and enhance Shiny app with AWS integration 2026-02-08 14:04:42 +00:00
dom 3873cb7b3f Enhance OCR timestamp extraction with flexible regex matching and improved error handling 2026-02-07 20:07:02 +00:00
dom 1e4a7b86b6 Refactor environment variables and remove unused launch configuration 2026-02-07 19:43:14 +00:00
dom 400f8a53f8 updated yaml 2026-02-06 15:38:22 +00:00
dom d71e1699d6 dsaf 2026-02-06 15:33:16 +00:00
dom 8414e72870 asdf 2026-02-06 15:31:21 +00:00
dom 6b39886745 awef 2026-02-06 15:15:48 +00:00
dom 2ce587b1e0 again 2026-02-06 14:38:43 +00:00
dom b9ad76f103 asedf 2026-02-06 14:34:33 +00:00
dom c923a12c7c debuggen im container 2026-02-06 14:05:27 +00:00