Add Gundi v2 push receiver for ICU and normalized data model

ICU now pushes camera events via a Gundi-v2-compatible API instead of
dropping file triplets into S3. The new gundi-receiver service accepts
them directly, so images are in the system within seconds instead of
waiting out the 90s age gate + poll interval. OCR of the burnt-in camera
timestamp stays mandatory (ICU only sends server times; OCR detects
drifting camera clocks) but now runs asynchronously after ingest.

The push path writes a normalized model that separates what the legacy
metadata table conflates: camera_status (status time series), image
(image properties) and detection (classifications), plus the
image_with_status view as the portal's migration target. A repeatable,
batched migration splits existing metadata/category rows into it.

The legacy poller is unchanged and keeps writing metadata/category until
the portal is switched over.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
dom
2026-09-18 13:54:21 +00:00
co-authored by Claude Opus 5
parent 94bfc60fc4
commit 2fc8dea10c
13 changed files with 1638 additions and 11 deletions
+5
View File
@@ -20,3 +20,8 @@ PG_DSN=postgresql://USER:PASSWORT@HOST:PORT/meles?sslmode=require
# --- DeerMapper API (nur noetig, wenn app.enable_deermapper_api: true) --- # --- DeerMapper API (nur noetig, wenn app.enable_deermapper_api: true) ---
DEERMAPPER_API_URL=https://webapp.deermapper.net/api/icu DEERMAPPER_API_URL=https://webapp.deermapper.net/api/icu
DEERMAPPER_API_KEY=__NEUEN_API_KEY_EINTRAGEN__ DEERMAPPER_API_KEY=__NEUEN_API_KEY_EINTRAGEN__
# --- Gundi-Receiver (Push-Schnittstelle fuer ICU) ---
# Diesen Wert bekommt ICU als "apikey". Mind. 24 Zeichen, z.B.:
# python3 -c "import secrets; print(secrets.token_urlsafe(32))"
GUNDI_API_KEY=__NEUEN_API_KEY_EINTRAGEN__
+3 -1
View File
@@ -16,7 +16,9 @@ COPY requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir -r /app/requirements.txt RUN pip install --no-cache-dir -r /app/requirements.txt
# Code + nicht-geheime Konfiguration ins Image (Secrets kommen zur Laufzeit via ENV). # Code + nicht-geheime Konfiguration ins Image (Secrets kommen zur Laufzeit via ENV).
COPY main.py /app/main.py COPY main.py gundi_pipeline.py gundi_receiver.py /app/
COPY schema/ /app/schema/
COPY tools/ /app/tools/
COPY config.yaml /app/config.yaml COPY config.yaml /app/config.yaml
# Haertung: nicht als root laufen. # Haertung: nicht als root laufen.
+72 -2
View File
@@ -21,8 +21,8 @@ Secrets werden **ausschliesslich** ueber Umgebungsvariablen geladen; die Werte i
Vollstaendigkeit und bricht mit klarer Meldung ab, falls ein Secret fehlt. Vollstaendigkeit und bricht mit klarer Meldung ab, falls ein Secret fehlt.
Unterstuetzte Umgebungsvariablen: `S3_ENDPOINT`, `S3_BUCKET`, `S3_ACCESS_KEY`, Unterstuetzte Umgebungsvariablen: `S3_ENDPOINT`, `S3_BUCKET`, `S3_ACCESS_KEY`,
`S3_SECRET_KEY`, `PG_DSN`, `DEERMAPPER_API_URL`, `DEERMAPPER_API_KEY`, `CONFIG_YAML`, `S3_SECRET_KEY`, `PG_DSN`, `DEERMAPPER_API_URL`, `DEERMAPPER_API_KEY`, `GUNDI_API_KEY`,
`DEBUG`. `CONFIG_YAML`, `DEBUG`.
## Betrieb (Docker) ## Betrieb (Docker)
@@ -33,6 +33,76 @@ docker compose up -d --build
docker compose logs -f docker compose logs -f
``` ```
## Gundi-Push-Pfad (`gundi-receiver`)
ICU pusht Events ueber eine Gundi-v2-kompatible API. Der Service `gundi-receiver`
(gleiches Image, `gundi_receiver.py`) nimmt sie entgegen:
| Route | Zweck |
|-------|-------|
| `POST /v2/events/` | `camera_status_report` oder `cameratrap_rep` (JSON, Objekt oder Liste) |
| `POST /v2/events/<object_id>/attachments/` | JPEG zum `cameratrap_rep` (multipart/form-data) |
| `GET /healthz` | Healthcheck |
Auth ueber den Header `apikey` = `GUNDI_API_KEY` aus der `.env`.
Ablauf: Event und JPEG werden durable abgelegt und sofort quittiert. Stufe 1
(Sekunden) schreibt processed-JPEG + Thumbnail und `image` + `detection`. Stufe 2
liest danach asynchron per OCR den Kamera-Zeitstempel, setzt `image.ocr_ts` und
`DateTimeOriginal`. Ein Sweep holt haengengebliebene Schritte nach und markiert
Events ohne Bild nach `abandon_after_hours` als `ABANDONED`.
### Datenmodell (`schema/001_domain_model.sql`)
| Tabelle | Inhalt |
|---------|--------|
| `remote_cam.camera_status` | Statusupdates / Kameraeigenschaften (Zeitreihe je IMEI) |
| `remote_cam.image` | Bildeigenschaften inkl. `servertime`, `ocr_ts`, `ocr_state` |
| `remote_cam.detection` | Objektklassifikationen, `confidence` 0..1 |
| `remote_cam.image_with_status` | View: Bild + zum `servertime` gueltiger Status |
| `remote_cam.gundi_inbox` | Transportschicht: Rohevents + Zustand des Push-Pfads |
Das Schema legt der Receiver beim Start an. Die Legacy-Tabellen `metadata` /
`category` werden vom alten Poller weiter befuellt, bis das Portal umgestellt ist.
### Bestandsdaten migrieren
Split `metadata` -> `image` + `camera_status`, `category` -> `detection`.
Wiederholbar, batchweise, liest die Alttabellen nur:
```bash
docker compose run --rm --no-deps gundi-receiver python -m tools.migrate_legacy
```
Vor dem Portal-Cutover erneut ausfuehren, um zwischenzeitlich neue Bilder
nachzuziehen. `--verify-only` prueft nur die Zaehlungen.
### Test
```bash
docker compose up -d --build gundi-receiver
docker compose run --rm --no-deps gundi-receiver \
python -m tools.gundi_smoke_test --url http://127.0.0.1:8090 --jpeg /pfad/zu/bild.jpg
```
Standardmaessig schreibt der Pfad unter `icu/gundi-test/` (siehe `gundi:` in
`config.yaml`); fuer den Produktivbetrieb `processed_prefix` / `thumb_prefix` auf
`icu/processed/` / `icu/thumbnails/` umstellen.
### nginx (`icu.meles.eu`)
vhost liegt unter [`deploy/nginx/icu.meles.eu`](deploy/nginx/icu.meles.eu):
```bash
sudo cp deploy/nginx/icu.meles.eu /etc/nginx/sites-available/
sudo ln -s /etc/nginx/sites-available/icu.meles.eu /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
sudo certbot --nginx -d icu.meles.eu --redirect
```
ICU-Endpunkt: `https://icu.meles.eu/v2/events/` (immer https — der 80->443-Redirect
ist ein 301 und wuerde POST in GET verwandeln).
## Sicherheitshinweise ## Sicherheitshinweise
- **Niemals** Secrets in `config.yaml` oder ins Git schreiben - nur in `.env`. - **Niemals** Secrets in `config.yaml` oder ins Git schreiben - nur in `.env`.
+17
View File
@@ -39,3 +39,20 @@ app:
enable_deermapper_api: false enable_deermapper_api: false
deermapper_api_timeout_seconds: 20 deermapper_api_timeout_seconds: 20
deermapper_api_image_field: "image" deermapper_api_image_field: "image"
# Gundi-v2-Push-Empfaenger (Service gundi-receiver).
# api key -> via ENV (.env): GUNDI_API_KEY (min. 24 Zeichen)
gundi:
enabled: true
bind_host: "0.0.0.0" # im Container; nach aussen nur 127.0.0.1:8090 (compose)
bind_port: 8090
inbox_prefix: "icu/gundi-test/inbox/"
processed_prefix: "icu/gundi-test/processed/" # spaeter: icu/processed/
thumb_prefix: "icu/gundi-test/thumbnails/" # spaeter: icu/thumbnails/
workers: 4
ocr_workers: 2
max_request_bytes: 33554432
abandon_after_hours: 24
sweep_seconds: 60
stale_after_seconds: 300
max_attempts: 3
+24
View File
@@ -0,0 +1,24 @@
# nginx-vhost fuer den Gundi-Receiver. Nur HTTP; TLS ergaenzt certbot --nginx.
# ICU muss direkt https:// verwenden: der 80->443-Redirect von certbot ist ein 301
# und wuerde POST in GET verwandeln.
server {
listen 80;
server_name icu.meles.eu;
client_max_body_size 32m; # = gundi.max_request_bytes
location /v2/events {
proxy_pass http://127.0.0.1:8090;
include /etc/nginx/proxy_params;
proxy_read_timeout 60s;
}
location = /healthz {
proxy_pass http://127.0.0.1:8090;
include /etc/nginx/proxy_params;
}
location / {
return 404;
}
}
+22
View File
@@ -18,3 +18,25 @@ services:
# 2) folgende Zeilen einkommentieren - bindet debugpy NUR an localhost: # 2) folgende Zeilen einkommentieren - bindet debugpy NUR an localhost:
# ports: # ports:
# - "127.0.0.1:5678:5678" # - "127.0.0.1:5678:5678"
# Gundi-v2-Push-Empfaenger. Gleiches Image, eigener Prozess: startet und
# stirbt unabhaengig vom Poller. Nur an localhost gebunden - TLS und
# oeffentliche Erreichbarkeit uebernimmt nginx (icu.meles.eu).
gundi-receiver:
build: .
container_name: melesicumover-gundi
restart: unless-stopped
command: ["python", "/app/gundi_receiver.py"]
env_file:
- .env
environment:
CONFIG_YAML: /app/config.yaml
DEBUG: "0"
ports:
- "127.0.0.1:8090:8090"
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8090/healthz', timeout=3)"]
interval: 30s
timeout: 5s
retries: 3
start_period: 20s
+646
View File
@@ -0,0 +1,646 @@
"""Push-Pfad (Gundi v2): Mapping ins Fachmodell, Ingest (Stufe 1), OCR (Stufe 2), Sweep."""
import os
import re
import subprocess
import tempfile
import threading
import time
import uuid as uuidlib
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional
import psycopg
import yaml
from dateutil import tz
from psycopg.types.json import Json
from main import (
AppConfig,
S3,
exif_write_with_exiftool,
make_thumbnail_bytes,
ocr_extract_timestamp,
to_exif_datetime,
)
SCHEMA_FILE = Path(__file__).resolve().parent / "schema" / "001_domain_model.sql"
SCHEMA_LOCK_KEY = 7_311_000_001
UUID_STRICT_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
EVENT_CAMERATRAP = "cameratrap_rep"
EVENT_STATUS = "camera_status_report"
MIN_API_KEY_LEN = 24
# -----------------------
# Config
# -----------------------
@dataclass
class GundiConfig:
enabled: bool
api_key: str
bind_host: str = "0.0.0.0"
bind_port: int = 8090
inbox_prefix: str = "icu/gundi-test/inbox/"
processed_prefix: str = "icu/gundi-test/processed/"
thumb_prefix: str = "icu/gundi-test/thumbnails/"
workers: int = 4
ocr_workers: int = 2
max_request_bytes: int = 32 * 1024 * 1024
abandon_after_hours: int = 24
sweep_seconds: int = 60
stale_after_seconds: int = 300
max_attempts: int = 3
def load_gundi_config(path: str) -> GundiConfig:
with open(path, "r", encoding="utf-8") as f:
raw = yaml.safe_load(f) or {}
g = raw.get("gundi") or {}
defaults = GundiConfig(enabled=False, api_key="")
def prefix(key: str) -> str:
value = str(g.get(key, getattr(defaults, key)))
return value if value.endswith("/") else value + "/"
gcfg = GundiConfig(
enabled=bool(g.get("enabled", False)),
api_key=os.environ.get("GUNDI_API_KEY", ""),
bind_host=str(g.get("bind_host", defaults.bind_host)),
bind_port=int(g.get("bind_port", defaults.bind_port)),
inbox_prefix=prefix("inbox_prefix"),
processed_prefix=prefix("processed_prefix"),
thumb_prefix=prefix("thumb_prefix"),
workers=max(1, int(g.get("workers", defaults.workers))),
ocr_workers=max(1, int(g.get("ocr_workers", defaults.ocr_workers))),
max_request_bytes=int(g.get("max_request_bytes", defaults.max_request_bytes)),
abandon_after_hours=int(g.get("abandon_after_hours", defaults.abandon_after_hours)),
sweep_seconds=int(g.get("sweep_seconds", defaults.sweep_seconds)),
stale_after_seconds=int(g.get("stale_after_seconds", defaults.stale_after_seconds)),
max_attempts=max(1, int(g.get("max_attempts", defaults.max_attempts))),
)
if gcfg.enabled and len(gcfg.api_key) < MIN_API_KEY_LEN:
raise SystemExit(
f"GUNDI_API_KEY fehlt oder ist kuerzer als {MIN_API_KEY_LEN} Zeichen "
"(gundi.enabled ist aktiv). Bitte in der .env setzen."
)
return gcfg
def connect(dsn: str) -> psycopg.Connection:
return psycopg.connect(dsn, connect_timeout=10)
def apply_schema(dsn: str) -> None:
sql = SCHEMA_FILE.read_text(encoding="utf-8")
with connect(dsn) as conn, conn.cursor() as cur:
cur.execute("SELECT pg_advisory_xact_lock(%s)", (SCHEMA_LOCK_KEY,))
cur.execute(sql)
def utc_now_str() -> str:
return datetime.now(timezone.utc).replace(microsecond=0).isoformat().replace("+00:00", "Z")
# -----------------------
# Validierung + Mapping
# -----------------------
class EventError(ValueError):
pass
def _num(v) -> Optional[float]:
if isinstance(v, bool) or not isinstance(v, (int, float)):
return None
return float(v)
def _int(v) -> Optional[int]:
if isinstance(v, bool):
return None
if isinstance(v, int):
return v
if isinstance(v, float) and v.is_integer():
return int(v)
if isinstance(v, str) and re.fullmatch(r"-?\d+", v.strip()):
return int(v.strip())
return None
def _bool(v) -> Optional[bool]:
if isinstance(v, bool):
return v
if isinstance(v, str) and v.strip().lower() in ("true", "false"):
return v.strip().lower() == "true"
return None
def _str(v) -> Optional[str]:
if v is None:
return None
s = str(v).strip()
return s or None
def parse_ts(value) -> datetime:
if not isinstance(value, str) or not value.strip():
raise EventError("recorded_at fehlt")
s = value.strip()
if s.endswith("Z"):
s = s[:-1] + "+00:00"
try:
dt = datetime.fromisoformat(s)
except ValueError:
raise EventError(f"recorded_at ungueltig: {value!r}") from None
if dt.tzinfo is None:
dt = dt.replace(tzinfo=timezone.utc)
return dt.astimezone(timezone.utc)
def coords(location) -> tuple[Optional[float], Optional[float]]:
"""ICU meldet unbekannte Koordinaten als 0.0/0.0 -> NULL."""
if not isinstance(location, dict):
return None, None
lat, lon = _num(location.get("lat")), _num(location.get("lon"))
if lat is None or lon is None or (lat == 0 and lon == 0):
return None, None
return lat, lon
def validate_event(ev) -> dict:
if not isinstance(ev, dict):
raise EventError("Event muss ein JSON-Objekt sein")
event_type = ev.get("event_type")
if not isinstance(event_type, str) or not event_type:
raise EventError("event_type fehlt")
source = ev.get("source")
if not isinstance(source, str) or not source.strip():
raise EventError("source fehlt")
details = ev.get("event_details")
if details is None:
details = {}
if not isinstance(details, dict):
raise EventError("event_details muss ein Objekt sein")
if event_type == EVENT_CAMERATRAP:
dets = details.get("detections", [])
if not isinstance(dets, list) or not all(isinstance(d, dict) for d in dets):
raise EventError("event_details.detections muss eine Liste von Objekten sein")
lat, lon = coords(ev.get("location"))
return {
"event_type": event_type,
"source": source.strip(),
"recorded_at": parse_ts(ev.get("recorded_at")),
"lat": lat,
"lon": lon,
"details": details,
}
def camera_status_row(env: dict, object_id: str, raw: dict) -> dict:
d = env["details"]
return {
"imei": env["source"],
"reported_at": env["recorded_at"],
"iccid": _str(d.get("iccid")),
"init_key": _str(d.get("init_key")),
"camera_name": _str(d.get("camera_name")),
"camera_model": _str(d.get("camera_model")),
"firmware_version": _str(d.get("firmware_version")),
"camera_language": _str(d.get("camera_language")),
"customer_timezone": _str(d.get("customer_timezone")),
"mcc": _str(d.get("mobile_country_code")),
"mnc": _str(d.get("mobile_network_code")),
"lac": _str(d.get("location_area_code")),
"cell_id": _str(d.get("cell_id")),
"battery_pct": _int(d.get("battery_status")),
"signal_strength": _int(d.get("signal_strength")),
"temperature_c": _int(d.get("temperature")),
"coordinate_switch": _bool(d.get("coordinate_switch")),
"work_period": _bool(d.get("work_period")),
"work_start": _str(d.get("work_start")),
"work_end": _str(d.get("work_end")),
"image_format": _str(d.get("image_format")),
"image_night": _str(d.get("image_night")),
"thumbnail_size": _str(d.get("thumbnail_size")),
"pir_interval": _str(d.get("pir_interval")),
"time_scan": _str(d.get("time_scan")),
"lat": env["lat"],
"lon": env["lon"],
"source": "gundi",
"source_ref": object_id,
"raw": Json(raw),
}
def detection_rows(env: dict, image_uuid: str) -> list[dict]:
rows = []
for det in env["details"].get("detections") or []:
category = _str(det.get("category"))
if category is None:
continue
pct = _num(det.get("confidence_percent"))
confidence = None if pct is None else min(max(pct / 100.0, 0.0), 1.0)
bbox = [_int(det.get(k)) for k in ("x_px", "y_px", "width_px", "height_px")]
if all(v in (None, 0) for v in bbox):
bbox = [None, None, None, None]
rows.append({
"image_uuid": image_uuid,
"category": category,
"confidence": confidence,
"bbox_x": bbox[0],
"bbox_y": bbox[1],
"bbox_w": bbox[2],
"bbox_h": bbox[3],
"model_version": "icu",
"source": "gundi",
})
return rows
def image_row(env: dict, object_id: str, raw: dict, size_bytes: int, processed_key: str, thumb_key: str) -> dict:
d = env["details"]
return {
"image_uuid": object_id,
"imei": env["source"],
"servertime": env["recorded_at"],
"ocr_state": "PENDING",
"camera_name": _str(d.get("cameratraprep_camera-name")),
"camera_make": _str(d.get("cameratraprep_camera-make")),
"lat": env["lat"],
"lon": env["lon"],
"size_bytes": size_bytes,
"processed_key": processed_key,
"thumbnail_key": thumb_key,
"source": "gundi",
"source_ref": object_id,
"raw": Json(raw),
}
def _insert(cur, table: str, row: dict, suffix: str = "") -> int:
cols = list(row)
cur.execute(
f"INSERT INTO {table} ({', '.join(cols)}) VALUES ({', '.join(['%s'] * len(cols))}) {suffix}",
[row[c] for c in cols],
)
return cur.rowcount
# -----------------------
# Receiver-seitige Persistenz (synchron im Request)
# -----------------------
def store_events(dsn: str, items: list[tuple[dict, dict]]) -> list[dict]:
"""Ein Request = eine Transaktion. Antwort erst nach Commit."""
results = []
with connect(dsn) as conn, conn.cursor() as cur:
for env, raw in items:
object_id = str(uuidlib.uuid4())
if env["event_type"] == EVENT_STATUS:
state = "INGESTED"
elif env["event_type"] == EVENT_CAMERATRAP:
state = "AWAITING_ATTACHMENT"
else:
state = "IGNORED"
_insert(cur, "remote_cam.gundi_inbox", {
"object_id": object_id,
"event_type": env["event_type"],
"source": env["source"],
"recorded_at": env["recorded_at"],
"payload": Json(raw),
"state": state,
})
if env["event_type"] == EVENT_STATUS:
_insert(
cur, "remote_cam.camera_status", camera_status_row(env, object_id, raw),
"ON CONFLICT (imei, reported_at) WHERE source = 'gundi' DO NOTHING",
)
results.append({
"object_id": object_id,
"created_at": utc_now_str(),
"event_type": env["event_type"],
"source": env["source"],
"state": state,
})
return results
def attach_image(dsn: str, s3: S3, gcfg: GundiConfig, object_id: str, jpg: bytes) -> str:
"""Legt das JPEG durable in der Inbox ab. Liefert 'stored' oder 'duplicate'."""
with connect(dsn) as conn, conn.cursor() as cur:
cur.execute(
"SELECT event_type, state FROM remote_cam.gundi_inbox WHERE object_id = %s FOR UPDATE",
(object_id,),
)
row = cur.fetchone()
if not row or row[0] != EVENT_CAMERATRAP:
raise LookupError(object_id)
if row[1] in ("RECEIVED", "INGESTED"):
return "duplicate"
key = f"{gcfg.inbox_prefix}{object_id}.jpg"
s3.put_bytes(key, jpg, "image/jpeg")
cur.execute(
"""
UPDATE remote_cam.gundi_inbox
SET state = 'RECEIVED', attachment_key = %s, attachment_received_at = now(),
attempts = 0, last_error = NULL, updated_at = now()
WHERE object_id = %s
""",
(key, object_id),
)
return "stored"
# -----------------------
# EXIF
# -----------------------
def exif_set_digitized(jpg: bytes, servertime: datetime, *, timeout_seconds: int) -> bytes:
"""Stufe 1: nur Serverzeit. DateTimeOriginal folgt nach dem OCR (Stufe 2)."""
dtd = to_exif_datetime(servertime.astimezone(timezone.utc))
with tempfile.TemporaryDirectory() as td:
path = os.path.join(td, "in.jpg")
with open(path, "wb") as f:
f.write(jpg)
subprocess.run(
[
"exiftool", "-overwrite_original",
f"-DateTimeDigitized={dtd}", "-OffsetTimeDigitized=+00:00",
f"-ModifyDate={dtd}", "-OffsetTime=+00:00",
path,
],
check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=timeout_seconds,
)
with open(path, "rb") as f:
return f.read()
# -----------------------
# Pipeline
# -----------------------
class Pipeline:
def __init__(self, cfg: AppConfig, gcfg: GundiConfig):
self.cfg = cfg
self.gcfg = gcfg
self.s3 = S3(cfg)
self._ingest_pool = ThreadPoolExecutor(gcfg.workers, thread_name_prefix="ingest")
self._ocr_pool = ThreadPoolExecutor(gcfg.ocr_workers, thread_name_prefix="ocr")
self._stop = threading.Event()
def start(self) -> None:
threading.Thread(target=self._sweep_loop, name="sweep", daemon=True).start()
def stop(self) -> None:
self._stop.set()
self._ingest_pool.shutdown(wait=False, cancel_futures=True)
self._ocr_pool.shutdown(wait=False, cancel_futures=True)
def submit_ingest(self, object_id: str) -> None:
self._ingest_pool.submit(self._guarded, self.ingest, object_id)
def submit_ocr(self, image_uuid: str) -> None:
self._ocr_pool.submit(self._guarded, self.ocr, image_uuid)
@staticmethod
def _guarded(fn, arg: str) -> None:
try:
fn(arg)
except Exception as e:
print(f"[GUNDI] {fn.__name__} {arg} unerwarteter Fehler: {type(e).__name__}: {e}", flush=True)
# --- Stufe 1 ---------------------------------------------------------
def ingest(self, object_id: str) -> None:
start = time.perf_counter()
try:
with connect(self.cfg.pg_dsn) as conn, conn.cursor() as cur:
cur.execute(
"""
SELECT payload, attachment_key FROM remote_cam.gundi_inbox
WHERE object_id = %s AND state = 'RECEIVED'
FOR UPDATE SKIP LOCKED
""",
(object_id,),
)
row = cur.fetchone()
if not row:
return
payload, attachment_key = row
env = validate_event(payload)
jpg = self.s3.get_bytes(attachment_key)
exif_note = "ok"
try:
processed = exif_set_digitized(
jpg, env["recorded_at"], timeout_seconds=self.cfg.exiftool_timeout_seconds
)
except Exception as e:
processed = jpg
exif_note = f"fail({type(e).__name__})"
thumb = make_thumbnail_bytes(jpg, self.cfg.thumb_max)
processed_key = f"{self.gcfg.processed_prefix}{object_id}.jpg"
thumb_key = f"{self.gcfg.thumb_prefix}{object_id}.jpg"
self.s3.put_bytes(processed_key, processed, "image/jpeg")
self.s3.put_bytes(thumb_key, thumb, "image/jpeg")
n_det = 0
inserted = _insert(
cur, "remote_cam.image",
image_row(env, object_id, payload, len(jpg), processed_key, thumb_key),
"ON CONFLICT (image_uuid) DO NOTHING",
)
if inserted:
for det in detection_rows(env, object_id):
n_det += _insert(cur, "remote_cam.detection", det)
cur.execute(
"""
UPDATE remote_cam.gundi_inbox
SET state = 'INGESTED', last_error = NULL, updated_at = now()
WHERE object_id = %s
""",
(object_id,),
)
except Exception as e:
self._record_ingest_failure(object_id, e)
return
try:
self.s3.delete_keys([attachment_key])
except Exception as e:
print(f"[GUNDI] inbox-cleanup {object_id} fehlgeschlagen: {e}", flush=True)
self.submit_ocr(object_id)
ms = int((time.perf_counter() - start) * 1000)
print(
f"[GUNDI] ingest object_id={object_id} imei={env['source']} detections={n_det} "
f"exif={exif_note} ms={ms}",
flush=True,
)
def _record_ingest_failure(self, object_id: str, err: Exception) -> None:
msg = f"{type(err).__name__}: {err}"[:1000]
with connect(self.cfg.pg_dsn) as conn, conn.cursor() as cur:
cur.execute(
"""
UPDATE remote_cam.gundi_inbox
SET attempts = attempts + 1,
state = CASE WHEN attempts + 1 >= %s THEN 'FAILED' ELSE state END,
last_error = %s, updated_at = now()
WHERE object_id = %s AND state = 'RECEIVED'
RETURNING attempts, state
""",
(self.gcfg.max_attempts, msg, object_id),
)
row = cur.fetchone()
if row:
print(f"[GUNDI] ingest object_id={object_id} FEHLER attempt={row[0]} state={row[1]} error={msg}", flush=True)
# --- Stufe 2 ---------------------------------------------------------
def ocr(self, image_uuid: str) -> None:
start = time.perf_counter()
try:
with connect(self.cfg.pg_dsn) as conn, conn.cursor() as cur:
cur.execute(
"""
SELECT imei, servertime, processed_key FROM remote_cam.image
WHERE image_uuid = %s AND ocr_state = 'PENDING'
FOR UPDATE SKIP LOCKED
""",
(image_uuid,),
)
row = cur.fetchone()
if not row:
return
imei, servertime, processed_key = row
tzinfo, tzname = self._camera_tz(cur, imei, servertime)
jpg = self.s3.get_bytes(processed_key)
ocr_ts = ocr_extract_timestamp(
jpg, self.cfg.ocr_crop_w_frac, self.cfg.ocr_crop_h_frac, tzinfo
)
exif_err = None
try:
patched = exif_write_with_exiftool(
jpg, ocr_ts, servertime, timeout_seconds=self.cfg.exiftool_timeout_seconds
)
self.s3.put_bytes(processed_key, patched, "image/jpeg")
except Exception as e:
exif_err = f"exif: {type(e).__name__}: {e}"[:1000]
cur.execute(
"""
UPDATE remote_cam.image
SET ocr_ts = %s, ocr_state = 'OK', ocr_attempts = ocr_attempts + 1,
ocr_error = %s, updated_at = now()
WHERE image_uuid = %s
""",
(ocr_ts, exif_err, image_uuid),
)
except Exception as e:
self._record_ocr_failure(image_uuid, e)
return
drift = int((servertime - ocr_ts).total_seconds())
ms = int((time.perf_counter() - start) * 1000)
print(
f"[GUNDI] ocr image_uuid={image_uuid} ocr_ts={ocr_ts.isoformat()} tz={tzname} "
f"servertime-ocr={drift}s exif={'ok' if exif_err is None else 'fail'} ms={ms}",
flush=True,
)
@staticmethod
def _camera_tz(cur, imei: str, servertime: datetime):
cur.execute(
"""
SELECT customer_timezone FROM remote_cam.camera_status
WHERE imei = %s AND customer_timezone IS NOT NULL AND reported_at <= %s
ORDER BY reported_at DESC LIMIT 1
""",
(imei, servertime),
)
row = cur.fetchone()
if not row:
cur.execute(
"""
SELECT customer_timezone FROM remote_cam.camera_status
WHERE imei = %s AND customer_timezone IS NOT NULL
ORDER BY reported_at ASC LIMIT 1
""",
(imei,),
)
row = cur.fetchone()
name = row[0] if row else "UTC"
zone = tz.gettz(name)
if zone is None:
return timezone.utc, "UTC"
return zone, name
def _record_ocr_failure(self, image_uuid: str, err: Exception) -> None:
msg = f"{type(err).__name__}: {err}"[:1000]
with connect(self.cfg.pg_dsn) as conn, conn.cursor() as cur:
cur.execute(
"""
UPDATE remote_cam.image
SET ocr_attempts = ocr_attempts + 1,
ocr_state = CASE WHEN ocr_attempts + 1 >= %s THEN 'FAIL' ELSE ocr_state END,
ocr_error = %s, updated_at = now()
WHERE image_uuid = %s AND ocr_state = 'PENDING'
RETURNING ocr_attempts, ocr_state
""",
(self.gcfg.max_attempts, msg, image_uuid),
)
row = cur.fetchone()
if row:
print(f"[GUNDI] ocr image_uuid={image_uuid} FEHLER attempt={row[0]} state={row[1]} error={msg}", flush=True)
# --- Sweep -----------------------------------------------------------
def _sweep_loop(self) -> None:
delay = 0
while not self._stop.wait(delay):
delay = self.gcfg.sweep_seconds
try:
self.sweep()
except Exception as e:
print(f"[GUNDI] sweep Fehler: {type(e).__name__}: {e}", flush=True)
def sweep(self) -> None:
stale = self.gcfg.stale_after_seconds
with connect(self.cfg.pg_dsn) as conn, conn.cursor() as cur:
cur.execute(
"""
SELECT object_id::text FROM remote_cam.gundi_inbox
WHERE state = 'RECEIVED' AND updated_at < now() - make_interval(secs => %s)
ORDER BY updated_at LIMIT 200
""",
(stale,),
)
ingest_ids = [r[0] for r in cur.fetchall()]
cur.execute(
"""
SELECT image_uuid::text FROM remote_cam.image
WHERE ocr_state = 'PENDING' AND source = 'gundi'
AND updated_at < now() - make_interval(secs => %s)
ORDER BY updated_at LIMIT 200
""",
(stale,),
)
ocr_ids = [r[0] for r in cur.fetchall()]
cur.execute(
"""
UPDATE remote_cam.gundi_inbox
SET state = 'ABANDONED', updated_at = now()
WHERE state = 'AWAITING_ATTACHMENT'
AND received_at < now() - make_interval(hours => %s)
""",
(self.gcfg.abandon_after_hours,),
)
abandoned = cur.rowcount
for object_id in ingest_ids:
self.submit_ingest(object_id)
for image_uuid in ocr_ids:
self.submit_ocr(image_uuid)
if ingest_ids or ocr_ids or abandoned:
print(
f"[GUNDI] sweep ingest_retry={len(ingest_ids)} ocr_retry={len(ocr_ids)} abandoned={abandoned}",
flush=True,
)
+252
View File
@@ -0,0 +1,252 @@
#!/usr/bin/env python3
"""Gundi-v2-kompatibler Empfaenger fuer ICU-Push-Events.
POST /v2/events/ -> {"object_id", "created_at"}
POST /v2/events/<object_id>/attachments/ -> {"object_id", "created_at"}
GET /healthz
"""
import hmac
import json
import os
import re
import signal
import threading
import time
from email.parser import BytesParser
from email.policy import default as email_policy
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import urlparse
from gundi_pipeline import (
UUID_STRICT_RE,
EventError,
GundiConfig,
Pipeline,
apply_schema,
attach_image,
load_gundi_config,
store_events,
utc_now_str,
validate_event,
)
from main import AppConfig, load_config
EVENTS_PATH = "/v2/events/"
ATTACHMENT_RE = re.compile(r"^/v2/events/([^/]+)/attachments(/?)$")
JPEG_MAGIC = b"\xff\xd8\xff"
def extract_jpeg(content_type: str, body: bytes) -> tuple[bytes | None, int]:
"""Erstes JPEG aus einem multipart/form-data-Body; Client-Dateinamen werden ignoriert."""
raw = f"Content-Type: {content_type}\r\nMIME-Version: 1.0\r\n\r\n".encode("latin-1") + body
msg = BytesParser(policy=email_policy).parsebytes(raw)
if not msg.is_multipart():
return None, 0
files = []
for part in msg.iter_parts():
if part.get_content_disposition() != "form-data" or not part.get_filename():
continue
files.append(part.get_payload(decode=True) or b"")
jpgs = [f for f in files if f.startswith(JPEG_MAGIC)]
return (jpgs[0] if jpgs else None), len(files)
class GundiServer(ThreadingHTTPServer):
daemon_threads = True
allow_reuse_address = True
def __init__(self, addr, cfg: AppConfig, gcfg: GundiConfig, pipeline: Pipeline):
super().__init__(addr, GundiHandler)
self.cfg = cfg
self.gcfg = gcfg
self.pipeline = pipeline
class GundiHandler(BaseHTTPRequestHandler):
server: GundiServer
server_version = "melesICUmover-gundi/1.0"
sys_version = ""
timeout = 60
def log_message(self, fmt, *args):
client = self.headers.get("X-Real-IP") if self.headers else None
print(f"[HTTP] {client or self.client_address[0]} {fmt % args}", flush=True)
# --- Antworten -------------------------------------------------------
def _json(self, code: int, data) -> None:
body = json.dumps(data, ensure_ascii=False).encode("utf-8")
self.send_response(code)
self.send_header("Content-Type", "application/json; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def _error(self, code: int, message: str, *, close: bool = False) -> None:
if close:
self.close_connection = True
self._json(code, {"error": message})
def _redirect(self, location: str) -> None:
# 308 statt 301: der Client muss POST + Body beibehalten.
self.close_connection = True
self.send_response(308)
self.send_header("Location", location)
self.send_header("Content-Length", "0")
self.end_headers()
# --- Request-Helfer --------------------------------------------------
def _authorized(self) -> bool:
got = (self.headers.get("apikey") or "").encode("utf-8")
return hmac.compare_digest(got, self.server.gcfg.api_key.encode("utf-8"))
def _read_body(self) -> bytes | None:
if self.headers.get("Transfer-Encoding"):
self._error(411, "Transfer-Encoding wird nicht unterstuetzt, Content-Length senden", close=True)
return None
raw_len = self.headers.get("Content-Length")
if raw_len is None:
self._error(411, "Content-Length erforderlich", close=True)
return None
try:
length = int(raw_len)
except ValueError:
self._error(400, "Content-Length ungueltig", close=True)
return None
if length < 0:
self._error(400, "Content-Length ungueltig", close=True)
return None
if length > self.server.gcfg.max_request_bytes:
self._error(413, "Request zu gross", close=True)
return None
body = self.rfile.read(length)
if len(body) != length:
self._error(400, "Body unvollstaendig", close=True)
return None
return body
# --- Routing ---------------------------------------------------------
def do_GET(self) -> None:
if urlparse(self.path).path == "/healthz":
self._json(200, {"status": "ok"})
else:
self._error(404, "not found")
def do_POST(self) -> None:
path = urlparse(self.path).path
if path == "/v2/events":
self._redirect(EVENTS_PATH)
return
attachment = ATTACHMENT_RE.match(path)
if attachment and not attachment.group(2):
self._redirect(path + "/")
return
if path != EVENTS_PATH and not attachment:
self._error(404, "not found", close=True)
return
if not self._authorized():
self._error(401, "Invalid or missing apikey", close=True)
return
body = self._read_body()
if body is None:
return
try:
if attachment:
self._handle_attachment(attachment.group(1).lower(), body)
else:
self._handle_events(body)
except Exception as e:
print(f"[GUNDI] Fehler {path}: {type(e).__name__}: {e}", flush=True)
self._error(500, "internal error", close=True)
# --- Handler ---------------------------------------------------------
def _handle_events(self, body: bytes) -> None:
if "application/json" not in (self.headers.get("Content-Type") or "").lower():
self._error(415, "Content-Type muss application/json sein")
return
try:
payload = json.loads(body.decode("utf-8"))
except (UnicodeDecodeError, json.JSONDecodeError):
self._error(400, "Ungueltiges JSON")
return
events = payload if isinstance(payload, list) else [payload]
if not events:
self._error(400, "Leere Event-Liste")
return
try:
envs = [validate_event(ev) for ev in events]
except EventError as e:
self._error(400, str(e))
return
results = store_events(self.server.cfg.pg_dsn, list(zip(envs, events)))
for r in results:
print(
f"[GUNDI] event object_id={r['object_id']} type={r['event_type']} "
f"imei={r['source']} state={r['state']}",
flush=True,
)
out = [{"object_id": r["object_id"], "created_at": r["created_at"]} for r in results]
self._json(200, out if isinstance(payload, list) else out[0])
def _handle_attachment(self, object_id: str, body: bytes) -> None:
if not UUID_STRICT_RE.match(object_id):
self._error(404, "Unknown object_id")
return
content_type = self.headers.get("Content-Type") or ""
if "multipart/form-data" not in content_type.lower():
self._error(415, "Content-Type muss multipart/form-data sein")
return
jpg, n_files = extract_jpeg(content_type, body)
if jpg is None:
self._error(400, "Kein JPEG-Anhang gefunden")
return
try:
outcome = attach_image(
self.server.cfg.pg_dsn, self.server.pipeline.s3, self.server.gcfg, object_id, jpg
)
except LookupError:
self._error(404, "Unknown object_id")
return
if outcome == "stored":
self.server.pipeline.submit_ingest(object_id)
print(
f"[GUNDI] attachment object_id={object_id} bytes={len(jpg)} files={n_files} outcome={outcome}",
flush=True,
)
self._json(200, {"object_id": object_id, "created_at": utc_now_str()})
def main() -> None:
cfg_path = os.environ.get("CONFIG_YAML", "/app/config.yaml")
cfg = load_config(cfg_path)
gcfg = load_gundi_config(cfg_path)
if not gcfg.enabled:
print("[GUNDI] gundi.enabled ist false - Receiver bleibt inaktiv.", flush=True)
while True:
time.sleep(3600)
apply_schema(cfg.pg_dsn)
pipeline = Pipeline(cfg, gcfg)
pipeline.start()
server = GundiServer((gcfg.bind_host, gcfg.bind_port), cfg, gcfg, pipeline)
def _shutdown(signum, frame):
threading.Thread(target=server.shutdown, daemon=True).start()
signal.signal(signal.SIGTERM, _shutdown)
signal.signal(signal.SIGINT, _shutdown)
print(
f"[GUNDI] lausche auf {gcfg.bind_host}:{gcfg.bind_port} "
f"inbox={gcfg.inbox_prefix} processed={gcfg.processed_prefix} thumbs={gcfg.thumb_prefix}",
flush=True,
)
try:
server.serve_forever()
finally:
server.server_close()
pipeline.stop()
if __name__ == "__main__":
main()
+6 -8
View File
@@ -25,14 +25,6 @@ import pytesseract
import yaml import yaml
if os.environ.get("DEBUG") == "1":
import debugpy
debugpy.listen(("0.0.0.0", 5678))
print("Waiting for debugger...")
debugpy.wait_for_client()
# ----------------------- # -----------------------
# Config # Config
# ----------------------- # -----------------------
@@ -1012,6 +1004,12 @@ def run_once(cfg: AppConfig) -> int:
def main(): def main():
if os.environ.get("DEBUG") == "1":
import debugpy
debugpy.listen(("0.0.0.0", 5678))
print("Waiting for debugger...")
debugpy.wait_for_client()
cfg_path = os.environ.get("CONFIG_YAML", "/app/config.yaml") cfg_path = os.environ.get("CONFIG_YAML", "/app/config.yaml")
cfg = load_config(cfg_path) cfg = load_config(cfg_path)
+179
View File
@@ -0,0 +1,179 @@
-- Normalisiertes Fachmodell: Kamerastatus, Bildeigenschaften, Objektklassifikationen.
-- Idempotent; wird vom gundi-receiver beim Start unter Advisory-Lock angewendet.
-- Setzt das bestehende Schema remote_cam voraus (die App-Rolle hat kein CREATE auf DB-Ebene).
-- Statusupdates / Kameraeigenschaften: eine Zeile je Statusmeldung (Zeitreihe).
CREATE TABLE IF NOT EXISTS remote_cam.camera_status (
status_id bigserial PRIMARY KEY,
imei text NOT NULL,
reported_at timestamptz NOT NULL,
received_at timestamptz NOT NULL DEFAULT now(),
iccid text,
init_key text,
camera_name text,
camera_model text,
firmware_version text,
camera_language text,
customer_timezone text,
mcc text,
mnc text,
lac text,
cell_id text,
camera_network text,
battery_pct integer,
signal_strength integer,
temperature_c integer,
coordinate_switch boolean,
work_period boolean,
work_start text,
work_end text,
image_format text,
image_night text,
thumbnail_size text,
pir_interval text,
time_scan text,
lat double precision,
lon double precision,
settings_changed_at timestamptz,
source text NOT NULL CHECK (source IN ('gundi', 'entrance')),
source_ref text,
raw jsonb
);
-- Legacy hat 173 doppelte (imei, servertime)-Paare, daher keine globale Eindeutigkeit.
CREATE UNIQUE INDEX IF NOT EXISTS camera_status_gundi_uq
ON remote_cam.camera_status (imei, reported_at) WHERE source = 'gundi';
CREATE UNIQUE INDEX IF NOT EXISTS camera_status_entrance_uq
ON remote_cam.camera_status (source_ref) WHERE source = 'entrance';
CREATE INDEX IF NOT EXISTS camera_status_imei_reported_idx
ON remote_cam.camera_status (imei, reported_at DESC);
-- Bildeigenschaften.
CREATE TABLE IF NOT EXISTS remote_cam.image (
image_uuid uuid PRIMARY KEY,
imei text NOT NULL,
servertime timestamptz NOT NULL,
ocr_ts timestamptz,
ocr_state text NOT NULL DEFAULT 'PENDING' CHECK (ocr_state IN ('PENDING', 'OK', 'FAIL')),
ocr_attempts integer NOT NULL DEFAULT 0,
ocr_error text,
camera_ts timestamptz,
camera_name text,
camera_make text,
lat double precision,
lon double precision,
size_bytes integer,
processed_key text,
thumbnail_key text,
source text NOT NULL CHECK (source IN ('gundi', 'entrance')),
source_ref text,
imported_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
raw jsonb
);
CREATE INDEX IF NOT EXISTS image_imei_servertime_idx
ON remote_cam.image (imei, servertime DESC);
CREATE INDEX IF NOT EXISTS image_servertime_idx
ON remote_cam.image (servertime DESC);
CREATE INDEX IF NOT EXISTS image_ocr_pending_idx
ON remote_cam.image (updated_at) WHERE ocr_state = 'PENDING';
-- Objektklassifikationen.
CREATE TABLE IF NOT EXISTS remote_cam.detection (
detection_id bigserial PRIMARY KEY,
image_uuid uuid NOT NULL REFERENCES remote_cam.image (image_uuid) ON DELETE CASCADE,
category text NOT NULL,
confidence double precision CHECK (confidence BETWEEN 0 AND 1),
bbox_x integer,
bbox_y integer,
bbox_w integer,
bbox_h integer,
model_version text,
source text NOT NULL CHECK (source IN ('gundi', 'entrance'))
);
CREATE INDEX IF NOT EXISTS detection_image_confidence_idx
ON remote_cam.detection (image_uuid, confidence DESC);
CREATE INDEX IF NOT EXISTS detection_category_idx
ON remote_cam.detection (category);
-- Transportschicht des Push-Pfads: Rohevents + Zustandsmaschine.
CREATE TABLE IF NOT EXISTS remote_cam.gundi_inbox (
object_id uuid PRIMARY KEY,
event_type text NOT NULL,
source text,
recorded_at timestamptz,
payload jsonb NOT NULL,
received_at timestamptz NOT NULL DEFAULT now(),
attachment_key text,
attachment_received_at timestamptz,
state text NOT NULL CHECK (state IN (
'AWAITING_ATTACHMENT', 'RECEIVED', 'INGESTED',
'FAILED', 'ABANDONED', 'IGNORED')),
attempts integer NOT NULL DEFAULT 0,
last_error text,
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS gundi_inbox_open_idx
ON remote_cam.gundi_inbox (state, updated_at)
WHERE state IN ('AWAITING_ATTACHMENT', 'RECEIVED');
-- Bild + der zum Aufnahme-Serverzeitpunkt gueltige Kamerastatus.
-- Tie-Breaker bei gleichem reported_at: der aus derselben Legacy-Zeile abgespaltene Status.
CREATE OR REPLACE VIEW remote_cam.image_with_status AS
SELECT
i.image_uuid,
i.imei,
i.servertime,
i.ocr_ts,
i.ocr_state,
i.camera_ts,
i.camera_name,
i.camera_make,
i.lat,
i.lon,
i.size_bytes,
i.processed_key,
i.thumbnail_key,
i.source,
i.imported_at,
s.status_id,
s.reported_at AS status_reported_at,
s.iccid,
s.init_key,
s.camera_name AS status_camera_name,
s.camera_model,
s.firmware_version,
s.camera_language,
s.customer_timezone,
s.mcc,
s.mnc,
s.lac,
s.cell_id,
s.camera_network,
s.battery_pct,
s.signal_strength,
s.temperature_c,
s.coordinate_switch,
s.work_period,
s.work_start,
s.work_end,
s.image_format,
s.image_night,
s.thumbnail_size,
s.pir_interval,
s.time_scan,
s.settings_changed_at
FROM remote_cam.image i
LEFT JOIN LATERAL (
SELECT cs.*
FROM remote_cam.camera_status cs
WHERE cs.imei = i.imei
AND cs.reported_at <= i.servertime
ORDER BY cs.reported_at DESC,
(cs.source_ref = i.image_uuid::text) DESC NULLS LAST,
cs.status_id DESC
LIMIT 1
) s ON TRUE;
+94
View File
@@ -0,0 +1,94 @@
-- Split der Legacy-Daten ins Fachmodell: jede remote_cam.metadata-Zeile ergibt
-- ein image und ein camera_status; remote_cam.category wird zu detection.
--
-- Erwartet die Temp-Tabelle _mig_batch(metadata_uuid text), die tools/migrate_legacy.py
-- je Batch fuellt. Wiederholbar: bereits migrierte Zeilen werden uebersprungen.
-- Alttabellen werden nur gelesen.
-- step: image
INSERT INTO remote_cam.image (
image_uuid, imei, servertime, ocr_ts, ocr_state, ocr_error, camera_ts,
camera_name, camera_make, lat, lon, size_bytes, processed_key, thumbnail_key,
source, source_ref, imported_at, updated_at
)
SELECT
m.metadata_uuid::uuid,
m.imei,
m.servertime,
m.ocr_ts,
CASE WHEN m.ocr_ts IS NULL THEN 'FAIL' ELSE 'OK' END,
CASE WHEN m.ocr_ts IS NULL THEN 'legacy: no ocr_ts' END,
m."ImageCreationDate",
m."CameraName",
m."CameraModel",
CASE WHEN m."Latitude" = 0 AND m."Longitude" = 0 THEN NULL ELSE m."Latitude" END,
CASE WHEN m."Latitude" = 0 AND m."Longitude" = 0 THEN NULL ELSE m."Longitude" END,
m."SizeInByte",
j.processed_jpg_key,
j.thumbnail_key,
'entrance',
m.metadata_uuid,
COALESCE(m.import_ts, now()),
now()
FROM _mig_batch b
JOIN remote_cam.metadata m ON m.metadata_uuid = b.metadata_uuid
LEFT JOIN remote_cam.import_job j ON j.image_uuid = m.metadata_uuid
WHERE m.imei IS NOT NULL AND m.servertime IS NOT NULL
ON CONFLICT (image_uuid) DO NOTHING;
-- step: camera_status
INSERT INTO remote_cam.camera_status (
imei, reported_at, received_at, iccid, init_key, camera_name, camera_model,
firmware_version, camera_language, customer_timezone, mcc, mnc, lac, camera_network,
battery_pct, signal_strength, temperature_c, coordinate_switch, work_period,
work_start, work_end, image_format, image_night, thumbnail_size, pir_interval, time_scan,
lat, lon, settings_changed_at, source, source_ref
)
SELECT
m.imei,
m.servertime,
COALESCE(m.import_ts, now()),
m.iccid,
m."INITKey",
m."CameraName",
m."CameraModel",
m."FirmwareVersion",
m."CameraLanguage",
m."CustomerTimezone",
m."MobileCountryCode",
m."MobileNetworkCode",
m."LocationAreaCode",
m."CameraNetwork",
m."BatteryStatus",
m."SignalStrength",
m."Temperature",
m."CoordinateSwitch",
m."WorkPeriod",
m."WorkStart",
m."WorkEnd",
m."ImageFormat",
m."ImageNight",
m."ThumbnailSize",
m."PIRInterval",
m."TimeScan",
CASE WHEN m."Latitude" = 0 AND m."Longitude" = 0 THEN NULL ELSE m."Latitude" END,
CASE WHEN m."Latitude" = 0 AND m."Longitude" = 0 THEN NULL ELSE m."Longitude" END,
m."DateTimeLastSettings",
'entrance',
m.metadata_uuid
FROM _mig_batch b
JOIN remote_cam.metadata m ON m.metadata_uuid = b.metadata_uuid
WHERE m.imei IS NOT NULL AND m.servertime IS NOT NULL
ON CONFLICT (source_ref) WHERE source = 'entrance' DO NOTHING;
-- step: detection
INSERT INTO remote_cam.detection (image_uuid, category, confidence, model_version, source)
SELECT i.image_uuid, c.category, c.score, c.version, 'entrance'
FROM _mig_batch b
JOIN remote_cam.category c ON c.image_uuid = b.metadata_uuid
JOIN remote_cam.image i ON i.image_uuid = b.metadata_uuid::uuid
WHERE c.category IS NOT NULL
AND NOT EXISTS (
SELECT 1 FROM remote_cam.detection d
WHERE d.image_uuid = i.image_uuid AND d.source = 'entrance'
);
+176
View File
@@ -0,0 +1,176 @@
#!/usr/bin/env python3
"""Spielt die Beispiel-Payloads aus der ICU-Doku gegen den Gundi-Receiver.
python -m tools.gundi_smoke_test --url http://127.0.0.1:8090 --jpeg bild.jpg
Ohne --jpeg wird ein synthetisches Bild mit Zeitstempel-Overlay erzeugt (braucht Pillow).
Standard-IMEI ist eine Test-IMEI, damit echte Kamerazeitreihen unberuehrt bleiben.
"""
import argparse
import http.client
import io
import json
import os
import sys
import uuid
from datetime import datetime, timedelta, timezone
from urllib.parse import urlparse
TEST_IMEI = "999000000000001"
def request(url: str, method: str, path: str, *, body: bytes = b"", headers: dict | None = None):
u = urlparse(url)
cls = http.client.HTTPSConnection if u.scheme == "https" else http.client.HTTPConnection
conn = cls(u.hostname, u.port, timeout=30)
try:
conn.request(method, path, body=body, headers=headers or {})
resp = conn.getresponse()
data = resp.read()
return resp.status, dict(resp.getheaders()), data
finally:
conn.close()
def post_json(url, path, key, payload):
headers = {"Content-Type": "application/json"}
if key is not None:
headers["apikey"] = key
return request(url, "POST", path, body=json.dumps(payload).encode(), headers=headers)
def post_file(url, path, key, jpg: bytes):
boundary = f"----smoke-{uuid.uuid4().hex}"
body = (
f"--{boundary}\r\n"
f'Content-Disposition: form-data; name="file1"; filename="../../evil.jpg"\r\n'
f"Content-Type: image/jpeg\r\n\r\n"
).encode() + jpg + f"\r\n--{boundary}--\r\n".encode()
return request(url, "POST", path, body=body, headers={
"Content-Type": f"multipart/form-data; boundary={boundary}",
"apikey": key,
})
def synthetic_jpeg(ts: datetime) -> bytes:
from PIL import Image, ImageDraw, ImageFont
w, h = 1920, 1080
im = Image.new("RGB", (w, h), (70, 90, 70))
draw = ImageDraw.Draw(im)
bar = int(h * 0.05)
draw.rectangle((0, h - bar, w, h), fill=(0, 0, 0))
font = ImageFont.load_default(size=int(bar * 0.8))
text = ts.strftime("%H:%M:%S %d-%m-%Y")
tw = draw.textlength(text, font=font)
draw.text((w - tw - 20, h - bar + bar * 0.08), text, fill=(255, 255, 255), font=font)
out = io.BytesIO()
im.save(out, format="JPEG", quality=90)
return out.getvalue()
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--url", default="http://127.0.0.1:8090")
ap.add_argument("--key", default=os.environ.get("GUNDI_API_KEY"))
ap.add_argument("--imei", default=TEST_IMEI)
ap.add_argument("--jpeg", help="Pfad zu einem echten Kamerabild")
args = ap.parse_args()
if not args.key:
print("--key oder GUNDI_API_KEY fehlt", file=sys.stderr)
return 2
failures = 0
def check(label: str, ok: bool, detail: str = "") -> None:
nonlocal failures
failures += 0 if ok else 1
print(f"[{'OK ' if ok else 'FAIL'}] {label}{(' - ' + detail) if detail else ''}")
now = datetime.now(timezone.utc).replace(microsecond=0)
status_payload = {
"source": args.imei,
"event_type": "camera_status_report",
"title": "Camera Status report - Smoke Test",
"recorded_at": (now - timedelta(minutes=10)).isoformat().replace("+00:00", "Z"),
"location": {"lat": 0.0, "lon": 0.0},
"event_details": {
"imei": args.imei, "iccid": "8900000000000000000", "init_key": "000000000000",
"camera_name": "Smoke Test", "camera_model": None, "firmware_version": "1.4.2",
"mobile_country_code": "232", "mobile_network_code": "01", "location_area_code": "1234",
"cell_id": "567890", "customer_timezone": "Europe/Vienna", "battery_status": 87,
"signal_strength": 3, "temperature": -4, "coordinate_switch": False, "work_period": True,
"image_format": "F4_3", "image_night": "MIN", "work_start": "06:00", "work_end": "21:00",
"thumbnail_size": "720p", "pir_interval": "30", "time_scan": "120", "camera_language": "en",
},
}
trap_payload = {
"source": args.imei,
"event_type": "cameratrap_rep",
"title": "Camera trap image - Smoke Test",
"recorded_at": now.isoformat().replace("+00:00", "Z"),
"location": {"lat": 47.8095, "lon": 13.0550},
"event_details": {
"cameratraprep_camera-name": "Smoke Test",
"cameratraprep_camera-make": "M350GW",
"detections": [
{"category": "deer", "confidence_percent": 96.2, "x_px": 0, "y_px": 0, "width_px": 0, "height_px": 0},
{"category": "human", "confidence_percent": 3.1, "x_px": 15, "y_px": 17, "width_px": 100, "height_px": 200},
],
},
}
code, _, _ = request(args.url, "GET", "/healthz")
check("healthz", code == 200, f"HTTP {code}")
code, _, _ = post_json(args.url, "/v2/events/", None, status_payload)
check("ohne apikey -> 401", code == 401, f"HTTP {code}")
code, _, _ = post_json(args.url, "/v2/events/", "falsch", status_payload)
check("falscher apikey -> 401", code == 401, f"HTTP {code}")
code, headers, _ = post_json(args.url, "/v2/events", args.key, status_payload)
check("ohne Slash -> 308", code == 308 and headers.get("Location") == "/v2/events/",
f"HTTP {code} Location={headers.get('Location')}")
code, _, data = post_json(args.url, "/v2/events/", args.key, {"event_type": "cameratrap_rep"})
check("unvollstaendiges Event -> 400", code == 400, f"HTTP {code} {data[:120]!r}")
code, _, data = post_json(args.url, "/v2/events/", args.key, status_payload)
status_id = json.loads(data).get("object_id") if code == 200 else None
check("camera_status_report", code == 200 and bool(status_id), f"HTTP {code} object_id={status_id}")
code, _, _ = post_json(args.url, "/v2/events/", args.key, status_payload)
check("camera_status_report erneut (idempotent in camera_status)", code == 200, f"HTTP {code}")
code, _, data = post_json(args.url, "/v2/events/", args.key, trap_payload)
object_id = json.loads(data).get("object_id") if code == 200 else None
check("cameratrap_rep", code == 200 and bool(object_id), f"HTTP {code} object_id={object_id}")
if args.jpeg:
with open(args.jpeg, "rb") as f:
jpg = f.read()
else:
jpg = synthetic_jpeg(now.astimezone(timezone(timedelta(hours=2))))
code, _, _ = post_file(args.url, f"/v2/events/{uuid.uuid4()}/attachments/", args.key, jpg)
check("Attachment auf unbekannte object_id -> 404", code == 404, f"HTTP {code}")
code, _, _ = post_file(args.url, "/v2/events/..%2F..%2Fx/attachments/", args.key, jpg)
check("Attachment auf ungueltige object_id -> 404", code == 404, f"HTTP {code}")
code, _, _ = post_file(args.url, f"/v2/events/{status_id}/attachments/", args.key, jpg)
check("Attachment auf Status-Event -> 404", code == 404, f"HTTP {code}")
if object_id:
code, headers, _ = post_file(args.url, f"/v2/events/{object_id}/attachments", args.key, jpg)
check("Attachment ohne Slash -> 308", code == 308, f"HTTP {code}")
code, _, data = post_file(args.url, f"/v2/events/{object_id}/attachments/", args.key, jpg)
check("Attachment", code == 200, f"HTTP {code} {data[:120]!r}")
code, _, _ = post_file(args.url, f"/v2/events/{object_id}/attachments/", args.key, jpg)
check("Attachment erneut (duplicate, 200)", code == 200, f"HTTP {code}")
print(f"\nobject_id cameratrap_rep: {object_id}")
print(f"object_id status_report: {status_id}")
print(f"{failures} Fehler")
return 1 if failures else 0
if __name__ == "__main__":
raise SystemExit(main())
+142
View File
@@ -0,0 +1,142 @@
#!/usr/bin/env python3
"""Migriert remote_cam.metadata/category batchweise ins Fachmodell (image, camera_status, detection).
python -m tools.migrate_legacy # PG_DSN aus der Umgebung
python -m tools.migrate_legacy --dsn postgresql://... --batch 20000
python -m tools.migrate_legacy --verify-only
Wiederholbar. Jeder Batch ist eine eigene Transaktion.
"""
import argparse
import os
import re
import sys
import time
from pathlib import Path
import psycopg
SCHEMA_DIR = Path(__file__).resolve().parent.parent / "schema"
SCHEMA_LOCK_KEY = 7_311_000_001
def load_steps() -> list[tuple[str, str]]:
sql = (SCHEMA_DIR / "002_migrate_legacy.sql").read_text(encoding="utf-8")
parts = re.split(r"^-- step: (\w+)\s*$", sql, flags=re.MULTILINE)
return [(parts[i], parts[i + 1].strip()) for i in range(1, len(parts), 2)]
def apply_schema(conn: psycopg.Connection) -> None:
sql = (SCHEMA_DIR / "001_domain_model.sql").read_text(encoding="utf-8")
with conn.transaction(), conn.cursor() as cur:
cur.execute("SELECT pg_advisory_xact_lock(%s)", (SCHEMA_LOCK_KEY,))
cur.execute(sql)
def migrate(conn: psycopg.Connection, batch_size: int) -> None:
steps = load_steps()
totals = {name: 0 for name, _ in steps}
with conn.cursor() as cur:
cur.execute("SELECT count(*) FROM remote_cam.metadata")
total_rows = cur.fetchone()[0]
cur.execute(
"CREATE TEMP TABLE IF NOT EXISTS _mig_batch (metadata_uuid text PRIMARY KEY) ON COMMIT DELETE ROWS"
)
conn.commit()
last = ""
seen = 0
t0 = time.perf_counter()
while True:
with conn.transaction(), conn.cursor() as cur:
cur.execute(
"""
INSERT INTO _mig_batch
SELECT metadata_uuid FROM remote_cam.metadata
WHERE metadata_uuid > %s ORDER BY metadata_uuid LIMIT %s
""",
(last, batch_size),
)
n = cur.rowcount
if n == 0:
break
cur.execute("SELECT max(metadata_uuid) FROM _mig_batch")
last = cur.fetchone()[0]
added = []
for name, sql in steps:
cur.execute(sql)
totals[name] += cur.rowcount
added.append(f"{name}+{cur.rowcount}")
seen += n
rate = seen / max(time.perf_counter() - t0, 1e-6)
print(f" {seen:>8}/{total_rows} {' '.join(added)} ({rate:.0f} Zeilen/s)", flush=True)
print("Neu angelegt: " + ", ".join(f"{k}={v}" for k, v in totals.items()), flush=True)
def verify(conn: psycopg.Connection) -> bool:
checks = [
("metadata == image(entrance)",
"SELECT (SELECT count(*) FROM remote_cam.metadata), "
"(SELECT count(*) FROM remote_cam.image WHERE source = 'entrance')"),
("metadata == camera_status(entrance)",
"SELECT (SELECT count(*) FROM remote_cam.metadata), "
"(SELECT count(*) FROM remote_cam.camera_status WHERE source = 'entrance')"),
("category == detection(entrance)",
"SELECT (SELECT count(*) FROM remote_cam.category), "
"(SELECT count(*) FROM remote_cam.detection WHERE source = 'entrance')"),
("metadata ohne ocr_ts == image FAIL",
"SELECT (SELECT count(*) FROM remote_cam.metadata WHERE ocr_ts IS NULL), "
"(SELECT count(*) FROM remote_cam.image WHERE source = 'entrance' AND ocr_state = 'FAIL')"),
("keine 0/0-Koordinaten (image, camera_status)",
"SELECT 0, (SELECT count(*) FROM remote_cam.image WHERE lat = 0 AND lon = 0) + "
"(SELECT count(*) FROM remote_cam.camera_status WHERE lat = 0 AND lon = 0)"),
("Stichprobe: image_with_status == metadata (200 Zufallszeilen, Abweichungen)",
"""
SELECT 0, count(*) FROM (
SELECT m.* FROM remote_cam.metadata m TABLESAMPLE SYSTEM (1) LIMIT 200
) m
JOIN remote_cam.image_with_status v ON v.image_uuid = m.metadata_uuid::uuid
WHERE v.status_reported_at IS DISTINCT FROM m.servertime
OR v.battery_pct IS DISTINCT FROM m."BatteryStatus"
OR v.firmware_version IS DISTINCT FROM m."FirmwareVersion"
OR v.status_camera_name IS DISTINCT FROM m."CameraName"
OR v.signal_strength IS DISTINCT FROM m."SignalStrength"
OR v.temperature_c IS DISTINCT FROM m."Temperature"
OR v.customer_timezone IS DISTINCT FROM m."CustomerTimezone"
OR v.ocr_ts IS DISTINCT FROM m.ocr_ts
"""),
]
ok_all = True
with conn.cursor() as cur:
for label, sql in checks:
cur.execute(sql)
expected, actual = cur.fetchone()
ok = expected == actual
ok_all &= ok
print(f"[{'OK ' if ok else 'FAIL'}] {label}: erwartet={expected} ist={actual}", flush=True)
conn.rollback()
return ok_all
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--dsn", default=os.environ.get("PG_DSN"))
ap.add_argument("--batch", type=int, default=20000)
ap.add_argument("--verify-only", action="store_true")
args = ap.parse_args()
if not args.dsn:
print("--dsn oder PG_DSN fehlt", file=sys.stderr)
return 2
with psycopg.connect(args.dsn, connect_timeout=10) as conn:
if not args.verify_only:
apply_schema(conn)
print("Schema angewendet. Migriere ...", flush=True)
migrate(conn, args.batch)
print("Verifikation:", flush=True)
return 0 if verify(conn) else 1
if __name__ == "__main__":
raise SystemExit(main())