🇬🇧 English | 🇹🇼 繁體中文

Construction Hazard Detection

AI-assisted construction-site safety monitoring for live cameras. The current runtime is centred on main.py, src/stream_processor.py, local YOLO worker processes, MediaMTX live publishing, PostgreSQL records, Redis coordination, and FastAPI services for management, notifications, streaming metadata, and violation records.

What It Detects

• Workers without hard hats.
• Workers without safety vests.
• Workers too close to machinery or vehicles.
• Workers inside cone-derived controlled areas.
• Machinery or vehicles too close to utility poles.

Supported label and notification languages include Traditional Chinese, Simplified Chinese, English, French, Thai, Vietnamese, Indonesian, and Japanese.

Hazard Detection Examples

Below are examples of real-time hazard detection by the system.

Runtime Architecture

The diagram summarises the live data path, the 11 configured detection classes, the safety-warning rules, and the video/metadata outputs. Redis is not used to store live video frames; MediaMTX owns live playback, while Redis keeps only small coordination state such as authentication cache, FCM token cache, compact warning metadata, overlay demand keys, and overlay ready keys.

Repository Map

• main.py: supervises configured streams and worker processes.
• src/: production runtime modules.
• examples/bff/: Web session, CSRF, media capability, and API gateway.
• examples/db_management/: users, groups, sites, stream configuration API.
• examples/local_notification_server/: FCM token and site notification API.
• examples/streaming_web/: labels, playback URLs, metadata channels, media-session auth, and WebRTC ICE settings.
• examples/violation_records/: violation record and image API.
• examples/YOLO_server_api/: optional standalone YOLO API.
• examples/mcp_server/: optional AI tools for detection, records, models and multi-platform notifications; setup.
• examples/YOLO_train/, examples/YOLO_evaluation/, examples/YOLO_data_augmentation/: training, export, evaluation and augmentation tools.
• scripts/: database initialisation and TensorRT rebuild helpers.

Notifications support FCM, LINE Messaging API, Messenger, Telegram, WeChat and HTTP broadcast. Model-development tools remain in this repository with optional dependencies.

Flutter Frontend

The Flutter source is maintained separately in visionnaire-flutter. frontend/web/ in this repository is an ignored, generated deployment artifact, not editable frontend source. See frontend/README.md for the release and backend-contract boundary.

Recommended Runtime

Use database mode unless you are testing a short-lived local JSON config.

1. PostgreSQL stores sites, users, stream configurations, and violations.
2. Redis stores auth/session cache, notification token cache, and compact live coordination keys.
3. MediaMTX serves RTSP ingest plus HLS/WebRTC playback.
4. main.py polls stream configuration and starts one stream process per active camera.
5. Local YOLO workers share GPU inference across cameras through shared memory.

The standalone YOLO server remains available for API testing or separate deployment, but the recommended high-throughput local path is the YOLO worker mode in src/yolo_worker.py.

Quick Start

1. Prepare Python

The project currently targets Python >=3.14,<3.15.

uv sync --locked --all-extras --no-extra yolo-gpu

The default dependency set contains only shared API runtime packages. Docker images install their own streaming, violation, notification, or yolo extra; use --extra yolo-gpu only on CUDA/TensorRT hosts.

Use uv sync --extra social-notifications for multi-platform notifications, or uv sync --extra mcp --extra social-notifications with MCP. Install uv sync --extra training for training, evaluation and augmentation; add --extra yolo-gpu for TensorRT export.

2. Download Models

hf download yihong1120/Construction-Hazard-Detection \
  --repo-type model \
  --include "models/pt/*.pt" \
  --local-dir .

Expected worker model filenames are models/pt/best_<model_key>.pt, for example models/pt/best_yolo26n.pt.

3. Create .env

Start from .env.example, then adjust hostnames and secrets.

Important values:

DATABASE_URL='postgresql+asyncpg://username:password@127.0.0.1/construction_hazard_detection'

REDIS_HOST='127.0.0.1'
REDIS_PORT=6379
REDIS_PASSWORD='set-a-strong-password'

OIDC_ENABLED=true
OIDC_ISSUER_URL=https://sso.example.com/realms/visionnaire
OIDC_JWKS_URL=https://sso.example.com/realms/visionnaire/protocol/openid-connect/certs
OIDC_AUDIENCE=visionnaire-api
OIDC_ACCOUNT_URL=https://sso.example.com/realms/visionnaire/account

DB_MANAGEMENT_API_URL='http://127.0.0.1:8005'
FCM_API_URL='http://127.0.0.1:8003'
VIOLATION_RECORD_API_URL='http://127.0.0.1:8002'
STREAMING_API_URL='http://127.0.0.1:8800'

YOLO_WORKER_CAMERAS_PER_ENGINE=3
# Override capacity by model size; unlisted model keys use the default above.
YOLO_WORKER_CAMERAS_PER_ENGINE_BY_MODEL=yolo26n=8,yolo26s=6,yolo26m=4,yolo26l=2,yolo26x=1
YOLO_WORKER_DEVICES=cuda:0,cuda:0
YOLO_WORKER_QUEUE_SIZE=64
YOLO_WORKER_RESULT_QUEUE_SIZE=8
YOLO_WORKER_RING_SLOTS=2
YOLO_WORKER_RING_SLOT_CLEANUP_SECONDS=120
YOLO_WORKER_BATCH_SIZE=8
YOLO_WORKER_BATCH_WAIT_MS=10
YOLO_WORKER_PRECISION=f16
# f32/f16 use models/pt/*.pt; int8 uses models/int8_engine/*.engine.

# Each camera keeps a bounded shared-memory frame ring and receives results on
# its own queue. For Docker, set YOLO_WORKER_SHM_SIZE high enough for
# camera_count * YOLO_WORKER_RING_SLOTS * maximum_frame_bytes.

MEDIA_PUBLISH_RTSP_BASE_URL='rtsp://127.0.0.1:8554'
MEDIA_PUBLIC_HLS_BASE_URL='/hazard/media'
MEDIA_PUBLIC_WEBRTC_BASE_URL='/hazard/media/webrtc'
MEDIA_PUBLISH_CLEAN_STREAM=true
MEDIA_PUBLISH_ANNOTATED_STREAM=true
# Detail clean video uses the shared latest-frame capture and Intel iGPU. This
# drops stale frames, avoids a second RTSP connection to each camera, and keeps
# HLS keyframes regular. Enable direct source restreaming only when the
# untouched camera bitstream is explicitly required.
MEDIA_PUBLISH_CLEAN_SOURCE_RESTREAM=false
MEDIA_PUBLISH_CLEAN_ENCODER=h264_vaapi
MEDIA_PUBLISH_ENCODER=h264_vaapi
MEDIA_PUBLISH_VAAPI_DEVICE=/dev/dri/renderD128
# Share viewer-demand reads between a camera's clean/detail/preview publishers.
MEDIA_DEMAND_CACHE_SECONDS=0.5

API services share the configured OIDC issuer, JWKS URL and API audience.

4. Start Infrastructure

Redis, PostgreSQL, and MediaMTX are required for the normal database-backed runtime:

docker compose up -d redis postgres media-server

Check that the containers are running:

docker compose ps redis postgres media-server
docker compose logs -f redis postgres media-server

If PostgreSQL and Redis are managed on the host and only MediaMTX is needed, use the standalone Compose file. It does not interpolate database or Redis credentials:

docker compose -f docker-compose.media.yml up -d
docker compose -f docker-compose.media.yml ps
docker compose -f docker-compose.media.yml logs -f media-server

The full docker-compose.yml starts its own PostgreSQL container, so it requires POSTGRES_DB, POSTGRES_USER, and POSTGRES_PASSWORD separately. Those fields cannot be safely or reliably reconstructed from DATABASE_URL.

When the Python services run on the host, use local addresses in .env:

DATABASE_URL='postgresql+asyncpg://username:password@127.0.0.1/construction_hazard_detection'
REDIS_HOST='127.0.0.1'
REDIS_PORT=6379
REDIS_PASSWORD='password'
MEDIA_PUBLISH_RTSP_BASE_URL='rtsp://127.0.0.1:8554'

When main.py or the FastAPI services run inside Docker Compose, use service names instead:

DATABASE_URL='postgresql+asyncpg://username:password@postgres/construction_hazard_detection'
REDIS_HOST='redis'
POSTGRES_DB='construction_hazard_detection'
POSTGRES_USER='construction_app'
POSTGRES_PASSWORD='set-a-strong-password'
REDIS_PASSWORD='set-a-strong-password'
MEDIA_PUBLISH_RTSP_BASE_URL='rtsp://media-server:8554'

Infrastructure roles:

• PostgreSQL stores users, sites, stream settings, and violation records.
• Redis stores authentication cache, token cache, compact live metadata, and overlay demand keys.
• MediaMTX receives RTSP streams from main.py and exposes HLS/WebRTC playback.

Redis and MediaMTX do not store violation images or database records.

For a new, empty PostgreSQL database, import the bootstrap schema if it was not mounted at container creation time. This script drops application tables, so never run it against a database that contains data:

cat ./scripts/init.postgres.sql | docker exec -i postgres-container \
  psql -U username -d construction_hazard_detection

For an existing database, follow the organisation's database change-management runbook. Migration assets and privileged operator tooling are intentionally not distributed with the application source.

5. Configure MediaMTX

MediaMTX is the live media server. main.py publishes processed H.264 streams to MediaMTX over RTSP, and viewers play those streams through HLS or WebRTC. Redis only stores compact metadata and demand keys; it does not carry video frames.

The Docker Compose service exposes local-only ports:

127.0.0.1:8554  RTSP ingest from main.py / ffmpeg
127.0.0.1:8890  HLS playback, mapped to MediaMTX port 8888
127.0.0.1:8889  WebRTC/WHEP playback

Use these settings when main.py runs on the host:

MEDIA_PUBLISH_RTSP_BASE_URL='rtsp://127.0.0.1:8554'
MEDIA_PUBLIC_HLS_BASE_URL='/hazard/media'
MEDIA_PUBLIC_WEBRTC_BASE_URL='/hazard/media/webrtc'

The host account that runs main.py needs access to the Intel render node:

sudo usermod -aG render "$USER"
# Sign out and back in before starting main.py.

With the configured on-demand publishers, clean detail video uses FFmpeg stream copy and consumes no encoder session. Only active preview and annotated paths use h264_vaapi on the Intel iGPU; MediaMTX then remuxes and distributes that one encoded stream to all viewers. This avoids the RTX 4090 GeForce NVENC eight-session policy without moving overlay drawing into the browser.

Use the Compose service name when main.py runs inside Docker:

MEDIA_PUBLISH_RTSP_BASE_URL='rtsp://media-server:8554'

Docker Compose maps /dev/dri/renderD128 into the detection container. Set INTEL_RENDER_GID to the host render group ID if it is not 992.

The streaming backend returns playback URLs shaped like:

/hazard/media/<media-path>/index.m3u8
/hazard/media/webrtc/<media-path>/whep

For public deployments, proxy MediaMTX through Nginx and protect it with examples.streaming_web media authorisation. Use examples/streaming_web/nginx.hazard-media.conf as the starting point:

Nginx /hazard/media/*        -> MediaMTX HLS port 8888
Nginx /hazard/media/webrtc/* -> MediaMTX WebRTC port 8889
Nginx /hazard/api/media-auth -> examples.streaming_web /media-auth

Recommended live-buffer defaults are already in docker-compose.yml:

MTX_HLSSEGMENTDURATION=2s
MTX_HLSSEGMENTCOUNT=14
MTX_HLSALWAYSREMUX=yes
MTX_HLSMUXERCLOSEAFTER=60s

Lower MTX_HLSSEGMENTCOUNT reduces disk and memory use. The default 14 segments (about 28 seconds at 2-second segments) gives wall tiles enough room to recover from a short publisher or network interruption.

6. Start APIs

Run each service from the repository root:

uvicorn examples.db_management.app:app --host 127.0.0.1 --port 8005 --workers 2 --timeout-graceful-shutdown 10
uvicorn examples.local_notification_server.app:app --host 127.0.0.1 --port 8003 --workers 2
uvicorn examples.violation_records.app:app --host 127.0.0.1 --port 8002 --workers 2
uvicorn examples.streaming_web.app:app --host 127.0.0.1 --port 8800 --workers 2 --timeout-graceful-shutdown 10

Optional standalone detector API:

uvicorn examples.YOLO_server_api.app:app --host 127.0.0.1 --port 8000 --workers 2

7. Start Stream Processing

python main.py

Optional polling interval:

python main.py --poll 5

Optional file-based mode for development:

python main.py --config config/configuration.json

Do not run database mode and JSON mode at the same time for the same cameras.

Live Viewing

Clients should call the streaming web backend:

• GET /labels
• GET /streams/{label}
• GET /metadata/stream-id/{label}/{stream_id} for SSE metadata
• WebSocket /ws/metadata-id/{label}/{stream_id} for metadata
• GET /webrtc/ice-servers when WebRTC needs STUN/TURN settings

Video playback comes from MediaMTX HLS/WebRTC URLs. Warning metadata is compact and usually contains only current warning state. Detection boxes, polygons, and labels are rendered into backend-published annotated video streams.

For public deployments, put MediaMTX behind Nginx auth_request and let examples.streaming_web validate JWT and site access.

Storage Retention

Violation images are stored under each service's configured static/ location. Without archive/NAS/external disk, a practical default is:

• keep images and DB records for 18 months;
• delete image files and matching database rows together;
• run cleanup during low-traffic hours;
• keep HLS segment retention short because MediaMTX HLS is a live buffer, not an archive.

Do not delete database records while keeping broken image paths, and do not delete image files while keeping records that the UI still needs to display.

Development Checks

pre-commit run -a
python -m pytest -q --tb=short

The mypy pre-commit hook checks production code (main.py, src, examples, and scripts). Tests are still validated by pytest and flake8; many test files intentionally use dynamic mocks and invalid payloads.

Dataset And Models

The project uses the Construction Hazard Detection model repository:

• Hugging Face: https://huggingface.co/yihong1120/Construction-Hazard-Detection
• Roboflow: https://universe.roboflow.com/side-projects/construction-hazard-detection

Labels:

0 Hardhat
1 Mask
2 NO-Hardhat
3 NO-Mask
4 NO-Safety Vest
5 Person
6 Safety Cone
7 Safety Vest
8 Machinery
9 Utility Pole
10 Vehicle

Licence

This project is licensed under the AGPL-3.0 Licence.