Problem
Agricultural field observations often arrive as unstructured image and voice input. Without a shared pipeline, it is hard to enrich them consistently, compare similar cases and prepare data for human review or future training.
PrototypeSchool group project (3 developers), backend/pipeline integration, ML-service coordination, dashboard review flow and Docker-based local deployment
Multimodal Observation Pipeline
A UCL group project for Shush Technologies: a Docker-based pipeline that turns field observations (image + voice) into structured, reviewable AI output for plant-health decision support.
Project snapshot
Status
Prototype
Project type
School project
Role
School group project (3 developers), backend/pipeline integration, ML-service coordination, dashboard review flow and Docker-based local deployment
Main focus
The system combines a .NET MAUI field app, Django API, Celery worker, FastAPI ML service and PostgreSQL/pgvector into one asynchronous pipeline: upload, speech-to-text, multimodal embeddings, candidate findings, similar-case retrieval and reviewer dashboard.
Stack
PythonDjangoFastAPIC# / .NET MAUIPostgreSQLpgvector
Solution
The system combines a .NET MAUI field app, Django API, Celery worker, FastAPI ML service and PostgreSQL/pgvector into one asynchronous pipeline: upload, speech-to-text, multimodal embeddings, candidate findings, similar-case retrieval and reviewer dashboard.
Technology
PythonDjangoFastAPIC# / .NET MAUIPostgreSQLpgvectorDockerCeleryRedisWhisperOpenCLIP
Product walkthrough
Click the dots or arrows to move through the main project screens.
01 / 05
Dashboard overview
The overview dashboard tracks pipeline health, review workload, data quality and training readiness. Reviewers can quickly see totals, quality queue items, image-only observations and items that need attention.
Pipeline flow
- MauiFlows uploads image, farmer signal and optional voice to the Django API.
- Celery runs the VoiceObservationPipeline asynchronously so the field user is not blocked by STT or ML processing.
- Whisper transcribes voice, the ML service generates OpenCLIP embeddings and enrichment, and similar observations are retrieved for review context.
- AI output is stored as reviewable decision support — not as a final diagnosis.
Built with
- Developed as a UCL Datamatiker group project for Shush Technologies with Python/Django, FastAPI, Celery, Redis, PostgreSQL/pgvector and Docker Compose.
- .NET MAUI powers the field app while the ML service is separated behind a versioned API contract.
- GitHub Actions handles CI; the MVP currently runs locally in Docker rather than in production.
- APK builds are distributed through GitHub Releases for demo/testing on Android devices.