Architecture Overview
Detailed design view for upload, processing, retrieval, ACL enforcement, and answer generation.
1. Design Goals
- End-to-end flow from upload to grounded answer
- Permission boundaries enforced in retrieval path
- Image/diagram-aware parsing with selective VLM routing
- Evaluation data persistence for continuous tuning
2. Component Diagram
3. Data Responsibilities
| Store | Main role |
|---|---|
| MinIO | Raw files, chunk temp data, snapshot/image artifacts |
| PostgreSQL | Status, metadata, structured blocks, usage/eval records |
| Elasticsearch | Hybrid retrieval index (text + dense vectors) |
| Redis | Chunk bitmap/progress, session/cache support |
| Kafka | Async parsing tasks and decoupled processing |
4. Key Paths
Upload Path
chunk upload -> merge -> task publish
chunk upload -> merge -> task publish
Parse Path
parse/chunk/structure -> embeddings -> index
parse/chunk/structure -> embeddings -> index
Query Path
intent/query understanding -> hybrid retrieval
intent/query understanding -> hybrid retrieval
Answer Path
ACL-filtered evidence -> grounded LLM response
ACL-filtered evidence -> grounded LLM response
5. Reliability and Governance
- Kafka multi-consumer processing for throughput
- Idempotency lock and done-marker to avoid duplicate indexing
- DLQ routing for repeated parse failures
- Access filtering applied before answer generation
- Usage/evaluation logs persisted for quality tuning