Open Source AI Stack — Reference Architecture

A fully open source software stack for SMBs, running on 1–2 servers, covering everything from the OS to end-user-facing analytics/AI applications — chat + RAG, classification/prediction, SSO, and security. Every component ships under an OSI-approved (or explicitly noted) open source license and runs entirely on-prem.

Layer stack (bottom → top)
1
Operating SystemHost OS
Ubuntu Server LTSDebian
2
Virtualizationoptional
Proxmox VE
3
Containers & Orchestrationrun/manage services
DockerDocker Composek3s (growth path)
4
Networking & EdgeTLS, routing
TraefikCaddy
5
Identity & SSOlogin, MFA, groups
KeycloakAuthentik
6
Secrets Managementcredentials, keys
OpenBaoSOPS + age
7
Data Platformrelational/vector/object/cache
PostgreSQL + pgvectorMinIORedisQdrant (optional)
8
Model Servingrun LLMs & embeddings
OllamavLLMllama.cppLiteLLM gateway
9
AI Application & Orchestrationchat UI, RAG, agents, ML
Open WebUI LangChain / LlamaIndex LangGraph / Flowise MLflow Airflow / Dagster scikit-learn / XGBoost
10
Observability & Guardrailsmetrics, logs, LLM safety
Prometheus/Grafana/LokiLangfusePresidioLLM Guard
Infrastructure Identity / Security / Observability Data Model serving Applications
Recommended 2-server topology

Server A — Platform

CPU box · 8–16 cores · 32–64GB RAM · no GPU required
Edge & Identity
TraefikKeycloakOpenBao
Data
PostgreSQL + pgvectorMinIORedis
App & Pipelines
Open WebUILiteLLMAirflowMLflow
Observability
PrometheusGrafanaLokiLangfuse

Server B — AI / Compute

GPU box (24GB+ VRAM recommended) · CPU-only viable at smaller scale
Model Serving
Ollama / vLLMEmbedding model server
Training / Registry
MLflow training jobsBatch scoring jobs
Stateless — can be rebuilt from scratch. All durable state lives on Server A.
Two worked examples

A · ChatGPT-like RAG assistant

Docs → MinIO→ Chunk + embed→ pgvector
User (Keycloak SSO)→ Open WebUI→ LiteLLM→ RAG retrieval (ACL-filtered)→ Ollama/vLLM→ Answer + citations

B · Classification / prediction

Postgres data→ Airflow/Dagster ETL→ Train (scikit-learn/XGBoost)→ MLflow registry
New record→ FastAPI scoring endpoint→ Write prediction → Postgres→ Grafana / business app