The 5-Tier Medallion Architecture for Clinical AI
The classic three-tier Medallion pattern (Bronze-Silver-Gold) is extended by two additional tiers — Diamond and Platinum — that represent a paradigm shift:
Bronze Layer — Raw Append-Only Ledger
The Bronze Layer is the immutable landing zone for all raw hospital data streams. It captures:
- Device telemetry, EHR event payloads, ADT messages, DICOM metadata, lab instrument feeds, pharmacy dispensing events, and third-party data connectors.
- Exactly as generated — no transformation, no filtering.
Apache-Powered Execution
High-throughput, low-latency event streaming from edge devices and HL7 FHIR API sources. Partitioning by department enables parallel consumption. Modern deployments use KRaft mode, eliminating ZooKeeper for simplified operations.
Micro-batch ingestion jobs consume from Kafka topics and append raw JSON/HL7 payloads as immutable Parquet/Iceberg files on the object store, with exactly-once processing semantics.
Acts as the underlying distributed filesystem. All raw files are append-only. Portable across AWS S3, Azure ADLS Gen2, GCS, and on-premises MinIO/Ceph via the S3A connector.
Provides ACID-compliant snapshot isolation, time-travel queries, and schema evolution without rewriting historical data.
Implemented via Iceberg's transaction model to handle thousands of simultaneous writer streams without lock contention.
A Gemma or lightweight open-weight LLM sidecar container performs PHI redaction on raw text fields before any payload leaves the hospital network perimeter.
Kubernetes Deployment
Kafka brokers run as StatefulSets with persistent volume claims (PVCs) on NVMe storage, managed by the Strimzi Kafka Operator. Spark Structured Streaming applications run as Kubernetes Deployments via the Spark Operator, auto-scaling executor pods based on Kafka consumer lag metrics exposed through KEDA.
Rationale
Silver Layer — Conformed Operational State
The Silver Layer cleans, normalizes, deduplicates, and structures raw Bronze data into an enterprise-wide schema.
- Single source of truth for operational and analytical workloads
- De-identification enforced here — all PHI has been stripped at the edge
- Silver enforces the schema contract and serves as the platform compliance boundary
Apache-Powered Execution
Parse raw JSON/HL7/FHIR payloads into structured Parquet tables with Iceberg's schema enforcement. Delta detection via MERGE INTO operations deduplicates late-arriving events.
Centralized schema registry for all Silver tables, consumed by Spark, Flink, Trino, and ML pipelines uniformly across clouds.
Row-level and column-level access policies enforce multi-tenant isolation using Attribute-Based Access Control (ABAC). Partition key (/TenantID/DepartmentID) ensures strict isolation.
Tracks full data lineage from Bronze ingest through every Spark transformation. PHI classification tags propagate automatically to all derived columns.
The Silver layer operates as the Command (Write) side. Long-term clinical states are written to Iceberg tables; operational real-time states are written to a low-latency distributed database (Cassandra or CockroachDB/YugabyteDB).
Kubernetes Deployment
Spark ELT jobs are managed by Apache Airflow’s KubernetesExecutor — each DAG task spawns a dedicated Kubernetes pod, providing complete workload isolation. Hive Metastore runs as a Kubernetes Deployment backed by a PostgreSQL StatefulSet.
Gold Layer — Dimensional Analytics
The Gold Layer transforms relational Silver data into highly aggregated, domain-specific KPIs and dimensional models.
- BI dashboards
- Executive reporting
- SQL-based operational analytics
Apache-Powered Execution
Executes dimensional modeling transformations — building Star Schemas, Fact tables (patient encounters, device events, claims) and Dimension tables.
Automated Spark jobs compact small files and expire old snapshots on a 15-minute schedule, maintaining fast OLAP query performance.
Federated SQL query engine deployed on Kubernetes, enabling sub-second analytical queries across Gold Iceberg tables without moving data.
Open-source BI dashboarding deployed on Kubernetes, connecting to Trino for interactive dashboards — a cloud-agnostic alternative to proprietary tools.
Orchestrates Gold transformation DAGs, SLA monitoring, and automated data quality checks using Great Expectations.
Rationale
- ALOS, readmission rates, device utilization, cost per case
- Business executives, department heads, and quality teams access long-term operational trends without incurring heavy compute costs on raw transactional tables
Diamond Layer — Semantic Spine & Knowledge Store
The Diamond Layer converts structured enterprise data and unstructured clinical assets into formats optimized for Large Language Models and ML inference. It serves as the organization’s Feature Store, RAG knowledge engine, and Ontological GraphRAG Gateway.
Apache-Powered Execution
Extract semantic chunks from clinical text at scale. Named Entity Recognition (NER) tags clinical entities (diagnosis, medication, procedure) before embedding.
High-performance in-memory columnar format used for zero-copy transfer of feature vectors between Spark feature computation and the vector store ingestion pipeline.
Manages clinical ML features — patient risk scores, device health indices, lab trend vectors — with versioned, point-in-time-correct retrieval.
Vector databases store high-density embeddings. Hybrid Search combines dense vector similarity with BM25 sparse lexical matching, merged via Reciprocal Rank Fusion (RRF).
Deployed as Kubernetes services using vLLM or Triton Inference Server, supporting BioBERT, Clinical-BERT, or BioMedLM models.
Maps absolute relational concepts. Nodes represent patients, specimens, or facilities; edges represent deterministic medical relationships and clinical timelines.
Every ingested chunk is deterministically tagged against SNOMED-CT (clinical terms), LOINC (laboratory observations), ICD-11 (disease classification), RxNorm (pharmaceuticals), and HGNC (genomic nomenclature).
Maintains a versioned history of all feature values for offline ML training, enabling time-travel audits.
Rationale
Platinum Layer — Agent Control Plane, State Ledger & xAI Runtime
The Platinum Layer is the operational hub for all autonomous multi-agent systems. It governs how AI agents interact with the data platform, enforces compliance boundaries, manages memory states, and provides an immutable audit log. It is also the primary Explainable AI (xAI) runtime, where every reasoning step, memory access, and tool invocation is captured as a verifiable, human-readable execution trajectory.
Apache-Powered Execution
All agent tool calls, LLM responses, Chain-of-Thought (CoT) thinking traces, reflection loops, and state transitions are published as structured events to a dedicated Kafka topic.
Every Kafka agent event is micro-batched into an append-only Iceberg table. Compliance teams and clinicians can query execution trajectories with full time-travel capability.
Produces a structured MEP containing: (1) trace ID, (2) memory reads with similarity scores, (3) extracted CoT steps, (4) tool call I/O, and (5) post-generation critique results.
Complex Event Processing on the agent event stream detects anomalous behavior (excessive calls, goal drift, policy violations) and triggers automated circuit breakers.
A versioned registry of all MCP tool contracts (OpenAPI schemas). Agents reference tool versions by contract ID, ensuring deterministic behavior and auditability.
NATS JetStream provides inter-agent messaging for supervisor/worker patterns, handoff workflows, and distributed task queues.
Traces every agent interaction across microservices — LLM calls, tool invocations, memory accesses — exporting to Tempo for visualization in Grafana.
NeMo GuardRails runs as a sidecar container, enforcing content safety, PHI detection, prompt injection protection, and hallucination detection before output delivery.
Serves the read side of agent state for rapid HITL checkpoint querying, session replay UI, and operational dashboards.
Rationale
Claude Cowork: Desktop Productivity Layer
Claude Cowork is Anthropic’s desktop agentic system for knowledge work — file organization, report assembly, spreadsheet analysis, and document synthesis. It operates on local files, folders, and connected applications, completing multi-step tasks with user-defined goals and approval gates.
Critical Compliance Boundary
Cowork’s Role at Each Medallion Layer
| Layer | Cowork Access | What It Actually Does |
|---|---|---|
| Bronze | No | Raw device telemetry and PHI-bearing streams never enter a Cowork-accessible folder. |
| Silver | Indirectly, on exports only | Data stewards export de-identified schema samples; Cowork checks for null-policy violations, format drift, and flags anomalies. Never has direct folder/connector access to live Silver stores. |
| Gold | Yes, heavily | Transforms exported Gold-layer CSV/Excel extracts (KPIs, star-schema rollups) into narrative reports, slide decks, and recurring briefings for department analysts and quality officers. |
| Diamond | Yes, as curation assistant | Before documents (protocols, guidelines, SOPs) are embedded into the vector index, Cowork organizes, deduplicates, standardizes formatting, and flags stale versions. Does not run the embedding pipeline itself. |
| Platinum | Documentation only | Drafts and versions tool-contract documentation, Supervisor-Critic evaluation rubrics, and audit-report summaries from exported ledger extracts — never executes governed clinical actions. |
Foundation Setup Requirements
Before any department uses Cowork, implement these platform-wide controls:
Internal MCP Connector Strategy — Build one internal MCP connector exposing read-only endpoints against Gold and Diamond layers (e.g.,
get_kpi_export(department, date_range),search_guidelines(query, department)). This connector calls through existing Apache Trino/vector-search services and inherits Apache Ranger’s row/column/department policies.Department Folder Scoping — Create shared-drive folder trees per department containing only:
- Weekly/monthly Gold KPI exports (CSV/XLSX) written by existing Airflow DAGs
- Department document libraries (protocols, guidelines, policy PDFs)
- A
working/subfolder Cowork is allowed to write into
Private Plugins per Department — Bundle skills (templates, tone, department conventions), connectors, and sub-agents into one install per department so users get context-specific tools by default.
Approval-Required Default — Keep approval-before-action on for anything leaving the department. Only relax for narrowly scoped, non-PHI, fully reversible tasks.
BAA Boundary as Hard Constraint — Treat the BAA exclusion as a design constraint, not a hygiene issue. Cowork must never see PHI — not “PHI with compensating controls,” not “PHI with extra logging.” The de-identification must be complete and verified before data lands in any Cowork-accessible folder.
SIEM Integration — Route Cowork’s OpenTelemetry activity stream (tool calls, file access, approval states) into existing SIEM/GRC tooling and review on the same cadence as other access logs. Note that this doesn’t resolve the BAA coverage question — it’s for operational monitoring only.
Edge De-identification & Multi-Tiered Memory
When orchestrating agents across a multi-tenant enterprise ecosystem, real-time compliance is managed by containerized Pre-LLM and Post-LLM Hooks. Deployed as sidecars or microservices, these hooks intercept payloads directly at the MCP Protocol Layer boundary:
- PII Redaction: Executes permanent, irreversible erasure of direct identifiers from unstructured text inputs during Pre-LLM ingestion.
- Tokenization: Swaps sensitive clinical fields with format-preserving random tokens, managed through an isolated, vault-backed microservice restricted by OPA sidecars.
- Pseudonymization: Replaces patient identifiers with artificial aliases under strict cryptographic governance, allowing longitudinal tracking without exposing identity.
- Masking: Applies structurally consistent placeholders to preserve database formats for non-production environments.
- Anonymization: Performs irreversible data destruction on demographics, protecting downstream aggregated research datasets.
- Synthetic Data Generation: Programmatically outputs statistically faithful, zero-PII datasets to execute prompt engineering and QA testing safely.
Multi-Tiered Persistent Memory Architecture
To execute complex, long-running clinical workflows across shifts, sessions, and physical locations, agents utilize a multi-tiered memory infrastructure running on Kubernetes:
- Working Memory: Fast, ephemeral workspace mapped to the agent pod’s active in-context window. Tracks active conversation threads and short-term reasoning traces.
- Episodic Memory: A timestamped, persistent record log stored in a distributed document tier, preserving clinical actions across session boundaries.
- Semantic Memory: Generalized organizational rules, local medical knowledge bases, and clinician preferences, kept factually consistent over time.
- Procedural Memory: Enforced standard operating procedures (SOPs), specialized clinical routing rules, and skill sets (e.g. SKILL.md libraries) stored as config maps or mounted volumes.
- Vector & Graph Stores: Milvus/Qdrant databases execute semantic similarity searches, while temporal graph databases (Neo4j) manage entity relationships and multi-hop reasoning.