Audit-Ready AI and Data Reference Systems

Reference System

Audit-Ready RAG System

A traceable, citation-grounded retrieval architecture tested against dense USCIS policy documents. The system demonstrates how retrieval, citations, deterministic configuration, and audit records can make AI outputs inspectable in policy-heavy environments.

The Architecture

A RAG pipeline that ingests PDF documents, chunks them with respect to legal hierarchy, stores embeddings in pgvector, and retrieves relevant context to generate grounded answers.

What Makes It Auditable

Source citations on grounded answers, an explicit no-source fallback when retrieval evidence is insufficient, retrieval choices designed to reduce redundancy, and architecture decisions made for repeatability.

Why This Matters

The same architecture pattern can be adapted for compliance documents, internal knowledge bases, policy libraries, and other controlled document collections.

Architecture Decisions

pgvector over Pinecone: SQL-joinable, no new infrastructure dependency, lower cost, and the data stays in your environment.
MMR retrieval over plain similarity: Maximal Marginal Relevance reduces redundancy when retrieving from repetitive legal text.
Temperature 0: Lower-variance model settings for more consistent, repeatable outputs in compliance-oriented use cases.
Recursive chunking: Respects nested legal document hierarchy instead of blindly splitting on token count.
Claude Sonnet 4.6 via AWS Bedrock: Strong performance on dense legal text with an enterprise-friendly deployment path.
Explicit configuration: Model settings, credentials, and data paths are made visible rather than assumed from ambient environment state.

AWS Bedrock Claude Sonnet 4.6 Titan Embeddings v2 PostgreSQL pgvector LangChain FastAPI Docker S3 Python

View Repo → Discuss Your Use Case →

Reference System

ETL Architecture for AI Governance

A working ETL reference implementation for ingesting, classifying, validating, and governing AI-processed data. It demonstrates traceable data movement, AI-assisted classification, data quality gates, audit records, and infrastructure-as-code patterns.

The Architecture

An Airflow-orchestrated pipeline that ingests raw documents, classifies them using Claude on Amazon Bedrock, applies data quality gates via Great Expectations, and lands validated records in Databricks Delta Lake with a record-level audit trail keyed by pipeline run and record ID.

What Makes It Auditable

Each processed record is tied to a pipeline run and audit record. AI classification decisions retain model version, prompt hash, and confidence score. Quality gates block invalid records before they reach governed tables. The local development stack is reproducible from a single Makefile command.

Local-to-Cloud Parity

The stack runs locally through Docker Compose and LocalStack, while Terraform modules document an AWS deployment path. Teams can test DAGs, quality gates, and classification logic without requiring a live client environment.

Architecture Decisions

Airflow over Prefect: The target production environment is Amazon MWAA, which provides a managed Airflow runtime with IAM integration, S3 DAG storage, and CloudWatch logging. Prefect would require self-hosting and rebuilding equivalent AWS integrations.
Databricks Delta Lake over raw S3: ACID transactions, schema enforcement, time travel for audit queries, and a query layer that supports both batch analytics and ad-hoc investigation.
Claude on Bedrock for classification: Enterprise deployment path designed for AWS-based regulated environments, with controlled access, logging, and deployment boundaries.
Great Expectations for quality gates: Declarative validation checks enforce required identifiers and timestamps, allowed document types, agencies and priority tiers, minimum page counts, and minimum document-text length before records continue downstream.
LocalStack for local AWS emulation: S3 runs locally in Docker for development and integration testing, reducing cloud spend and credential exposure while preserving the AWS S3 API surface.
Terraform with remote state: Versioned S3 state backend with DynamoDB locking. Every infrastructure change is tracked, reviewable, and reproducible across environments.
Makefile-driven workflows: One command to stand up the full stack, run tests, lint, or tear down, making the development workflow explicit and repeatable.
Strict code quality tooling: Ruff for linting, mypy for type checking, and pytest for automated tests, enforced in CI.

Apache Airflow 3.2 Amazon Bedrock Claude Sonnet Databricks Delta Lake PostgreSQL Great Expectations Terraform Docker Compose LocalStack Python

View Repo → Discuss Your Use Case →

Apply the same patterns to your environment.

Archetype Core can adapt these reference patterns to governed data pipelines, document workflows, Databricks modernization, and traceable AI systems in a defined client environment.

Discuss Your Use Case

Working reference systems, source available for inspection, documented decisions.

Audit-Ready RAG System

The Architecture

What Makes It Auditable

Why This Matters

Architecture Decisions

ETL Architecture for AI Governance

The Architecture

What Makes It Auditable

Local-to-Cloud Parity

Architecture Decisions

Apply the same patterns to your environment.

Need auditable data infrastructure?