AURUM

Raw data in. Hallmarked golden records out.

AURUM is a vendor-agnostic Master Data Management reference implementation. Five stages — named for the journey from raw ore to hallmarked gold — cover the full MDM lifecycle across 7 domains.

Status: v0.2.1. This is an active reference build. Some components are working code, some are scaffolded interfaces, some are planned for v0.2.0. The Component Status table tells you exactly what's what — no overpromises. See ROADMAP.md for v0.2.0 commitments.

Why AURUM?

Traditional MDM programmes share a common shape: a vendor, an SI, a long delivery timeline, and knowledge that walks out the door when the consultants do. AURUM was built because that pattern keeps repeating.

The Knowledge Problem

Time to Ship a DQ Rule

The Invisible Entity Problem

What You're Actually Buying

Diagrams

Visual walkthroughs of the pipeline, decision model, and knowledge graph layer — all render natively on GitHub:

📊 docs/diagrams/AURUM_DIAGRAMS.md

Includes: AURUM 5-stage pipeline · Three-tier stewardship model · gbrain memory layer · Nightly dream cycle · Knowledge compounding curve

❓ docs/AURUM_FAQ.md

Common questions answered in plain English: Where is data stored? · Enterprise deployment options · Air-gapped setup · Does data go to OpenAI? · Is it production-ready? · What does gbrain add?

The Five Stages

In metallurgy, gold starts as raw ore. It is assayed, unearthed, refined, unfurled into the world, and hallmarked with proof of provenance. MDM follows the same arc — and every stage name tells you what happens there.

Source Systems
      │
      ▼
┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
│  ASSAY   │──▶│ UNEARTH  │──▶│  REFINE  │──▶│  UNFURL  │──▶│   MARK   │
│          │   │          │   │          │   │          │   │          │
│ Test the │   │ Surface  │   │ Fuse many│   │ Issue to │   │ Stamp    │
│ raw ore  │   │ what's   │   │ into one │   │ the world│   │ lineage  │
│          │   │ buried   │   │ golden   │   │          │   │ &        │
│          │   │          │   │ record   │   │          │   │ provenance│
└──────────┘   └──────────┘   └──────────┘   └──────────┘   └──────────┘
                                    │
                            Golden Record Store

Domains

Note: All 7 domain profilers ship working in v0.2.0. Sample data covers all 7 domains end-to-end (600–300 rows each, deliberately dirty). Domain-specific matchers (SKU normalization, legal-entity disambiguation, lifecycle-aware matching) remain on the v0.3.0 roadmap.

Component Status

Three statuses, no fudge:

• ✅ Working — runs end-to-end, tested, demoable
• 🔧 Stub — interface exists, full logic deferred
• 📋 Planned — directory exists, work scheduled (see ROADMAP.md)

ASSAY — Stage 01

UNEARTH — Stage 02

REFINE — Stage 03

UNFURL — Stage 04

MARK — Stage 05

Runtimes

Power Platform Companion Track

Certification Program

Demo

AI / ML in v0.1.3 — what's actually there

The repo claims AI augmentation across stages. In v0.1.3 the working AI/ML components are:

• REFINE classical ML matching — RapidFuzz token scoring + Jellyfish Jaro-Winkler + composite weighting. Deterministic, fast, well-understood. Not LLM-based — and that's the right choice for matching at scale.
• UNEARTH Isolation Forest anomaly detector — flags rows whose feature combinations are unlike the rest of the dataset. Generic per-column feature engineering (length, character composition, null state) so it works across all 7 domains without per-domain tuning. Deterministic.
• UNEARTH LLM rule generator — translates steward prose ("phone numbers must be UAE-format if country is UAE") into deterministic Python rules via the Anthropic SDK with tool-use structured output. Saves to a generated/ directory for human review; never auto-promotes. AST safety guards reject imports, eval/exec, and missing check functions.
• MCP server — exposes the AURUM pipeline as an MCP-compatible server invokable from Claude Code, Cursor, and any Hermes/Nous-based agentic runtime.

LLM-based components (rule generation, anomaly explanation, match tiebreakers) are designed and scaffolded but not yet implemented. They land in v0.2.0. The architectural choice — where LLMs add value vs. where they add risk in MDM — is documented in docs/architecture/ai-strategy.md.

Quickstart

git clone https://github.com/RajaMDM/AURUM.git
cd AURUM
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python shared/sample_data/generate_all.py
python demo/end_to_end_demo.py

The demo runs ASSAY → UNEARTH → REFINE → UNFURL → MARK end-to-end with assertions for cluster integrity, frankenrecord detection, and trust scoring.

CLI usage

python -m runtimes.cli --help
python -m runtimes.cli assay shared/sample_data/output/customers_dirty.csv
python -m runtimes.cli unearth customer shared/sample_data/output/customers_dirty.csv
python -m runtimes.cli anomaly shared/sample_data/output/customers_dirty.csv --domain customer
python -m runtimes.cli demo

Every CLI command supports --json-out for machine-readable output.

HTTP API

uvicorn runtimes.api.main:app --reload --port 8000
# OpenAPI docs:  http://localhost:8000/docs
# ReDoc:         http://localhost:8000/redoc

# Examples
curl -F file=@shared/sample_data/output/customers_dirty.csv http://localhost:8000/assay
curl -F file=@shared/sample_data/output/customers_dirty.csv http://localhost:8000/unearth/customer
curl -F file=@shared/sample_data/output/customers_dirty.csv "http://localhost:8000/anomaly?domain=customer"

Repository Structure

AURUM/
├── shared/              # Domain models · Sample data · Utilities
├── assay/               # Stage 01: Ingestion, schema mapping, migration
├── unearth/             # Stage 02: Profiling, DQ, anomaly detection
├── refine/              # Stage 03: Matching, mastering, golden record
├── unfurl/              # Stage 04: Publication, APIs, subscriptions
├── mark/                # Stage 05: Reverse integration, lineage
├── runtimes/            # CLI · FastAPI · MCP · Streamlit · Orchestration
├── power-platform/      # Dataverse schemas · Power Automate · AI Builder
├── certification/       # Foundation → Practitioner → Architect
├── docs/                # Architecture · Methodology · Roles & Agents
└── demo/                # End-to-end pipeline walkthrough

MCP Compatibility

The AURUM pipeline is invocable as an MCP server with three working tools:

python runtimes/mcp/server.py

Compatible with Claude Code, Cursor, Hermes/Nous, and any MCP-compliant runtime.

Use-Case Library

The use_cases/ directory contains 41 real-world MDM scenario playbooks across all 7 domains — organised in three tiers: single-domain, cross-domain pairs, and a grand scenario where all 7 domains interact.

→ Browse all 41 use cases

Narratives & Deep Dives

Contributing

See CONTRIBUTING.md. High-priority contributions for v0.2.0:

• Domain-specific matchers (Product SKU normalization, Vendor legal-entity disambiguation, Asset lifecycle-aware matching)
• LLM rule generator for UNEARTH
• ML anomaly detector for UNEARTH
• Reverse sync engine for MARK
• Power Automate flow exports
• Real Power Apps (model-driven) screenshots

Author

Built and maintained by Raja Shahnawaz Soni — Enterprise Data Management Practitioner, speaker at Informatica World 2023, and builder of The MDM Lab, BrainDrop, SYNAPTIQ, QualIQ, and Agents for Good.

"Anyone can describe a system. I'd rather hand you a working one and say — here, try it."

License

MIT © Raja Shahnawaz Soni — see LICENSE