About the Project

Goodman is a compliance intelligence workbench for India's Digital Personal Data Protection Act. It scans database schemas, classifies PII fields, maps legal obligations, detects cross-law conflicts, grounds claims against legal text, and generates audit-ready compliance artifacts, all with ML-powered purpose inference.

Inspiration

India's DPDP Act (2023) and its rules (2025) create a new compliance layer for any org handling personal data. Most tools are manual questionnaires or generic GDPR templates. We wanted a tool that actually understands your schema, your column names, your data types, and tells you what obligations you trigger, where your risks are, and what documents you need to generate.

What it does

Upload a .sql, .csv, or .json schema and get a full compliance package:

• Data inventory with PII categories, sensitivity, and purpose
• Obligation mapping from 30+ DPDP controls
• Conflict detection across IT Act, DPDP, and sector rules
• Grounding report with claim-level confidence scores
• ML purpose inference (sklearn TF-IDF + LogisticRegression, 720 training examples across 6 domains)
• Indic language summary via Sarvam AI
• Delta Lake audit log of all scans
• MLflow experiment tracking
• Compliance kit ZIP with consent packages, breach playbooks, retention policies, and DPA templates

How we built it

• FastAPI backend with an async scan endpoint and a React/Vite dashboard
• Discovery engine parses SQL CREATE TABLE, CSV headers, JSON schemas
• PII classifier uses a 5-tier taxonomy (name, email, phone, financial, health, etc.) with sector-specific heuristics
• Purpose classifier runs TF-IDF + LogisticRegression (sklearn) across 6 purpose categories: payments, kyc, care delivery, marketing, employment, service delivery
• Grounding verifier uses sentence-transformers to score obligation claims against the official DPDP Rules text
• Artifact generator produces Markdown templates filled with scan results
• Sarvam AI generates multilingual executive summaries
• Delta Lake / deltalake persists scan history; MLflow tracks experiments
• Deployed as a Databricks App with bundled deployment

Challenges we ran into

• Databricks Apps run in a sandboxed container without Spark access, so we had to build a sklearn fallback that loads a pre-trained pipeline rather than relying on real-time Spark MLlib
• PySpark 4.x breaks array_max() on probability columns, so we pinned to pyspark==3.5.1
• Databricks bundle deploy/auth profile mismatch required explicit --profile flags on every command

Accomplishments that we're proud of

• End-to-end schema to compliance kit in one API call
• Pre-trained purpose classifier (75KB, 836 features) that loads instantly from disk with no per-request training
• Real grounding with transformer embeddings against actual DPDP Rules text
• Fully deployed on Databricks Cloud with Delta Lake persistence

What we learned

• Databricks Apps are great for lightweight web services but the Spark runtime isolation means the "Spark as compute core" story requires a separate job submission path
• Synthetic training data with domain vocabulary is surprisingly effective for bag-of-words ML on schema classification
• Hardening a tool for a real regulatory context (DPDP) forces you to think about legal grounding, not just technical accuracy

What's next for Goodman

• Submit Spark jobs from the App to a real cluster for distributed schema scanning and MLlib training
• Add column-level data profiling (value distributions, cardinality) from actual sample data
• Expand grounding to cover IT Act Section 43A and DPDP Section 16 consent requirements
• Integrate with Databricks Unity Catalog for automatic schema discovery

Built With

• databricks
• faiss
• gradio
• huggingface
• indictrans2
• mlflow
• mllib
• param-1
• particle
• pyspark
• python
• sarvam-m
• transformers