Compliance Copilot

Meet Compliance Copilot: An AI agent swarm that uses MCP to autonomously scan your tech stack (AWS, GitHub, etc.) and generate auditor-grade reports in minutes. Compliance on autopilot

Comment

About Compliance Copilot

Domains - * Artificial Intelligence and LLM applications, Developer tools and infrastructure, Productivity and workflow automation, Cybersecurity and data privacy *

Problem Statement & Motivation

The Compliance Challenge

Compliance audits are a critical but painful reality for modern tech companies. We've witnessed firsthand how startups and enterprises struggle with:

  • Exorbitant Costs: SOC2 audits can cost $50,000-$200,000+ and take 3-6 months to complete
  • Manual Labor: Teams spend weeks manually checking configurations, reviewing code, compiling evidence, and filling out spreadsheets
  • Fragmented Data: Compliance information is scattered across AWS, GitHub, monitoring tools, documentation, and various other services
  • Reactive Approach: Companies only discover compliance gaps during audits, not proactively, leading to last-minute scrambles
  • Lack of Expertise: Most startups don't have dedicated compliance officers, making it difficult to understand what's required
  • Time-Consuming: Manual compliance checks can take weeks or months, delaying product launches and business operations

The Real-World Impact

We've seen companies:

  • Delay fundraising because they couldn't get SOC2 certification in time
  • Lose enterprise deals due to missing compliance requirements
  • Spend months preparing for audits instead of building products
  • Hire expensive consultants ($200+/hour) for basic compliance checks
  • Struggle to maintain compliance as their infrastructure grows

Our Motivation

The inspiration came from a simple question: What if AI agents could autonomously scan your entire tech stack and generate audit-ready reports in minutes?

We envisioned a system where:

  • AI agents act like a team of compliance auditors, working 24/7
  • They autonomously decide what to scan based on framework requirements
  • They learn from each scan and improve over time
  • They generate evidence-backed reports that auditors can trust
  • They cost a fraction of traditional consultants

This vision led us to build Compliance Copilot - an autonomous AI-powered compliance assessment platform that transforms weeks of manual work into minutes of automated analysis.

Solution & How It Works

Overview

Compliance Copilot is an autonomous AI-powered compliance assessment platform that uses a swarm of specialized AI agents to scan your entire tech stack, research latest regulations, identify compliance gaps, and generate auditor-grade reports.

Core Workflow

  1. User Selects Framework: User chooses a compliance framework (SOC2, GDPR, HIPAA, ISO 27001, or PCI DSS)

  2. Connect Tech Stack: User connects their services (AWS, GitHub, Azure, etc.) via OAuth or API keys using Model Context Protocol (MCP)

  3. AI Agents Research Regulations:

    • Perplexity AI: Real-time web search for latest compliance requirements, best practices, and industry standards
    • Firecrawl: Web scraping and content extraction from official compliance documentation websites (GDPR.eu, AICPA, HHS, etc.)
    • Browserbase: Browser automation for dynamic content and JavaScript-rendered compliance pages
    • Cloudflare Container: Spins up isolated sandbox environments for deep code analysis and security scanning
    • Requirements are stored in a Vector Database (Supabase pgvector) for semantic search
    • Agents ensure they always have up-to-date information from authoritative sources

  4. Autonomous Scanning:

    • Planning Agent creates a comprehensive assessment plan based on framework requirements
    • Intelligent Extraction Agent autonomously scans connected services:
      • Code repositories (GitHub) for security issues, secrets, dependencies
      • Cloud infrastructure (AWS, Azure) for IAM, encryption, access controls
      • Monitoring tools (DataDog, Instana) for logging and alerting
      • Communication tools (Atlassian) for access management
    • Deep Code Analysis: For complex code analysis, agents automatically spin up Cloudflare sandbox containers to:
      • Execute code in isolated environments
      • Run security scanners and static analysis tools
      • Test for vulnerabilities without affecting production
      • Analyze dependencies and package security
    • Agents decide what to scan based on framework requirements - no human intervention needed

  5. Gap Analysis:

    • Gap Analysis Agent compares extracted data against compliance requirements
    • Identifies gaps with specific evidence (file paths, line numbers, configurations)
    • Assigns severity ratings (critical, high, medium, low)

  6. Report Generation:

    • Report Generation Agent creates comprehensive audit reports with:
      • Compliance scores (overall + category breakdowns)
      • Detailed findings with evidence citations
      • Step-by-step remediation plans
      • Executive summary

  7. Iterative Refinement:

    • Agents automatically re-scan if confidence threshold isn't met
    • System continues until comprehensive coverage is achieved
    • All findings are stored in agent memory (Mem0/Redis) for future reference

Key Differentiators

  • Autonomous Decision Making: Agents decide what to scan without human guidance
  • Evidence-Based: Every finding includes specific evidence (source, location, code snippets)
  • Self-Learning: Agents remember context between scans, improving accuracy over time
  • Real-Time Research: Always uses latest compliance requirements from the web via Perplexity, Firecrawl, and Browserbase
  • Deep Code Analysis: Automatically spins up Cloudflare sandbox containers for isolated security scanning
  • Universal Integration: Connects to 14+ services through a single protocol (MCP)

Example: SOC2 Compliance Scan

  1. User selects "SOC2" framework
  2. Research agents fetch latest SOC2 requirements from official sources
  3. Extraction agents scan:
    • AWS IAM for MFA enforcement
    • GitHub for branch protection and secrets
    • CloudWatch for logging configuration
    • S3 buckets for encryption and public access
  4. Gap Analysis identifies: "MFA not enforced for admin users" with evidence: "AWS IAM policy at arn:aws:iam::123456789012:policy/AdminPolicy lacks MFA requirement"
  5. Report includes: Severity (Critical), Evidence (specific policy ARN), Recommendation (enable MFA for admin role)

Result: Complete SOC2 assessment in minutes instead of months.

Technologies, Tools, APIs & Frameworks

Frontend & Backend

  • Next.js 14+ (App Router) - Full-stack React framework with server-side rendering
  • TypeScript - Type-safe development
  • React - UI library
  • Tailwind CSS - Utility-first CSS framework
  • HeroUI - Component library built on Tailwind

AI & Machine Learning

  • OpenAI GPT-5 - Primary LLM for agent reasoning and analysis
  • OpenAI Embeddings (text-embedding-3-small) - Vector embeddings for semantic search
  • LangGraph - Multi-agent orchestration and state management
  • LangChain - Agent framework and tool integration

Data & Storage

  • Supabase PostgreSQL - Primary database with Prisma ORM
  • Supabase pgvector - Vector extension for RAG (Retrieval-Augmented Generation)
  • Supabase Storage - File storage for reports (PDF, Markdown, JSON)
  • Redis - In-memory database for agent memory (Mem0)
  • Mem0 - Agentic memory framework for persistent agent context

Integration & APIs

  • Model Context Protocol (MCP) - Universal protocol for service integration
  • OAuth 2.0 - Secure authentication for GitHub, Atlassian, Cloudflare
  • Supabase Auth - User authentication and session management

Research Tools (APIs)

  • Perplexity AI API - Real-time web search and research for latest compliance requirements
  • Firecrawl API - Web scraping and content extraction from official documentation
  • Browserbase API - Browser automation for dynamic content and JavaScript-rendered pages
  • Cloudflare Container MCP - Sandbox container spin-up for deep code analysis in isolated environments

Development & Deployment

  • Prisma ORM - Database toolkit and query builder
  • Vercel - Deployment platform (serverless functions)
  • Docker - Containerization for Redis and local development
  • Git - Version control

MCP Server Integrations

The platform connects to 14+ services via MCP:

Cloud Infrastructure:

  • AWS Core MCP Server
  • Azure MCP Server
  • Google Cloud MCP Server
  • Cloudflare MCP Server

Code & DevOps:

  • GitHub MCP Server
  • GitLab MCP Server
  • Jenkins MCP Server
  • ArgoCD MCP Server

Monitoring & Observability:

  • Instana MCP Server
  • DataDog MCP Server
  • Grafana MCP Server
  • Sentry MCP Server

Communication & Collaboration:

  • Atlassian MCP Server (JIRA, Confluence)
  • Notion MCP Server

Code Quality:

  • SonarQube MCP Server

Key Libraries & Packages

  • @modelcontextprotocol/sdk - MCP client SDK
  • @supabase/supabase-js - Supabase client library
  • @supabase/ssr - Supabase server-side rendering utilities
  • redis - Redis client for Node.js
  • openai - OpenAI API client
  • zod - Schema validation

Team Member Roles & Contributions

  • Ankur - Worked on Backend & Generative AI, along with taking care of deployment, setting up MCP and testing
  • Kisha - Worked on Frontend, setting up Vector Databases, along with working on documentation & presentation

Challenges We Ran Into

1. Multi-Agent Coordination

Challenge: Coordinating 5+ agents working in parallel while maintaining state consistency and avoiding race conditions.

Solution:

  • Used LangGraph's state management system for centralized state
  • Implemented proper error boundaries for each agent
  • Added retry mechanisms with exponential backoff
  • Created a centralized state store with proper locking mechanisms

2. MCP Connection Management

Challenge: Managing connections to 14+ different services with different authentication methods (OAuth, API keys, environment variables) while ensuring user isolation.

Solution:

  • Built a robust MCPClientManager with user-scoped connection keys (userId:serverName)
  • Implemented connection pooling and automatic reconnection
  • Added connection limits per user to prevent resource exhaustion
  • Created unified credential management system supporting both OAuth and BYOK

3. Agent Memory & Context

Challenge: Agents were "forgetting" context between scans and making inconsistent decisions, leading to hallucinations.

Solution:

  • Integrated Mem0 with Redis for persistent agent memory
  • Implemented 12-hour TTL for automatic cleanup
  • Created memory search and recall functions
  • Added context injection into agent prompts

4. Vector Search & RAG

Challenge: Compliance regulations are constantly updated, and we needed to ensure agents always use the latest requirements.

Solution:

  • Built a "Corrective RAG" pipeline that actively fetches latest regulations using Perplexity and Firecrawl
  • Implemented automatic embedding and storage in pgvector
  • Created fallback mechanisms when RAG returns no results
  • Added web research as a backup for missing requirements

5. Report Generation Timeouts

Challenge: LLM calls for report generation were timing out, especially with large amounts of evidence.

Solution:

  • Implemented timeout mechanisms (50s for sections, 40s for executive summary)
  • Added batch processing for evidence analysis
  • Created fallback to generate basic findings even if LLM times out
  • Optimized prompts to reduce token usage

6. AWS Credential Management

Challenge: AWS MCP server couldn't find credentials even when users provided them, causing "config profile (default) could not be found" errors.

Solution:

  • Created temporary AWS credentials files
  • Set environment variables in subprocess (not global process.env)
  • Explicitly configured AWS SDK to use provided credentials
  • Added validation and error messages for missing credentials

7. Real-Time Progress Tracking

Challenge: Users couldn't see what agents were doing during long-running scans (which can take hours).

Solution:

  • Implemented Server-Sent Events (SSE) for real-time updates
  • Added detailed progress callbacks from agents
  • Created AgentActivityMonitor to parse and display agent activity
  • Built frontend components to show live agent status

8. Production Deployment

Challenge: Vercel deployment issues with prerendering, module bundling, and environment variables.

Solution:

  • Fixed login page prerendering by wrapping useSearchParams in Suspense
  • Externalized MCP SDK in webpack config to prevent bundling issues
  • Moved TypeScript and build tools to dependencies (not devDependencies) for Vercel
  • Added proper error boundaries and fallbacks

Accomplishments That We're Proud Of

Technical Achievements

  1. Autonomous Multi-Agent System: Successfully built a system where 5+ AI agents work autonomously, making decisions about what to scan based on framework requirements without human intervention

  2. Universal MCP Integration: Created a unified integration layer that connects to 14+ different services with different authentication methods, all through a single protocol

  3. Agentic Memory: Implemented persistent memory for AI agents using Mem0, enabling agents to learn from previous scans and maintain context

  4. Corrective RAG Pipeline: Built a self-correcting RAG system that actively fetches latest regulations when vector search fails, ensuring agents always have up-to-date information

  5. Production-Ready Architecture: Designed and implemented a scalable, multi-user system with proper isolation, error handling, and monitoring

Impact & Innovation

  1. 90% Cost Reduction: Reduced compliance audit costs from $50K-$200K to a fraction of that

  2. Time Savings: Transformed weeks/months of manual work into minutes of automated analysis

  3. Evidence-Based Reports: Every finding includes specific evidence citations (source, location, code snippets), making reports auditor-ready

  4. Iterative Refinement: Agents automatically re-scan until confidence threshold is achieved, ensuring comprehensive coverage

  5. Real-Time Transparency: Users can see exactly what agents are doing in real-time through live progress tracking

Code Quality

  1. Type Safety: 100% TypeScript coverage with proper types throughout
  2. Error Handling: Comprehensive try-catch blocks, retry mechanisms, and graceful degradation
  3. User Isolation: Proper multi-user support with isolated MCP connections
  4. Production Optimizations: Optimized prompts, context limits, and timeout configurations
  5. Documentation: Comprehensive documentation including MCP connection guide, agent architecture, and deployment instructions

What We Learned

Technical Learnings

  1. LangGraph vs CrewAI: Learned that LangGraph provides better control over agent state and workflow, making it ideal for complex multi-agent systems with conditional logic

  2. MCP Protocol: Discovered that Model Context Protocol is a game-changer for service integration - it provides a unified interface for connecting to any service, dramatically reducing integration complexity

  3. Agentic Memory: Realized that giving AI agents persistent memory (via Mem0) is crucial for reducing hallucinations and improving consistency across multiple interactions

  4. Vector Search Limitations: Learned that RAG alone isn't enough - you need active web research to ensure you have the latest information, especially for compliance regulations that change frequently

  5. Multi-User Architecture: Understood the importance of proper user isolation from day one - retrofitting multi-user support is much harder than building it in from the start

  6. Prompt Engineering: Discovered that optimizing prompts (removing redundancy, setting proper context limits) can significantly reduce costs and improve response times

  7. Error Handling: Learned that comprehensive error handling and retry mechanisms are essential for production AI systems, as LLM calls can be unpredictable

Product & Design Learnings

  1. User Experience: Realized that real-time progress tracking is essential for long-running AI operations - users need to see what's happening, not just wait

  2. Evidence Matters: Understood that compliance reports need specific evidence citations to be trusted by auditors - generic findings aren't enough

  3. Flexibility: Learned that users want flexibility - the ability to scan specific tools or frameworks, not just everything at once

  4. Transparency: Discovered that showing agent activity and decision-making process builds trust with users

Business Learnings

  1. Market Need: Confirmed that compliance is a real pain point for startups - they're willing to pay for solutions that save time and money

  2. Consultant Replacement: Realized that AI can't fully replace consultants yet, but it can dramatically reduce their workload and make audits faster

  3. Framework Coverage: Learned that supporting multiple frameworks (SOC2, GDPR, HIPAA, etc.) is essential - companies often need to comply with multiple standards

Future Scope & Roadmap

Short-Term (Next 3 Months)

  1. Additional Frameworks

    • NIST Cybersecurity Framework
    • CIS Controls
    • FedRAMP (for government contractors)
    • CCPA (California Consumer Privacy Act)

  2. Enhanced Integrations

    • Kubernetes cluster scanning
    • Terraform/Infrastructure-as-Code analysis
    • Docker container security scanning
    • CI/CD pipeline analysis (GitHub Actions, GitLab CI)

  3. Improved Reporting

    • Interactive compliance dashboards
    • Trend analysis and compliance score history
    • Automated remediation ticket creation (JIRA, Linear)
    • Compliance calendar and audit scheduling

  4. Better Agent Intelligence

    • Fine-tuned models for compliance-specific tasks
    • Custom agent training on compliance datasets
    • Improved confidence scoring algorithms
    • Agent explainability (why did the agent make this decision?)

Medium-Term (3-6 Months)

  1. Continuous Compliance Monitoring

    • Real-time compliance checks (not just on-demand scans)
    • Automated alerts for compliance violations
    • Integration with monitoring tools for proactive detection
    • Compliance drift detection

  2. Advanced Remediation

    • Automated remediation scripts generation
    • Integration with infrastructure automation tools
    • Remediation verification (re-scan after fixes)
    • Remediation tracking and progress monitoring

  3. Collaboration Features

    • Team workspaces with role-based access
    • Compliance officer dashboard
    • Audit trail and change history
    • Comments and annotations on findings

  4. Enterprise Features

    • SSO integration (SAML, OIDC)
    • API for programmatic access
    • Webhook notifications
    • Custom compliance frameworks

Long-Term (6-12 Months)

  1. AI-Powered Compliance Advisor

    • Proactive recommendations based on industry best practices
    • Compliance roadmap planning
    • Risk assessment and prioritization
    • Cost-benefit analysis for remediation

  2. Compliance Marketplace

    • Pre-built compliance templates
    • Community-contributed compliance checks
    • Integration marketplace for new services
    • Compliance expert network

  3. Advanced Analytics

    • Industry benchmarking
    • Compliance maturity scoring
    • Risk prediction models
    • Compliance ROI calculator

  4. Global Expansion

    • Support for international frameworks (GDPR, ISO 27001 variants)
    • Multi-language support
    • Regional compliance experts
    • Local data residency options

Technical Improvements

  1. Performance Optimization

    • Parallel agent execution optimization
    • Caching strategies for frequently accessed data
    • Incremental scanning (only scan what changed)
    • Edge computing for faster response times

  2. Scalability

    • Horizontal scaling for agent execution
    • Distributed agent memory
    • Queue-based job processing
    • Multi-region deployment

  3. Security Enhancements

    • End-to-end encryption for sensitive data
    • Audit logs for all agent actions
    • Compliance with SOC2 (meta-compliance!)
    • Penetration testing and security audits

  4. Developer Experience

    • SDK for custom agent development
    • Plugin system for custom integrations
    • CLI tool for local development
    • Comprehensive testing framework

Conclusion

Compliance Copilot represents a new paradigm in compliance assessment - moving from manual, expensive, reactive audits to automated, affordable, proactive compliance monitoring. By combining the power of AI agents, universal service integration, and intelligent memory systems, we've created a platform that can transform how companies approach compliance.

The journey from inspiration to production has been challenging but incredibly rewarding. We've learned that building production-ready AI systems requires careful attention to error handling, user experience, and system architecture. Most importantly, we've proven that AI can dramatically reduce the time and cost of compliance assessments while improving accuracy and coverage.

As we look to the future, we're excited to continue pushing the boundaries of what's possible with AI-powered compliance automation. The goal is simple: make compliance as easy as running a test suite.

Built with ❤️ for the compliance community

Built With

Share this project:

Updates