The SynthoraAI - AI-Powered Article Content Curator is a comprehensive, AI-powered system designed to aggregate, summarize, and present curated government-related articles. This monorepo, multi-services project is organized into seven main components:

• Backend: Provides a robust RESTful API to store and serve curated articles.
• Crawler: Automatically crawls and extracts article URLs and metadata from government homepages and public API sources.
• Frontend: Offers an intuitive Next.js-based user interface for government staff (and potentially the public) to browse and view article details.
• Newsletter: Sends daily updates to subscribers with the latest articles.
• Agentic AI Pipeline: Production-hardened multi-agent system on LangGraph + LangChain 1.x — resilience (retry/circuit-breaker/timeout), OpenTelemetry + Prometheus observability, a non-root container image, and Kubernetes/Helm/Terraform artifacts; zero known Python CVEs.
• Chat Orchestration: TypeScript-based dual-provider (Anthropic + Google) chat layer with 16 specialized agents, intent routing, grounding validation, and cost tracking.
• MCP Server + ACP Layer: Model Context Protocol server exposing the agentic pipeline as 28 tools, 14 resources, and 7 prompts (each wrapped by tool_middleware), plus a production-grade Agent Communication Protocol (ACP) for agent-to-agent messaging with Redis-backed multi-replica support.

High-Level Architecture: Each component is maintained in its own directory:

• Backend: backend/
  • Live: https://ai-content-curator-backend.vercel.app/
• Frontend: frontend/
  • Live: https://synthoraai.vercel.app/
• Crawler: crawler/
  • Live: https://ai-content-curator-crawler.vercel.app/
• Newsletter: newsletters/
  • Live: https://ai-content-curator-newsletters.vercel.app/
• Agentic AI Pipeline: agentic_ai/
  • Documentation: agentic_ai/README.md

Additionally, the project includes a set of shell scripts and a Makefile for automating common tasks, as well as a CLI tool for managing crawling and article operations. It also fully supports AWS, Terraform, & Kubernetes deployments with blue/green deployments, canary releases, and rolling updates via GitHub Actions. Enterprise-grade observability is provided through Splunk (via OpenTelemetry Collector and Kinesis Data Firehose), Prometheus, and Grafana.

• Overview
• Architecture
• Collaboration & Agile Workflow with Jira
• User Interface
• Backend
• Crawler
• Frontend
• Newsletter Subscription
• Agentic AI Pipeline
• Article Q&A Feature
• Sitewide AI Chat
• Intelligent Recommendation System
• Command Line Interface (CLI)
• Shell Scripts & Makefile
• Testing
• Continuous Integration / Continuous Deployment (CI/CD)
• Deployment
• License
• Contact
• Conclusion

The SynthoraAI - AI-Powered Article Content Curator system is designed to provide government staff with up-to-date, summarized content from trusted government sources and reputable news outlets. By leveraging AI (Google Generative AI / Gemini) for summarization and using modern web technologies, this solution ensures that users receive concise, accurate, and timely information.

Below is a high-level diagram outlining the system architecture:

flowchart LR
    subgraph Sources[Trusted Sources]
        GovSites[Government Websites]
        NewsAPIs[Public News APIs]
    end

    Sources -->|URLs & Metadata| Crawler
    Crawler[[Crawler Service<br/>Next.js API Routes]] -->|Summaries & Topics| MongoDB[(MongoDB Atlas)]
    Crawler -->|AI Prompts| GoogleAI[(Google Generative AI)]
    MongoDB --> Backend[[Backend API<br/>Next.js + Express]]
    Backend -->|Cached Responses| Redis[(Redis)]
    Backend -->|REST/JSON| Frontend[[Frontend Web App<br/>Next.js + React]]
    Backend -->|Digest Payload| Newsletter[[Newsletter Service<br/>Next.js Serverless]]
    Newsletter -->|Emails| Subscribers[(Subscribers)]
    Frontend -->|Browse & Manage| Users[(Staff & Public Users)]
    Backend -.->|AI-Assisted Features| GoogleAI
    Crawler -.->|Cron & Shell Scripts| Automation[Shell / Make CLI]
    Backend -.->|Operational Scripts| Automation

    subgraph Observability[Observability Stack]
        OTEL[Splunk OTEL Collector]
        Splunk[(Splunk)]
        Prometheus[Prometheus]
        Grafana[Grafana]
    end

    Backend -.->|OTLP + logs| OTEL
    Frontend -.->|OTLP + logs| OTEL
    Crawler -.->|logs| OTEL
    Newsletter -.->|logs| OTEL
    OTEL -->|HEC| Splunk
    OTEL -->|remote write| Prometheus
    Prometheus --> Grafana

To illustrate how articles move through the platform, the following sequence captures the typical daily refresh:

sequenceDiagram
    participant Cron as Vercel Cron / Shell
    participant Crawl as Crawler Service
    participant Back as Backend API
    participant AI as Google Generative AI
    participant DB as MongoDB
    participant UI as Frontend
    participant Mail as Newsletter Service

    Cron->>Crawl: Trigger fetchAndSummarize job
    Crawl->>AI: Request summaries & topics
    AI-->>Crawl: Return condensed insights
    Crawl->>DB: Upsert articles & analytics
    Back->>DB: Query latest curated data
    UI->>Back: Fetch paginated articles
    Mail->>Back: Request latest articles
    Back-->>Mail: Deliver curated digest data
    Mail->>Subscribers: Send newsletter email

This project consists of 4 primary microservices that interact with each other:

1. Crawler:
   • Crawls government homepages and public API sources to extract article URLs and metadata.
   • Uses Axios and Cheerio for static HTML parsing, with Puppeteer as a fallback for dynamic content.
   • Scheduled to run daily at 6:00 AM UTC via a serverless function on Vercel.
   • Provides a basic landing page with information about the crawler and links to the backend and frontend.
   • Deployed on Vercel at https://ai-content-curator-crawler.vercel.app.
2. Backend:
   • Built with Express.js and Next.js, serving as a RESTful API for the frontend.
   • Integrates Google Generative AI (Gemini) for content summarization.
   • Stores articles in MongoDB using Mongoose, with fields for URL, title, full content, summary, source information, and fetch timestamp.
   • Scheduled serverless function to fetch and process new articles daily at 6:00 AM UTC.
   • Deployed on Vercel at https://ai-content-curator-backend.vercel.app.
3. Newsletter Service:
   • Allows users to subscribe to a newsletter for daily updates on the latest articles.
   • Integrated with Resend API for managing subscriptions and sending emails.
   • By default, the newsletter is sent daily at 9:00 AM UTC, from the email address with the sonnguyenhoang.com domain.
   • Deployed on Vercel as a serverless function, at https://ai-content-curator-newsletters.vercel.app.
4. Frontend:
   • Built with Next.js and React, providing a modern, mobile-responsive UI for browsing and viewing curated articles.
   • Fetches and displays a paginated list of articles from the backend API, with filtering options.
   • Dedicated pages for full article content, AI-generated summaries, source information, and fetched timestamps.
   • User authentication for marking articles as favorites, commenting, discussions, and upvoting/downvoting comments.
   • 3D visualizations + components built with React Three Fiber + WebGL and Three.js for an engaging user experience.
   • AI-powered features such as article Q&A, bias analysis, and related article recommendations using vector similarity search with Pinecone.
   • and many more features in the app!
   • Deployed on Vercel at https://synthoraai.vercel.app/.

Additionally, there are 3 advanced AI components:

1. Agentic AI Pipeline:
   • A sophisticated multi-agent system built with Python, leveraging LangGraph and LangChain for advanced content processing. This pipeline handles tasks such as article summarization, topic extraction, bias analysis, and more. It is designed to be modular and extensible, allowing for the addition of new agents and tools as needed. The pipeline is exposed via a FastAPI HTTP bridge for seamless integration with other services.
2. Orchestration Layer:
   • Python (agentic_ai/orchestration/): Enterprise article-processing orchestration atop the LangGraph pipeline — content supervision, cost budgeting, error recovery with circuit breaking, dead-letter queuing, and concurrent batch processing. mypy --strict clean with 100% line coverage across all modules. Exposed via a FastAPI HTTP bridge (agentic_ai/api.py on port 8000) for cross-service integration.
   • TypeScript (orchestration/): Dual-provider LLM chat layer — unified Anthropic + Google client, intent-based agent routing (16 agents), grounding validation, prompt caching, cost tracking, and structured observability. strict TypeScript with 233 Jest tests behind an enforced coverage gate. Integrated into the backend via /api/orchestrator/* endpoints. See orchestration/README.md.
3. MCP Server + ACP Layer (mcp_server/): Model Context Protocol server exposing the agentic pipeline over stdio — 28 tools, 14 resources, and 7 prompts for Claude Code and IDE integration. Includes ACP with Redis-backed agent registry and inter-agent message routing for production multi-replica deployments. Configured via .mcp.json.

This monorepo, microservices architecture is designed to be modular and scalable, allowing for easy updates and maintenance. Each component can be developed, tested, and deployed independently, ensuring a smooth development workflow.

The team follows an Agile (Kanban) workflow tracked in Jira, with a GitHub branch/PR process keyed to Jira issue IDs.

The user interface is built with Next.js and React, providing a modern, mobile-responsive experience. Below are some screenshots of the application (some screenshots may be outdated and not reflect the latest UI - visit https://synthoraai.vercel.app/ for the latest version):

many more pages and features are available in the app - we encourage you to explore!

The Backend is responsible for storing articles and serving them via RESTful endpoints. It integrates AI summarization, MongoDB for storage, and runs within a Next.js environment using Express.js for API routes.

• Data Ingestion:
  Receives article URLs and data from the crawler and external API sources.
• Content Summarization:
  Uses Google Generative AI (Gemini) to generate concise summaries.
• Storage:
  Persists articles in MongoDB using Mongoose with fields for URL, title, full content, summary, source information, and fetch timestamp.
• API Endpoints:
  • GET /api/articles – Retrieves a paginated list of articles (supports filtering via query parameters such as page, limit, and source).
  • GET /api/articles/:id – Retrieves detailed information for a specific article.
• Scheduled Updates:
  A serverless function (triggered twice daily at 6:00 AM and 6:00 PM UTC) fetches and processes new articles, so that the system remains up-to-date!
• User Authentication:
  Supports email + password registration, login, and JWT-based authentication, plus passwordless passkey (WebAuthn / FIDO2) sign-in. Users can sign up with a passkey only, add additional passkeys to an existing account, and manage them (list / rename / delete) at /account/passkeys. Both flows issue the same JWT, so the rest of the system is unchanged.
• Privacy & Data Controls:
  GDPR-style data-subject endpoints — GET /api/privacy/export returns a JSON archive of all data held for the user (profile, favorites, comments, ratings, passkey metadata, newsletter subscription; password hashes and raw keys excluded). Account deletion is a two-step, token-gated flow (POST /api/privacy/request-deletion issues a 1-hour token; DELETE /api/privacy/account verifies it and cascade-deletes the user and all linked records).
• Favorite Articles:
  Authenticated users can mark articles as favorites for quick access.
• Newsletter Subscription:
  Users can subscribe to a newsletter for daily updates on the latest articles. This feature is integrated with a third-party service (Resend) for managing subscriptions and sending emails.
• Bias Detection & Analysis:
  The app also includes a bias detection & analysis feature, powered by Google Generative AI, to analyze articles for potential bias and provide deep article insights to users.
• User Ratings:
  Users can rate articles, allowing for feedback and quality assessment.
• Dark Mode:
  The frontend offers a dark mode option for improved readability and user experience.
• Discussions & Comments:
  Users can also discuss and comment on articles, fostering engagement and collaboration.
• Upvote/Downvote Comments:
  Users can upvote or downvote comments to highlight valuable contributions.

1. Prerequisites:

   • Node.js (v18 or later)
   • MongoDB (local or cloud)
   • Vercel CLI (for deployment)

2. Clone the Repository:

   git clone https://github.com/hoangsonww/AI-Gov-Content-Curator.git
   cd AI-Gov-Content-Curator/backend
   
   # then, fill in the environment variables as described in .env.example file
   # approved team members, please contact me for the actual .env file and API keys!

3. Install Dependencies (inside backend/):

   npm install

Create a .env file in the ROOT directory with the following (this will be shared across all components in this monorepo):

MONGODB_URI=<your_mongodb_uri>
GOOGLE_AI_API_KEY=<your_google_ai_api_key>
GOOGLE_AI_API_KEY1=<your_google_ai_api_key1_optional>
GOOGLE_AI_API_KEY2=<your_google_ai_api_key2_optional>
GOOGLE_AI_API_KEY3=<your_google_ai_api_key3_optional>
AI_INSTRUCTIONS=Summarize the articles concisely and naturally (change if needed)
NEWS_API_KEY=<your_news_api_key>
NEWS_API_KEY1=<your_news_api_key1>
PORT=3000
CRAWL_URLS=https://www.state.gov/press-releases/,https://www.bbc.com/news,https://www.nytimes.com/,https://www.dallasnews.com/news/,https://www.houstonchronicle.com/,,https://www.whitehouse.gov/briefing-room/,https://www.congress.gov/,https://www.statesman.com/news/politics-elections/
AICC_API_URL=https://ai-content-curator-backend.vercel.app/
RESEND_API_KEY=<your_resend_api_key>
RESEND_FROM="AI Curator <your_email>"
UNSUBSCRIBE_BASE_URL=<your_unsubscribe_base_url>

# Auth
JWT_SECRET=<long_random_string>

# WebAuthn / Passkey configuration
# RP_ID MUST equal the FRONTEND apex domain (eTLD+1), not the backend's host.
RP_ID=localhost
RP_NAME=SynthoraAI
RP_ORIGIN=http://localhost:3000,https://synthoraai.vercel.app

Refer to the .env.example file for more details on each variable.

Start the development server:

npm run dev

Access endpoints:

• GET http://localhost:3000/api/articles
• GET http://localhost:3000/api/articles/:id

1. Configure Environment Variables in your Vercel project settings.

2. Create or update the vercel.json in the root of the backend directory:

   {
     "version": 2,
     "builds": [
       {
         "src": "package.json",
         "use": "@vercel/next"
       }
     ],
     "crons": [
       {
         "path": "/api/scheduled/fetchAndSummarize",
         "schedule": "0 6,18 * * *"
       }
     ]
   }

3. Deploy with:

   vercel --prod

The Crawler automatically retrieves article links and metadata from government homepages and public API sources. It uses Axios and Cheerio for static HTML parsing and falls back to Puppeteer when necessary.

• Article Extraction:
  Crawls specified URLs to extract article links and metadata.

• Error Handling & Resilience:
  Implements a retry mechanism and fallback to Puppeteer for dynamic content fetching when encountering issues (e.g., HTTP 403, ECONNRESET).

• Scheduling:
  Deployed as a serverless function on Vercel, scheduled via cron (runs daily at 6:00 AM UTC).

• Next.js UI:
  Provides a basic landing page with information about the crawler and links to the backend and frontend.

1. Prerequisites:

• Node.js (v18 or later)
• NPM (or Yarn)
• Vercel CLI (for deployment)

2. Clone the Repository:

   git clone https://github.com/hoangsonww/AI-Gov-Content-Curator.git
   cd AI-Gov-Content-Curator/crawler

3. Install Dependencies:

   npm install

Start the Next.js development server to test both the UI and crawler function:

npm run dev

• UI: http://localhost:3000/
• Crawler Function: http://localhost:3000/api/scheduled/fetchAndSummarize

Alternatively, run the crawler directly:

npx ts-node schedule/fetchAndSummarize.ts

# or
npm run crawl

Also, there are 2 more scripts for the crawler:

1. Fetch and crawl all past articles (will run indefinitely, unless you stop it):

   npx ts-node scripts/fetchPastArticles.ts
   
   # or
   npm run fetch:past

2. Fetch and crawl all newest/latest articles:

    npx ts-node scripts/fetchLatestArticles.ts
   
    # or
    npm run fetch:latest

Run these locally to test the crawler functionality. You can also run them in a Docker container if you prefer.

1. Set Environment Variables in the Vercel dashboard.

2. Create or update the vercel.json in the crawler directory:

   {
     "version": 2,
     "builds": [
       {
         "src": "package.json",
         "use": "@vercel/next"
       }
     ],
     "crons": [
       {
         "path": "/api/scheduled/fetchAndSummarize",
         "schedule": "0 6 * * *"
       }
     ]
   }

3. Deploy with:

   vercel --prod

The Frontend is built with Next.js and React, providing a modern, mobile-responsive UI for browsing and viewing curated articles.

• Article Listing:
  Fetches and displays a paginated list of articles from the backend API. Supports filtering by source.

• Article Detail View:
  Dedicated pages display full article content, AI-generated summaries, source information, and fetched timestamps.

• Responsive Design:
  The UI is optimized for both desktop and mobile devices.

• Authentication:
  Users can register and log in with email + password, or sign in with a passkey (Face ID, Touch ID, Windows Hello, or a hardware key) for a faster, passwordless experience. New accounts may be created passkey-only. The frontend stores the issued JWT in localStorage and sends it on the Authorization header for protected routes.

• Chatbot Q&A Feature: Users can ask questions about specific articles, powered by RAG (Retrieval-Augmented Generation) using Google Generative AI. The sitewide chat supports inline message editing with conversation branching—edit any prior message and the conversation forks from that point with fresh AI responses.

• Related Articles (Vector Similarity Search):
  Displays related articles based on vector similarity search using embeddings generated by Google Generative AI.

• Article Bias Analysis:
  Analyzes articles for potential bias using Google Generative AI and provides insights to users.

• Article Ratings:
  Users can rate articles, allowing for feedback and quality assessment.

• Discussions & Comments:
  Users can discuss and comment on articles, fostering engagement and collaboration.

• Upvote/Downvote Comments:
  Users can upvote or downvote comments to highlight valuable contributions.

• Search Functionality:
  Users can search for articles using keywords, with results fetched from the backend API.

• Translation Feature:
  Articles can be translated into multiple languages using Google Translate API.

• Favorite Articles:
  Authenticated users can mark articles as favorites for quick access. This is stored in the backend and displayed in the frontend.

• Privacy & Data Controls:
  The /account/privacy page (linked from the auth dropdown) lets users download a JSON archive of all their data and delete their account through a deliberate two-step flow (request → type DELETE → confirm).

• Newsletter Subscription:
  Users can subscribe to a newsletter for daily updates on the latest articles. This feature is integrated with a third-party service (Resend) for managing subscriptions and sending emails.

• Dark Mode:
  The frontend offers a dark mode option for improved readability and user experience.

• Additional UI Components:
  Includes components like HeroSlider, LatestArticles, ThemeToggle, and more for an enhanced user experience.

• Static Site Generation (SSG):
  The frontend uses Next.js's SSG capabilities to pre-render pages for improved performance and SEO.

1. Prerequisites:

• Node.js (v18 or later)
• NPM or Yarn

2. Clone the Repository:

   git clone https://github.com/hoangsonww/AI-Gov-Content-Curator.git
   cd AI-Gov-Content-Curator/frontend

3. Install Dependencies:

   npm install

   or

   yarn

(Optional) Create a .env.local file in the frontend directory to configure the API URL:

NEXT_PUBLIC_API_URL=https://your-backend.example.com

Start the development server:

npm run dev

Access the application at http://localhost:3000.

1. Configure Environment Variables in the Vercel dashboard (e.g., NEXT_PUBLIC_API_URL).

2. Vercel automatically detects the Next.js project; if needed, customize with a vercel.json.

3. Deploy with:

   vercel --prod

Alternatively, you can deploy directly from the Vercel dashboard.

The app also includes a newsletter subscription feature, allowing users to sign up for updates. This is integrated with a third-party service (Resend) for managing subscriptions.

• Subscription Form:
  Users can enter their email addresses to subscribe to the newsletter.
• Unsubscribe Option:
  Users can unsubscribe from the newsletter at any time.
• Daily Updates:
  Subscribers receive daily updates with the latest articles. Only the latest articles are sent to subscribers, ensuring they receive the most relevant information.

1. Prerequisites: Sign up for a Resend account and obtain your API key.
2. Go to Domain Settings: In your Resend dashboard, navigate to the "Domains" section and add your domain (you'll have to have purchased a domain name that you have access to). Render will ask that you verify your domain ownership by adding a TXT record to your DNS settings, as well as adding an MX record to your DNS settings, and more. Follow the instructions provided by Resend to complete this step.
3. Configure Environment Variables: Create a .env file in the newsletters directory with the following variables:
   • RESEND_API_KEY: Your Resend API key.
   • RESEND_DOMAIN: The domain you added in the Resend dashboard.
4. Deploy the CRON Job: Simply run vercel --prod in the newsletters directory to deploy the CRON job that sends daily updates to subscribers.
5. Configure the CRON Job: In your Vercel dashboard, navigate to the "Functions" section and set up a CRON job that runs daily at 9:00 AM UTC. This job will send the latest articles to subscribers.
6. That's it! Your newsletter subscription feature is now set up and ready to go. Users can subscribe to receive daily updates with the latest articles.

The Agentic AI Pipeline is a production-hardened multi-agent system built with LangGraph + LangChain 1.x that processes articles through a series of specialized AI agents — content analysis, summarization, classification, sentiment analysis, and quality assurance.

The Agentic AI Pipeline implements an assembly line architecture where each specialized agent performs a specific task in sequence. Built on LangGraph's state machine framework, the pipeline ensures reliable, scalable, and sophisticated multi-agent orchestration.

graph LR
    A[Article Input] --> B[Content Analyzer]
    B --> C[Summarizer]
    C --> D[Classifier]
    D --> E[Sentiment Analyzer]
    E --> F[Quality Checker]
    F --> G{Quality Pass?}
    G -->|Yes| H[Output]
    G -->|No & Retry| B
    G -->|Max Retries| H

• 🤖 Multi-Agent Architecture: Five specialized agents working in concert

  • Content Analyzer: Extracts structure, entities, and key information
  • Summarizer: Generates concise, accurate summaries
  • Classifier: Categorizes content into 15+ topic categories
  • Sentiment Analyzer: Analyzes emotional tone and objectivity
  • Quality Checker: Validates outputs with automatic retry logic

• 🔄 Assembly Line Processing: LangGraph state machine with conditional routing and a bounded quality-retry loop

• 🩺 Resilience: Retry + per-provider circuit breaker + timeout (guarded_call) on every external call

• 📡 Observability: OpenTelemetry tracing + a typed Prometheus registry (synthora_* metrics) + trace-correlated, secret-redacted JSON logs

• 🔐 Security: SecretStr credentials, typed errors, input sanitization, token-bucket rate limiting, production fail-fast config

• 🔌 MCP Server: Model Context Protocol server (stdio) — every tool wrapped by tool_middleware

• 🛰️ ACP Layer: Agent Communication Protocol for inter-agent messaging (register -> heartbeat -> send -> inbox -> ack)

• 📬 Durable Agent Comms: Redis-backed ACP store for multi-replica deployments with TTL, retention, and liveness pruning

• 🐳 Container & Infra: Non-root multi-stage image; Kubernetes manifests, Helm chart, and Terraform module

• 🧪 Operational Preflight: Live ACP roundtrip checks are part of make mcp-preflight

• ☁️ Cloud-Ready: Production configs for AWS Lambda and Azure Functions

• 📊 Quality Assurance: Built-in quality checking with automatic retry mechanisms

• ⚡ Production-Ready: Comprehensive logging, monitoring, and error handling

• 🔐 Secure: Secrets management via AWS Secrets Manager and Azure Key Vault

The pipeline uses an assembly line architecture where articles flow through multiple specialized agents:

1. Intake Node: Validates input and initializes state
2. Content Analysis: Extracts structure, entities, dates, and style
3. Summarization: Generates 150-200 word summaries
4. Classification: Categorizes into relevant topics
5. Sentiment Analysis: Analyzes tone, objectivity, urgency, and controversy
6. Quality Check: Validates outputs and determines if retry is needed
7. Output Node: Returns final results

Technology Stack:

• LangChain: Framework for LLM-powered applications
• LangGraph: State machine orchestration for multi-agent systems
• Python 3.11+: Modern Python with async/await support
• Redis: State management and caching
• MongoDB: Data persistence
• Prometheus: Metrics and monitoring
• Splunk + OpenTelemetry: Centralized log aggregation, distributed tracing, and enterprise observability via OTEL Collector
• MCP Python SDK (FastMCP): Model Context Protocol server implementation
• ACP Store Backends: Redis (production) + in-memory fallback (non-production)

MCP + ACP Surface:

• MCP: 28 tools, 14 resources, 7 prompts
• ACP Tools: acp_register_agent, acp_unregister_agent, acp_heartbeat, acp_send_message, acp_fetch_inbox, acp_acknowledge_message, acp_list_agents, acp_get_message
• ACP Resources: acp://agents, acp://stats, acp://messages/recent

Cloud Deployment:

• AWS: Lambda, API Gateway, S3, SQS, Secrets Manager, CloudWatch, Kinesis Firehose → Splunk HEC
• Azure: Functions, Storage Queues, Blob Storage, Key Vault, Application Insights

Beads Subarchitecture:

• Beads are the atomic unit of work in the agentic architecture. Each bead is a discrete, well-scoped task that an agent (human or AI) can claim, execute, and verify independently
• Beads follow a PENDING → CLAIMED → IN_PROGRESS → REVIEW → DONE lifecycle with a BLOCKED escape state, and use file-level reservations (.beads/.status.json) to prevent concurrent-edit conflicts across agents
• Service-scoped IDs (ORCH-001, CRAWL-005, PIPE-012, etc.) tie every bead to the service it changes, enabling per-service tracking and parallelism
• A compound learning loop records structured session logs in .agent-sessions/ after each completed bead, so future agents benefit from accumulated experience
• See .beads/README.md for the full specification and .agent-sessions/README.md for session log format

Quick Start:

# Navigate to the agentic_ai directory
cd agentic_ai

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env with your API keys
# For production ACP, set:
# ACP_ENABLED=true
# ACP_BACKEND=redis
# REDIS_HOST=<redis-host>
# REDIS_PORT=6379

# Run the MCP server
PYTHONPATH=.. python -m mcp_server

# Run production-readiness preflight (includes live ACP checks)
make mcp-preflight

Use Programmatically:

from agentic_ai.core.pipeline import AgenticPipeline
import asyncio

# Initialize pipeline
pipeline = AgenticPipeline()

# Process an article
result = asyncio.run(pipeline.process_article({
    "id": "article-123",
    "content": "Your article content...",
    "url": "https://example.com/article",
    "source": "government"
}))

print(f"Summary: {result['summary']}")
print(f"Topics: {result['topics']}")
print(f"Quality Score: {result['quality_score']}")

Deploy to Cloud:

# Deploy to AWS
cd agentic_ai/aws
./deploy.sh production

# Deploy to Azure
cd agentic_ai/azure
./deploy.sh production

Docker Deployment:

cd agentic_ai
docker-compose up -d

For comprehensive documentation including:

• Detailed agent specifications
• MCP server API reference
• Cloud deployment guides
• Performance optimization tips
• Monitoring and observability setup
• Integration examples

Please see the complete documentation: agentic_ai/README.md

For full MCP + ACP protocol and runtime diagrams, see MCP-ACP.md.

The article Q&A feature allows users to ask questions about specific articles and receive AI-generated answers. This feature is integrated into the frontend and backend, providing a seamless experience for users.

The AI will have access to the article content and will use RAG to generate answers based on the information provided in the article. This feature is designed to enhance user engagement and provide quick answers to common questions.

The chatbot is given an identity (ArticleIQ) and is designed to answer questions related to the articles. The AI will have access to the article content and will use RAG (Retrieval-Augmented Generation) to generate answers based on the information provided in the article:

• Ask Questions: Users can ask questions about specific articles directly from the article detail page.
• AI-Generated Answers: The AI will generate answers based on the content of the article, providing users with relevant information.
• User-Friendly Interface: The Q&A feature is integrated into the article detail page, making it easy for users to ask questions and receive answers without navigating away from the content.
• RAG Integration: The AI will use RAG (Retrieval-Augmented Generation) to provide accurate and contextually relevant answers based on the article content.
• Real-Time Responses: Users will receive answers in real-time, enhancing the overall user experience and engagement with the content.

In addition to the site-wide chatbot, article-specific chatbots are also available on each article detail page. These chatbots are tailored to the content of the specific article, allowing users to ask questions and receive answers that are directly relevant to the article they are reading.

1. Prerequisites: Ensure that you have the backend and frontend set up as described above.
2. Install Dependencies: Make sure you have the necessary dependencies installed in both the backend and frontend directories. You can run npm install in the root directory to install dependencies for all components.
3. Configure Environment Variables: Ensure that you have the necessary environment variables set up in your .env file. This includes the Google AI API key and any other required variables.
4. Deploy the Backend: If you haven't already, deploy the backend to Vercel using vercel --prod in the backend directory. Or, just run locally with npm run dev.
5. Deploy the Frontend: If you haven't already, deploy the frontend to Vercel using vercel --prod in the frontend directory. Or, just run locally with npm run dev.
6. Test the Feature: Once everything is set up, you can test the article Q&A feature by navigating to an article detail page of an article and asking questions. The AI will generate answers based on the content of the article.
7. That's it! The article Q&A feature is now integrated into the existing system, providing users with an enhanced experience and quick access to information.

To use the article Q&A feature, simply navigate to the article detail page of an article and look for the Q&A section. You can ask questions related to the article, and the AI will generate answers based on the content provided.

Feel free to ask any questions related to the article, and the AI will do its best to provide accurate and relevant answers. This feature is designed to enhance user engagement and provide quick access to information without having to read through the entire article.

The sitewide chat lets users ask open-ended questions across the full corpus—not just a single article—while keeping every claim cited. The system provides two chat paths: a direct Gemini-powered RAG pipeline and an orchestrated multi-agent pipeline.

• RAG over the whole library: The backend converts queries to gemini-embedding-001 vectors, searches Pinecone for top matches, and builds a context block with [Source N] slots.
• Streaming Gemini replies: Gemini 2.0 Flash / Flash Lite streams text via Server-Sent Events (SSE) with automatic API-key/model failover and history compaction to stay within token budgets.
• Inline citations & warnings: Responses carry citation metadata plus hallucination checks (missing citations, invalid refs, overconfident claims, uncited numbers). The frontend renders clickable superscripts and yellow warnings if issues are detected.

• 16 specialized agents: 8 Anthropic (Claude) primary + 8 Google (Gemini) fallback agents covering article search, Q&A, topic exploration, trend analysis, bias detection, clarification, and quality review.
• Intent-based routing: LLM-based intent classification routes queries to the best-fit agent, with keyword heuristic fallback.
• Dual-provider failover: Anthropic primary with automatic Google failover after retry exhaustion (3 attempts, exponential backoff with jitter).
• Grounding validation: 10 canonical rules applied post-generation to detect hallucinations, missing citations, and unsupported claims.
• Cost tracking: Real-time daily budget enforcement with per-model cost breakdown.
• Streaming support: SSE streaming via /api/orchestrator/chat/stream.

• frontend/pages/ai_chat.tsx provides multiple conversations, local storage persistence, typing indicators, and interactive source cards.
• Message editing with conversation branching: Users can click the pencil icon on any sent message to edit it inline. Submitting the edit truncates the conversation at that point (removing all subsequent messages) and re-sends the edited message with only the preceding history, effectively branching the conversation from the edit point.

sequenceDiagram
    participant User
    participant UI as Frontend (ai_chat.tsx)
    participant API as Backend /api/chat/sitewide
    participant Vec as Pinecone (ai-gov-articles)
    participant LLM as Gemini 2.0 Flash

    User->>UI: Ask question
    UI->>API: POST userMessage + trimmed history
    API->>Vec: Embed query (gemini-embedding-001) & semantic search
    Vec-->>API: Top K articles + metadata
    API->>LLM: Stream request with context + citations + guardrails
    LLM-->>API: SSE chunks (text)
    API-->>UI: SSE events (status/context/citations/chunk/warnings/done)
    UI-->>User: Live updates, clickable citations, warnings if any

• Endpoint: POST /api/chat/sitewide
• Events: status, context, citations, chunk, warnings, done
• Payloads: JSON per event (e.g., {"message":"Generating response..."}, {"sources":[{number,title,url,score}]}, {"text":"partial reply"})
• Frontend handling: Streams append text into the active message bubble; citations hydrate source cards; warnings show a yellow banner above the AI reply.

SynthoraAI employs a sophisticated, multi-layered recommendation engine to deliver personalized and contextually relevant content to users.

The Related Articles feature leverages Pinecone, a high-performance vector database, to find semantically similar articles:

flowchart LR
    Article[Current Article] --> Embed[Generate Embedding]
    Embed --> Pinecone[(Pinecone Index)]
    UI[Article Page] --> API[/api/articles/:id/similar/]
    API --> Pinecone
    Pinecone --> API
    API --> UI

• Embedding Generation: Article content is transformed into high-dimensional vector embeddings using state-of-the-art NLP models
• Vector Storage: Embeddings are indexed in Pinecone for lightning-fast similarity searches
• Semantic Matching: When viewing an article, the system queries Pinecone to retrieve the most semantically similar articles based on cosine similarity
• Real-time Results: Users see up to 6 related articles displayed in an interactive carousel on each article detail page

This approach goes beyond simple keyword matching, understanding the deeper semantic relationships between articles to surface truly relevant content.

The Recommended Articles section uses a lightweight machine learning model running directly in the browser:

• Privacy-First: All computation happens client-side—no user data is sent to external servers
• Behavioral Analysis: The model analyzes user interactions (views, time spent, favorites, ratings) to build a personalized preference profile
• Real-Time Adaptation: Recommendations dynamically update based on current session behavior and historical patterns
• Hybrid Approach: Combines collaborative filtering with content-based signals for optimal accuracy
• Performance Optimized: Uses WebAssembly and quantized model weights to ensure fast inference without impacting page load times

Together, these systems ensure users discover relevant content through both semantic similarity and personalized behavioral patterns.

The aicc command gives you a single entrypoint to manage your entire monorepo—frontend, backend, crawler—and perform content‐curation tasks.

From the project root:

# Install dependencies
npm install

# Link the CLI so `aicc` is on your PATH
npm link

Run aicc with no arguments to display help:

aicc

Examples:

# Run frontend + backend + crawler in parallel
aicc dev

# Build only the backend
aicc build backend

# Start crawler in prod mode
aicc start crawler

# Lint & format everything
aicc lint

Kick off your scheduled crawler (schedule/fetchAndSummarize.ts) in the crawler package:

aicc crawl

This will cd crawler and run npm run crawl under the hood.

Interact with your backend’s /api/articles endpoints directly from the CLI:

Flags for create & update:

• --title <string> — Article title
• --content <string> — Full article content (stored in content)
• --summary <string> — Brief summary
• --topics <topic1> ... — One or more topic tags
• --source <string> — Source identifier

Examples:

# Create a new article
aicc article create \
  --title "AI in 2025" \
  --content "Deep dive into AI trends..." \
  --summary "Key trends in AI" \
  --topics ai machine-learning \
  --source "manual-cli"

# Get an article
aicc article get 64a1f2d3e4b5c6a7d8e9f0

# List up to 5 articles
aicc article list --limit 5

# Update title and topics
aicc article update 64a1f2d3e4b5c6a7d8e9f0 \
  --title "AI Trends 2025" \
  --topics ai trends

# Delete an article
aicc article delete 64a1f2d3e4b5c6a7d8e9f0

With aicc in your toolbox, you can develop, build, run, lint, crawl, and manage content—all from one unified interface.

The project includes several shell scripts and a Makefile to simplify common tasks. These scripts are located in the scripts directory and can be executed directly from the command line.

Various shell scripts are provided for tasks such as:

• Starting the backend or frontend
• Running the crawler
• Building the project
• Running tests
• Deploying to Vercel
• and more...

These scripts are designed to be easy to use and can be executed with a simple command. Visit the shell directory for more details on each script.

To run a shell script, use the following command:

chmod +x scripts/<script_name>.sh
./scripts/<script_name>.sh

The daily.sh script is a shell script that automates the process of running the crawler and sending out the newsletter. It is designed to be run daily, and it performs the following tasks:

• Runs the crawler to fetch the latest articles.
• Processes the articles and generates summaries.
• Cleanups any temporary files or artifacts, as well as dirty/corrupted articles.
• Sends out the newsletter to subscribers with the latest articles.
• Performs any other necessary tasks related to the daily operation of the application.

To run the daily.sh script, use the following command:

chmod +x daily.sh
./daily.sh

Please ensure that you have the necessary permissions and environment variables set up before running the script.

Also, you can set up a cron job to run this script automatically at a specified time each day. To do so, simply run the install_daily_cron.sh script, which will install the cron job for you.

chmod +x install_daily_cron.sh
./install_daily_cron.sh

This will create a cron job that runs the daily.sh script every day at 16:00 (4:00 PM) UTC. You can adjust the timing in the install_daily_cron.sh script if needed.

To confirm, run the following command to view your cron jobs:

crontab -l

This will display a list of all your cron jobs, including the one you just created for the daily.sh script.

The Makefile provides a convenient way to run common tasks using the make command. It includes targets for building, testing, and deploying the project.

To use the Makefile, navigate to the project root directory and run:

make <target>

To see all available targets, run:

make help

This will display a list of all targets defined in the Makefile along with their descriptions.

The backend uses Jest + Supertest (with an in-memory MongoDB) for unit and integration tests. From the backend workspace root, run:

# Install dependencies (npm ci is correct here—it installs exactly from package-lock)
npm ci

# Run all tests once
npm run test

# Rerun tests on file changes
npm run test:watch

# Generate a coverage report
npm run test:coverage

The frontend uses Playwright for end-to-end testing. From the frontend workspace directory, run:

# Install dependencies
npm ci

# Headless E2E run (default)
npm run test:e2e

# Run in headed mode (open real browser windows)
npm run test:e2e:headed

# Open the HTML report after a run
npm run test:e2e:report

By default, the Playwright report is served at http://localhost:9323 (or another port as printed in your console).

The crawler uses Jest + ts-jest to test the fetchAndSummarize job. From the crawler workspace root, run:

# Install dependencies
npm ci

# Run all crawler tests once
npm run test

# (Optional) Re-run on file changes
npm run test -- --watch

# To execute an actual crawl against your configured URLs:
npm run crawl

Make sure required environment variables (e.g. MONGODB_URI, CRAWL_URLS, CRAWL_MAX_LINKS, etc.) are defined before invoking npm run crawl and any test commands. Using npm ci in each workspace ensures a clean, reproducible installation based on your lockfile.

The project uses GitHub Actions for CI/CD. The workflow is defined in .github/workflows/ci.yml. It includes:

• Linting: Runs ESLint and Prettier checks on all code changes.
• Testing: Executes unit tests for the backend and frontend.
• Deployment: Automatically deploys the backend and frontend to Vercel on successful merges to the main branch.
• Docker: Builds and pushes Docker images for the backend and crawler.
• Cron Jobs: Configures scheduled tasks for the backend and crawler.
• Environment Variables: Sets up environment variables for the backend and crawler.
• and more...

Additional .yml files are also available for specific tasks, such as backend-ci.yml, crawler-ci.yml, and frontend-ci.yml.

The project fully supports deployment with AWS, Kubernetes, and Terraform for infrastructure as code (IaC). It utilizes blue/green and canary deployment strategies for zero-downtime releases.

For detailed deployment instructions, refer to the infrastructure/ directory, which contains Terraform scripts and Kubernetes manifests.

This project is licensed under the MIT License. See the LICENSE file for more details.

If you have any questions or suggestions, feel free to reach out to the repository maintainer:

• David Nguyen
  • LinkedIn
  • GitHub
  • Email
  • Website

I will be happy to assist you with any questions or issues you may have regarding this project.

The SynthoraAI - AI-Powered Article Content Curator project brings together a powerful backend, an intelligent crawler, a newsletter service, and a modern frontend to deliver up-to-date, summarized government-related articles. Leveraging advanced technologies like Google Generative AI, Next.js, Express.js, and MongoDB, the system is both scalable and robust. Whether you’re a government staff member or a curious public user, this solution provides a streamlined, user-friendly experience to quickly access relevant, summarized content.

Thank you for exploring this project! If you have any questions, suggestions, or contributions, feel free to reach out. Your feedback is invaluable in making this project even better. Cheers to a more informed world! 🚀

🔝 Back to Top