DEV Community: Nathan Tarbert

How Microsoft Agent Framework + AG-UI Enable Agentic UX & Generative UI

Nathan Tarbert — Thu, 20 Nov 2025 15:13:16 +0000

Building with the Microsoft Agent Framework + AG-UI

Microsoft’s Agent Framework (MAF) is an open-source agent framework that has recently emerged, offering exceptional ways to build agents and multi-step workflows in .NET or Python. Paired with AG-UI, the frontend/runtime layer that enables those agents to appear in your app with a clean UI, streaming responses, and shared state, etc., delivers a seamless agentic experience.

The integration between the two is simple but powerful: MAF handles the reasoning and tool-use, and AG-UI bridges the interactions between the agent and the users.

Microsoft Agent Framework (MAF) and AG-UI together offer a clean separation:

MAF handles the agent’s intelligence, workflows, memory, and tool use.
AG-UI is the open protocol that standardizes how agents communicate with user interfaces.

Once an agent speaks AG-UI, it becomes immediately compatible with any AG-UI-aware frontend, including Microsoft’s own Blazor client and CopilotKit.

This combination is what turns MAF agents from “backend processes” into interactive, stateful, user-facing agents.

What the Microsoft Agent Framework Provides

MAF provides the core building blocks:

Agents that call models, use tools, and keep state across turns.
Workflows for multi-step or multi-agent processes.
Model and memory providers so your agents can store data or thread context.

Check out MAF GitHub ⭐️

It’s practical if you want to move past “chat completion” and give the model structure, state, and the ability to act.

What AG-UI Adds to the Picture

AG-UI is the open protocol that defines how an agent communicates with a user interface.

It standardizes things like:

Streaming messages
State updates
Tool invocations
UI events
Multi-agent routing
Interruptions / tool-progress events
Frontend → agent actions

This removes the need to design your own transport, your own formatting, your own event system, or your own UI message schema.

Instead of every framework inventing its own UI contract, AG-UI is the shared layer they converge on.

Check out AG-UI's GitHub ⭐️

Microsoft Agent Framework + AG-UI

1. MAF agents become UI-ready out of the box

No additional custom wiring.
No custom WebSockets.
No bespoke JSON formats.

The agent speaks AG-UI, which removes any additional complexities.

2. Multi-step workflows can surface naturally in UI

Because AG-UI supports state sync and streaming, a MAF workflow can show:

intermediate states
partial outputs
tool progress
agent-to-agent handoffs
moments where the agent needs user input

This is critical for user trust and usability.

3. Typed state becomes visible to the user

MAF agents often maintain a structured state (objects, lists, workflow context).

AG-UI transports this state to the frontend, where it can drive live UI components.

4. Use of tools

AG-UI can show:

When a tool was invoked
What parameters were used
What the tool returned

This is a major improvement over hidden function-calling flows.

5. Use any AG-UI-compatible frontend

Once your MAF agent speaks AG-UI, you can plug it into:

CopilotKit
Microsoft Blazor
Terminal Client
Kotlin SDK (mobile, JVM)

This gives the agent a portable, future-proof interface layer.

Why This Matters for Developers

A complete agent loop

Agent → protocol → frontend → user → agent

No custom engineering required.

A more interpretable agent

With streaming, intermediate states, and exposed tool use, you can debug and trust the agent in a way plain chat completions don’t allow.

This allows a path to build real applications

SaaS copilots
Co-creation copilots
automation agents

All while retaining clear structure and maintainability.

Additionally, the developer doesn't need to reinvent the wheel.

--> MAF handles reasoning + tools.

--> AG-UI handles communication + events.

--> The user is now able to collaborate with the agent in real time.

The stack is clean.

Summary

Microsoft Agent Framework gives you an agent with: tools, workflows, state, orchestration, and model thinking.

AG-UI gives that agent a standardized way to interact with the user, whether on mobile or desktop.

Together, they let you build agents that aren’t constrained to the backend, but usable, visible, and connected to real applications.

To stay up to date, please follow CopilotKit on Twitter and join the Discord community.

Vibe Code an Agentic App in Under 7 Minutes

Nathan Tarbert — Tue, 26 Aug 2025 11:13:18 +0000

Building agentic applications is powerful-but the complexity level starts to rise.

Things like managing context, wiring actions, and giving agents reliable access to documentation can eat up time before you even get to shipping features.

Not to worry, that's where the new Vibe-Coding Server comes in.

What is the Vibe-Coding Server?

The Vibe-Coding Server, built on MCP (Model Context Protocol), gives coding agents structured access to both CopilotKit documentation and actual code examples from the repos.

Instead of dumping entire pages or manually tagging docs every time, the server automatically fetches only the relevant snippets. This means:

Faster responses, more efficient development
Lower token usage
Better context for your agents

With the Vibe-Coding Server in place, you can layer in agentic features

Together, they move your agent from being conversational → operational.

See it in action here and for better video quality, on YouTube:

I recorded a walkthrough of the Vibe-Coding Server in action- including generated code examples, actions, readables, and the drop-in chat component, so you can see how everything fits together.

Would love to hear what you're connecting MCP to!

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join our Discord.

Two New Ways to Build with AG-UI: LlamaIndex and Agno

Nathan Tarbert — Thu, 19 Jun 2025 14:07:18 +0000

TL;DR

It’s been a huge month for AG-UI, the open protocol for interactive AI agents. Today, we’re expanding support with two new frameworks:

LlamaIndex — enabling retrieval-augmented agents to connect with users in real-time
Agno — bringing intuitive multi-agent workflows into the AG-UI ecosystem

Developers now have even more power to build rich, interactive agent experiences.

🚀 AG-UI Hits a Major Milestone

One month after launch, AG-UI continues to grow and evolve. Today’s update brings two new integrations that expand what's possible with agentic interfaces:

Agno: A modular agent orchestration framework now compatible with AG-UI, so you can expose your Agno agents to users through real-time interfaces.
LlamaIndex: Build production agents that can find information, synthesize insights, generate reports, and take actions over the most complex enterprise data.

These additions make it easier than ever to build high-quality user-facing AI with your favorite agent frameworks.

What Is AG-UI?

AG-UI (Agent-User Interaction Protocol) is a lightweight spec that connects backend AI agents with frontend applications. It turns agents into live, interactive participants inside your UI, not just silent executors behind an API.

It’s the difference between a black box backend and a fully visible, controllable copilot.

Think: Cursor vs. Devin.

Why It Matters

Today’s agents are often disconnected from the user. If you want real-time interaction, you’re left wiring up:

Streaming outputs
Tool invocation feedback
Shared session state
UI rendering logic
Messaging pipelines

AG-UI solves this by giving agents and apps a shared event protocol, so they can speak the same language out of the box.

How It Works

AG-UI defines 16+ event types that power live agent behavior, from tool calls to token streaming to UI state updates.

Agents can either:

Emit AG-UI events directly
Or use an adapter to convert outputs into AG-UI format

Clients subscribe to an event stream (via SSE or WebSockets), render the events live, and send back user input or control signals.

This unlocks dynamic, real-time interaction between agents and users.

Adoption & Growth

In less than 30 days, AG-UI has gained serious traction:

Integrated with: LangChain, CrewAI, Mastra, AG2, Agno, LlamaIndex
In progress: AWS, A2A, ADK, AgentOps, Human Layer (Slack)
3,700+ GitHub stars
Thousands of developers are building interactive agents

Build with AG-UI

Getting started is fast:

npx create-ag-ui-app

Let me know what you are building!

Follow CopilotKit on Twitter and say hi, and join our active Discord Community!

Introducing AG-UI: The Protocol Where Agents Meet Users

Nathan Tarbert — Tue, 13 May 2025 14:44:13 +0000

I'm thrilled to announce AG-UI, the Agent-User Interaction Protocol, a streamlined bridge connecting AI agents to real-world applications.

This brand new protocol is brought to you by CopilotKit

Check out AG-UI's GitHub ⭐️

What is AG-UI?

AG-UI is an open, lightweight protocol that streams a single sequence of JSON events over standard HTTP or an optional binary channel. These events—messages, tool calls, state patches, lifecycle signals—flow seamlessly between your agent backend and front-end interface, maintaining perfect real-time synchronization.

Get started in minutes using our TypeScript or Python SDK with any agent backend (OpenAI, Ollama, LangGraph, or custom code). Visit docs.ag-ui.com for the specification, quick-start guide, and interactive playground.

Agent-User Interaction

Today’s AI Agent ecosystem is maturing. Agents are going from interesting viral demos to actual production use, including by some of the biggest enterprises in the world.

However, the ecosystem has largely focused on backend automation, processes that run independently with limited user interaction. Workflows that are set off, or happen automatically, whose output is then used.

Common use-cases include data migration, research and summarization, form-filling, etc.

Repeatable and simple workflows, where accuracy can be ensured, or where 80% accuracy is good enough.

These have already been big productivity boosters. Automating time-consuming and tedious tasks.

Where Agents Meet Users

Coding tools (Devin vs. Cursor)

Throughout the adoption of generative AI, coding tools have been canneries in the coal mine, and Cursor is the best example of a user-interactive agent. An AI agent that works alongside users, in a shared workspace.

This contrasts with Devin, which was the promise of a fully autonomous agent that automates high-level work.

For many of the most important use-cases, Agents are helpful if they can work alongside users. This means users can see what the agent is doing, can co-work on the same output, and easily iterate together in a shared workspace.

The Challenges of Building a User-Interactive Agent

Creating these collaborative experiences presents significant technical challenges:

Real-time streaming: LLMs produce tokens incrementally; UIs need them instantly without blocking on the full response.
Tool orchestration: Modern agents call functions, run code, hit APIs. The UI must show progress and results, sometimes ask for human approval, and then resume the run—all without losing context.
Shared mutable state: Agents often generate plans, tables, or code folders that evolve step-by-step. Shipping entire blobs each time wastes bandwidth; sending diffs demands a clear schema.
Concurrency & cancellation: A user might fire off multiple queries, stop one mid-flight, or switch threads. The backend and front-end need thread IDs, run IDs, and an orderly shutdown path.
Security boundaries: Streaming arbitrary data over WebSockets is easy until you need CORS, auth tokens, and audit logs that an enterprise will sign off on.
Framework sprawl: LangChain, CrewAI, Mastra, AG2, home-grown scripts—all speak slightly different dialects. Without a standard, every UI must reinvent adapters and edge-case handling.

The AG-UI Solution

AG-UI addresses these challenges through a simple yet powerful approach:

Your client makes a single POST to the agent endpoint, then listens to a unified event stream. Each event has a type(e.g., TEXT_MESSAGE_CONTENT, TOOL_CALL_START, STATE_DELTA), and minimal payload. Agents emit events as they occur, and UIs respond appropriately—displaying partial text, rendering visualizations when tools complete, or updating interfaces when state changes.

Built on standard HTTP, AG-UI integrates smoothly with existing infrastructure while offering an optional binary serializer for performance-critical applications.

What This Enables

Please like and follow

AG-UI establishes a consistent contract between agents and interfaces, eliminating custom WebSocket formats and text parsing hacks. With this unified protocol:

Components become interchangeable: Use CopilotKit's React components with any AG-UI source
Backend flexibility: Switch between cloud and local models without UI changes
Multi-agent coordination: Orchestrate specialized agents through a single interface
Enhanced development: Build faster with richer experiences and zero vendor lock-in

AG-UI isn't just a technical specification—it's the foundation for the next generation of AI-enhanced applications where humans and agents seamlessly collaborate.

Please check out the Getting Started docs, and I look forward to you trying it out.

Interested in connecting with other AI developers?
Join the AG-UI Discord and join the AI developer community: https://go.copilotkit.ai/AG-UI-Discord

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool.

Turn Your React App Into an MCP Client in Minutes

Nathan Tarbert — Tue, 06 May 2025 15:45:43 +0000

I'm on the CopilotKit team, and I'm excited to announce we've just added built-in support for MCP. The update went live today.

For those unfamiliar, CopilotKit is a self-hostable, full-stack framework for building user interactive agents and copilots.. Our focus is on allowing your agents to take control of your application (by human approval), communicate what it's doing, and generate a completely custom UI for the user.

Check out CopilotKit's GitHub ⭐️

What’s an MCP Client?

It’s a web-based, client (React in this case) that lets you chat with any MCP server in your own app. All you need is a URL from Composio to get started.

MCP lets you connect LLMs to external tools in a standardized way. Now you can use a chat interface to talk to any MCP-compatible server, right from your React app, with no agent framework required.

Quickstart:

With one command, you can start talking to MCP servers locally, from your own Next.js app.

 copilotkit@latest init -m MCP

What we built:

To show it off, I connected a simple self-hosted ToDo app to two platforms using MCP:

Asana – Send blog ideas as tasks, assign them to myself, and set due dates.

Typefully – Pull blog titles and save them as draft tweets.

Stack:

UI: CopilotKit

MCP servers: Composio

Framework: Next.js

Agentic framework: None

The code is open source, and contributions are welcome.

GitHub: https://github.com/CopilotKit/copilotkit-mcp-demo

Docs: https://docs.copilotkit.ai/guides/model-context-protocol

Video demo: https://x.com/CopilotKit/status/1917976289547522074

Twitter Announcement: https://x.com/CopilotKit/status/1917976289547522074

Would love to hear what you're connecting MCP to!

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join our Discord.

How To Use an MCP React Client with Multiple AI Agents

Nathan Tarbert — Tue, 22 Apr 2025 15:13:34 +0000

Hey everyone👋

I run the community at CopilotKit, a free and open-source framework for building intelligent companions and in-app infrastructure.🤔

Think about it this way:🧐

CopilotKit allows your agents to control your application, communicate what its doing, and generate completely custom UI.

Check out CopilotKit's GitHub ⭐️

What is MCP?

I recorded this quick demo to show just one example of what you can build with the (Open Multi-Agent Canvas with MCP.

It looks like a sandbox, but it's so much more. With MCP, agents can work together and take real actions, like sending messages to Slack, triggering APIs, or updating your app—all in one interface. Before, doing this kind of multi-agent coordination was messy. Now it’s much simpler.

Resources:

CopilotKit GitHub Repo

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join our Discord.

Build Full-Stack AI Agents with Custom React Components (CopilotKit + CrewAI)

Nathan Tarbert — Fri, 28 Mar 2025 14:35:02 +0000

Hey everyone👋

I manage the community at CopilotKit, a free and open-source framework for building intelligent companions and, in-app infrastructure.🤔

Think about it like this:🧐 CopilotKit allows your agents to take control of your application, communicate what its doing, and generate completely custom UI.

Check out CopilotKit's GitHub ⭐️

Build Full-Stack Agents

Atai, the Co-Founder and CEO of CopilotKit produced this video to showcase how simple it is to build agents using CrewAI and bring them into your app in easy-to-understand steps.

Agentic Apps

A new wave of Agent-Native apps has emerged - such as Replit's AI Agent, Vercel's new v0, and ChatGPT's Canvas.

These apps deeply integrate an AI agent into a native application, allowing for deep human/AI collaboration and human-in-the-loop flows.

In this video, we will show you how to add an agent to your app in just a few minutes.

Resources:

CopilotKit GitHub Repo

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join our Discord.

Timestamps if you want to jump around:

00:00 CoAgents intro with CrewAI
00:22 Use the new CLI tool to add Crews to your React app
02:09 Created App with Shared agent state
03:50 React component to render the agent streaming
06:04 Agentic Generative UI
06:25 The CrewAI feature viewer demo

How to Build a Full Stack Agentic Researcher in 5 Minutes ⚡️⚡️

Nathan Tarbert — Wed, 26 Feb 2025 18:59:07 +0000

Hey everyone👋

I'm a DevRel and manage our community at CopilotKit, a free and open-source framework for building intelligent companions and, in-app infrastructure.🤔

Think about it like this:🧐 you can embed a chat interface similar to ChatGPT that doesn't just respond but actively takes actions within your app's frontend functions (Client Side) based on LLM outputs.

Agentic Apps

A new wave of Agent-Native apps has emerged - such as Replit's AI Agent, Vercel's new v0, and ChatGPT's Canvas.

These apps deeply integrate an AI agent into a native application, allowing for deep human/AI collaboration and human-in-the-loop flows.

In this video, we will show you how to add an agent to your app in just a few minutes.

Resources:

CopilotKit GitHub Repo

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join our Discord.

Timestamps if you want to jump around:

00:00 CoAgents intro
01:51 Add agent chat to the app
03:35 Share State between the Agent<>App
05:20 Stream agent state
07:04 Agentic GenUI
08:25 Human-in-the-Loop

Bringing a DeepSeek R1 LangGraph Agent Into The Real World Using CopilotKit

Nathan Tarbert — Wed, 05 Feb 2025 21:52:00 +0000

Hey everyone!

I'm a DevRel at CopilotKit, an open-source framework for agentic UI and in-app agents.

I wanted to share a video we produced to show how easy it is to give your app AI superpowers in just minutes.

The DeepSeek R1 LangGraph agent demo is powered by one simple hook, the useCoAgent hook (https://docs.copilotkit.ai/reference/....
Then, viola, you are cooking with gas so to speak!

Resources:

Follow CopilotKit on Twitter and say hi, and if you'd like to build something cool, join our Discord.

Build Your Own RAG Copilot with Anthropic, Pinecone, & CopilotKit

Nathan Tarbert — Wed, 29 Jan 2025 15:09:59 +0000

TL;DR

In this tutorial, we’ll walk you through building an AI-powered RAG Copilot step by step for your product Knowledge Base using the Anthropic AI API, Pinecone API, and CopilotKit🪁.

Here’s what we’ll cover:

Building a simple product knowledge base using Next.js
Adding CopilotKit UI components to your app client along with CopilotKit’s API endpoint.
Integrating Pinecone API to create searchable indexes for your knowledge base articles.

The result is a fully functioning RAG Knowledge Base application using CopilotKit:

What is CopilotKit

CopilotKit is the leading open-source framework for integrating production-ready AI-powered copilots into your applications. It provides a feature-rich SDK that supports various AI copilot use cases, including context awareness, copilot actions, and generative UIs.

This means you can focus on defining your copilot's role, rather than getting bogged down in the technicalities of building one from scratch or dealing with complex integrations.

Check out CopilotKit's GitHub ⭐️

Prerequisites

Before we get started, here are a few things you’ll need:

Familiarity with building web apps using React.js. A quick note—you’ll also need some knowledge of TypeScript, as we’ll be using it for this project.
Ensure Node.js >20 is installed on your system. If not, you can download and install it from the official Node.js website.

Once you’ve covered these, we’ll set up the development environment we’ll be working in.

Here’s a quick preview of what we’ll be building:

Setting Up the Project

To get started, run these commands to create a new directory for the project, and scaffold the Next.js application boilerplate source files and folders:

mkdir product-knowledge-base && cd product-knowledge-base
npx create-next-app product-knowledge-base

Follow the setup prompts. You can select as shown below (adjust based on your project requirements).

Once the project is created, navigate to the project directory, and verify everything works by running the development server:

cd product-knowledge-base
npm run dev

At this stage, your app should be running locally at http://localhost:3000.

Next, let’s install the necessary dependencies for this project. These include:

Mantine UI.
CopilotKit Packages.
Pinecone SDK.
Anthropic AI SDK.
Axios.

Run the following command to install them:

yarn add @anthropic-ai/sdk @mantine/core @mantine/hooks @copilotkit/react-core @copilotkit/react-ui @copilotkit/runtime lucide-react axios @pinecone-database/pinecone

Now, let’s set up the file structure for the project. Here is an overview of the main files and directories we’ll create:

src/app/ui/service/index.ts: To handle API calls to the backend to fetch dummy posts.
src/app/ui/components/KnowledgeBase.tsx: The main UI component for the knowledge base.
src/app/lib/types/post.ts:
src/app/lib/data/data.ts: dummy post data for the knowledge base.
src/app/api/copilotkit/route.ts: CopilotKit API endpoint.
src/app/api/posts/route.ts: dummy posts API endpoint.

Here’s how your project structure will look:

product-knowledge-base/
├── src/
│   ├── app/
│   │   ├── ui/
│   │   │   ├── service/
│   │   │   │   └── index.ts
│   │   │   ├── components/
│   │   │   │   └── KnowledgeBase.tsx
│   │   ├── lib/
│   │   │   ├── types/
│   │   │   │   └── post.ts
│   │   │   ├── data/
│   │   │   │   └── data.ts
│   │   ├── api/
│   │   │   ├── copilotkit/
│   │   │   │   └── route.ts
│   │   │   ├── posts/
│   │   │   │   └── route.ts

With this setup, you now have a working development environment for this tutorial.

Building the Knowledge Base Frontend

First, import the CopilotKit and Mantine UI providers, then wrap your entire app with them so they're available globally. Here's how you can update your layout.tsx file:

import { MantineProvider } from "@mantine/core";

import "@mantine/core/styles.css";
import "@copilotkit/react-ui/styles.css";

import { CopilotKit } from "@copilotkit/react-core";

export default function RootLayout({
  children,
}: Readonly<{
  children: React.ReactNode;
}>) {
  return (
    <html lang="en">
      <body>
        <CopilotKit runtimeUrl="/api/copilotkit">
          <MantineProvider>{children}</MantineProvider>
        </CopilotKit>
      </body>
    </html>
  );
}

When wrapping the app with these providers, remember to pass the runtimeUrl="<endpoint-url>" prop to the CopilotKit provider.

Designing the Knowledge Base Component

In this section, we’ll go over the code you need to build the knowledge base component. Let's start by defining the Post interface. Add the following code in your src/app/lib/types/post.ts:

export interface Post {
  id: number;
  title: string;
  summary: string;
  content: string;
  category: string;
  createdAt: string;
}

Next, navigate to the src/app/ui/service/index.ts file, and add the following code to handle API requests to fetch posts from the app’s backend:

import axios from 'axios';
import { Post } from '@/app/lib/types/post';

const API_BASE_URL = process.env.NEXT_PUBLIC_API_BASE_URL 

export const fetchPosts = async (): Promise<Post[]> => {
  const response = await axios.get(`${API_BASE_URL}/api/posts`);
  return response.data;
};

Let's create a .env file in the root directory of your project, and add the following base URL.

NEXT_PUBLIC_API_BASE_URL='http://localhost:3000'

Now, let’s create the Knowledge Base UI component. Inside the src/app/ui/components/KnowledgeBase.tsx, start by adding the following imports:

"use client"

import { useState, useEffect } from "react";
import {
  Container,
  Title,
  Grid,
  Card,
  Text,
  Badge,
  Group,
  Stack,
  Box,
  Modal,
  List,
} from "@mantine/core";
import { BookOpen } from "lucide-react";
import { Post } from "@/app/lib/types/post";
import { fetchPosts } from "@/app/ui/service";

Next, define the KnowledgeBase functional component and initialize the following states:

export function KnowledgeBase() {
  const [posts, setPosts] = useState<Post[]>([]);
  const [loading, setLoading] = useState(true);
  const [selectedPost, setSelectedPost] = useState<Post | null>(null);

  if (loading) {
    return <Text>Loading...</Text>;
  }

  return (
    <Container size="md" py="xl" ml="xl">
      <Stack gap="xl">
        <Group justify="center" align="center">
          <BookOpen size={32} />
          <Title order={1}>CopilotKit Product Knowledge Base</Title>
        </Group>
      </Stack>
    </Container>
  );
}

Now, let’s define a function to fetch the list of posts from the API:

useEffect(() => {
  const loadPosts = async () => {
    try {
      const data = await fetchPosts();
      setPosts(data);
    } catch (error) {
      console.error("Error loading posts:", error);
    } finally {
      setLoading(false);
    }
  };

  loadPosts();
}, []);

To display the posts fetched from the app’s backend, add the following code. We will render the list of posts in a grid layout with cards:

{/* Display posts */}
<Grid>
  {posts.map((post) => (
    <Grid.Col key={post.id} span={{ base: 12, sm: 6, md: 4 }}>
      <Card
        shadow="sm"
        padding="lg"
        radius="md"
        withBorder
        onClick={() => setSelectedPost(post)}
        style={{ cursor: "pointer" }}
      >
        <Stack gap="md">
          <Title order={3}>{post.title}</Title>
          <Badge color="blue" variant="light">
            {post.category}
          </Badge>
          <Text size="sm" c="dimmed">
            {post.summary}
          </Text>
          <Text size="xs" c="dimmed">
            Posted on: {new Date(post.createdAt).toLocaleDateString()}
          </Text>
        </Stack>
      </Card>
    </Grid.Col>
  ))}
</Grid>

To show the content of a single post, in this case, we’ll keep things straightforward by displaying dummy content in a modal component when an article card is clicked. To do that, add the following code:

{/* Modal for displaying selected post */}
{selectedPost && (
  <Modal
    opened={!!selectedPost}
    onClose={() => setSelectedPost(null)}
    title={selectedPost.title}
    centered
    size="xl"
  >
    <Stack gap="md">
      <List>
        {selectedPost.content
          .split("")
          .filter((item) => item.trim() !== "")
          .map((item, index) => (
            <List.Item key={index}>{item}</List.Item>
          ))}
      </List>
    </Stack>
  </Modal>
)}

Next, define the following function to update the selected post state when users click on the post’s card.

  const handlePostClick = (post: Post) => {
    setSelectedPost(post);
  };

Finally, to render this component in the browser, import it in your src/app/page.tsx file with the following code (make sure to delete all the boilerplate Next.js code):

import KnowledgeBase from "@/app/ui/components/KnowledgeBase";

export default function Home() {
  return (
    <div>
      <KnowledgeBase />
    </div>
  );
}

Adding CopilotKit UI Components

The next step involves adding CopilotKit UI components into the knowledge base interface. CopilotKit’s React SDK provides neatly designed and easily customizable UI components. These include sidebars, pop-ups, text areas, and a headless UI component. For this example, we will use the `CopilotSidebar` component to render an in-app chatbot interface.

To add CopilotKit’s UI sidebar component, in your src/app/ui/components/KnowledgeBase.tsx, add the following import:

import { CopilotSidebar } from "@copilotkit/react-ui";

Once imported, add the component in the JSX return statement:

<Group justify="center" style={{ width: "100%" }}>
  <Box style={{ flex: 1, maxWidth: "350px" }}>
    <CopilotSidebar
      instructions="Help the user get the right knowledge base articles for their query"
      labels={{
        initial: "Welcome! Describe the query you need assistance with.",
      }}
      defaultOpen={true}
      clickOutsideToClose={false}
    />
  </Box>
</Group>

The component accepts various props, including instructions, labels, defaultOpen, and clickOutsideToClose. Importantly, the instructions prop allows you to provide additional context to help the underlying Copilot AI LLM better understand and respond to user queries.

Using CopilotKit Hooks for AI Actions

The React CopilotKit SDK also provides a set of useful hooks that enable you to define custom actions for your app’s AI Copilot. For this example, we will use the useCopilotAction hook to define the intended action, that is, to retrieve knowledge base articles based on a user query.

To do that, start by importing the useCopilotAction hook in your KnowledgeBase component file as follows:

import { useCopilotAction } from "@copilotkit/react-core";

After importing, you can initialize the hook, and specify the action you want the copilot to perform. In this case, we’ll define an action named “FetchKnowledgebaseArticles” to retrieve relevant articles based on the supplied user query. Let's write the code for it:

useCopilotAction({
  name: "FetchKnowledgebaseArticles",
  description: "Fetch relevant knowledge base articles based on a user query",
  parameters: [
    {
      name: "query",
      type: "string",
      description: "User query for the knowledge base",
      required: true,
    },
  ],
  render: "Getting relevant answers to your query...",
});

This action setup includes several important elements. The name attribute provides a unique identifier for the action, while the description explains its purpose and when it should be used.

Additionally, the parameters array defines the inputs required for the action, such as the user’s query in this case. And lastly, the render property allows you to specify what will be displayed while the action is being executed. For this example, we’ll display a simple status message to keep the user informed about the ongoing process.

Integrating the App’s Backend

To complete the entire app’s workflow, let’s build the backend by adding endpoints for fetching posts, integrating CopilotKit functionality, and the Pinecone API to create searchable indexes for the knowledge base articles.

To do that, first, go to this GitHub repository file containing the dummy posts data, copy, and paste it into your local src/app/lib/data/data.ts file.

Next, in the src/app/api/posts/route.ts file, add the following code to set up the dummy posts API endpoint:

import { NextResponse } from 'next/server';
import { posts } from '@/app/lib/data/data';

export async function GET() {
  return NextResponse.json(posts);
}

At this point, start your dev server and head over to your app’s localhost URL. You should see the list of posts displayed, along with the CopilotKit sidebar component.

Going back to the main intent of the guide, which is, integrating an AI Copilot for product knowledge bases. Typically, most product knowledge bases are loaded with all sorts of content—blog articles, FAQs, internal SOPs, API documentation, and more. That’s a lot more than the three dummy posts we’re working with in this example.

Normally, these knowledge bases have integrated Algolia Search API to let users quickly search through resources. Now for the AI copilot, we want to go beyond just “search-show”.Essentially, we want to take advantage of LLMs’ ability to understand natural language to implement “conversational-like” search functionality for knowledge bases’ resources.

Instead of simply returning static search results, the Copilot will allow users to “chat” about the resources, ask follow-up questions, and get answers in a much more user-friendly way. This is arguably more intuitive.

To make this happen, we need to create LLM-searchable indexes—essentially, data sources that the AI can query to fetch the right information. For this, we’ll use Pinecone's vector database API to create a searchable index for the dummy data.

Creating Searchable Indexes using Pinecone API SDK

Pinecone is a vector database service designed for fast, scalable, and accurate intelligent search capabilities in applications. It allows you to store and retrieve vector representations of data efficiently, making it ideal for tasks like semantic search and recommendation systems.

What this means is that instead of relying solely on Anthropic AI LLM powering the Copilot to generate responses based on preexisting training knowledge, you can customize and contextualize the LLM so that your Copilot can address user queries and generate responses based on your app’s data. Ideally, this is why Pinecone becomes useful—it allows you to create vector databases for your data that are easily searchable with large language models.

Essentially, in this example, we are simply integrating Pinecone to create indexes for our knowledge base data. This way, the Copilot can first search through the data and generate more relevant, contextualized responses, along with accurately addressing follow-up questions based on the same knowledge base data.

Here’s a quick overview of what we’ll do:

Set up Pinecone.
Generating knowledge base content embeddings for the sample articles.
Indexing and querying the knowledge base data.

In the src/app/api/copilotkit/route.ts file, let’s start by making the following imports:

import { Pinecone } from '@pinecone-database/pinecone';
import {posts} from "@/app/lib/data/data";

Next, define your environment variables for the Pinecone and Anthropic API keys:

 const ANTHROPIC_API_KEY = process.env.NEXT_PUBLIC_ANTHROPIC_API_KEY;
 const PINECONE_API_KEY = process.env.NEXT_PUBLIC_PINECONE_API_KEY;

It’s a good practice to add a check to ensure these keys are provided Don’t worry—we’ll cover the steps for getting the Anthropic API key later on.

if (!ANTHROPIC_API_KEY || !PINECONE_API_KEY) {
  console.error('Missing required API keys. ');
  process.exit(1);
}

Now, initialize the Pinecone SDK, and set up the necessary configurations:

const pinecone = new Pinecone({ apiKey: PINECONE_API_KEY });
const model = 'multilingual-e5-large';
const indexName = 'knowledge-base-data';

We can now create the Pinecone index. An index is essentially a structured storage(numerical representations of data), that allows you to efficiently search and retrieve data based on vector similarity.

Ideally, for a production app, you would typically make API calls to retrieve your posts’ data dynamically. However, in this case, we will use the dummy data to simulate this process.

To create a vector database for our knowledge base data, we need to initialize a Pinecone index for the data.

Here’s the function to achieve this:

// Function to create the Pinecone index
const initializePinecone = async () => {
  const maxRetries = 3;
  const retryDelay = 2000;

  for (let i = 0; i < maxRetries; i++) {
    try {
      const indexList = await pinecone.listIndexes();
      if (!indexList.indexes?.some(index => index.name === indexName)) {
        await pinecone.createIndex({
          name: indexName,
          dimension: 1024,
          metric: 'cosine',
          spec: {
            serverless: {
              cloud: 'aws',
              region: 'us-east-1',
            },
          },
        });
        await new Promise(resolve => setTimeout(resolve, 5000));
      }
      return pinecone.index(indexName);
    } catch (error) {
      if (i === maxRetries - 1) throw error;
      console.warn(`Retrying Pinecone initialization... (${i + 1}/${maxRetries})`);
      await new Promise(resolve => setTimeout(resolve, retryDelay));
    }
  }
  return null; 
};

Once it’s set up, you can use it to store and retrieve your knowledge base data.

Next, we need to generate embeddings for your knowledge base. This step is crucial for enabling your application to efficiently store and search through large volumes of data using vector embeddings.

Embedding your knowledge base content is simply converting the raw data—typically text—into vectors that represent the semantic meaning of that data. Pinecone then stores these vectors in the index.

With the index in place, your application can perform operations like similarity search, allowing it to retrieve the most relevant data quickly based on the vector similarity between your queries and stored content.

To do that, add the following function:

// Initialize Pinecone and prepare the index
(async () => {
    try {
      const index = await initializePinecone();
      if (index) {
        const embeddings = await pinecone.inference.embed(
          model,
          posts.map(d => d.content),
          { inputType: 'passage', truncate: 'END' }
        );

        const records = posts.map((d, i) => ({
          id: d.id.toString(),
          values: embeddings[i]?.values ?? [],
          metadata: { text: d.content },
        }));  
        await index.namespace('knowledge-base-data-namespace').upsert(
          records.map(record => ({
            ...record,
            values: record.values || [],
          }))
        );
      }
    } catch (error) {
      console.error('Error initializing Pinecone:', error);
      process.exit(1);
    }
  })();

There are a few things we need to note:

Embeddings: These are vector representations of content, such as text or articles, that capture the semantic meaning of the data. In this case, embeddings are generated using Pinecone’s model, multilingual-e5-large, which processes the content and converts it into vectors. Note that you can also use other models, such as OpenAI, which provides an embedding API for this task.
Namespace: A namespace in Pinecone is a logical partitioning of your index. It allows you to organize data within the index and perform operations within specific segments of your data. In this case, the namespace is set to ‘knowledge-base-data-namespace’, which groups the knowledge base content together.
Records: These represent the data being inserted into Pinecone. Each record consists of an id, values (the embedding), and metadata (e.g., the text of the article). The values are what Pinecone uses to perform similarity searches, while the metadata provides additional context for each record.

Now, for setup this to work, you will need to get the Pinecone API key.

How to Get Your Pinecone API Key:

To get your Pinecone AI API Key, follow these steps:

Head over to the Pinecone Developer Console.

Select the API Keys tab, then click Create API Key. You can either use the default key or create a new one.

Specify the name of the key and click Create key.

Once you’ve completed these steps, return to your .env file, and paste the key:

NEXT_PUBLIC_PINECONE_API_KEY=your-api-key

Integrating the CopiloKit Node.js Endpoint

The final step is to add the CopilotKit Node.js endpoint to handle requests from the frontend.

Before proceeding, you’ll need to set up the Anthropic API to make requests to their service. To do that:

Create an Anthropic Account by visiting the Anthropic AI documentation to create an account and set up billing.

After logging into the Anthropic API console, generate your API key.

Once you have the API key, add it to your .env file in the root of your project:

Be sure to set up billing and quotas to enable your app to make API requests.

Remember, we added the CopilotKit API URL in the app’s client to allow it to forward requests to the CopilotKit backend for processing. To make this work, we need to define the CopilotKit API endpoint to manage and handle those requests. Let’s start by importing the following in the src/app/api/copilotkit/route.ts file:

import { CopilotRuntime, AnthropicAdapter, copilotRuntimeNextJSAppRouterEndpoint } from "@copilotkit/runtime";

import Anthropic from "@anthropic-ai/sdk";
import { NextRequest } from 'next/server'

The Copilot Runtime is the backend engine of CopilotKit, allowing apps to interact with LLMs. With it, you can define backend action for your copilots. And there are a ton of tasks you could specify, including executing internal database calls and managing different processes and workflows. However, for this specific example, we’ll define an action to query the Pinecone index for relevant articles based on the user query.

To do that, add the following code:

const runtime = new CopilotRuntime({
  actions: () => [
    {
      name: 'FetchKnowledgebaseArticles',
      description: 'Fetch relevant knowledge base articles based on a user query',
      parameters: [
        {
          name: 'query',
          type: 'string',
          description: 'The User query for the knowledge base index search to perform',
          required: true,
        },
      ],
      handler: async ({ query }: { query: string }) => {
        try {
          const queryEmbedding = await pinecone.inference.embed(
            model,
            [query],
            { inputType: 'query' }
          );
          const queryResponse = await pinecone
            .index(indexName)
            .namespace('knowledge-base-data-namespace')
            .query({
              topK: 3,
              vector: queryEmbedding[0]?.values || [],
              includeValues: false,
              includeMetadata: true,
            });
          return { articles: queryResponse?.matches || [] };
        } catch (error) {
          console.error('Error fetching knowledge base articles:', error);
          throw new Error('Failed to fetch knowledge base articles.');
        }
      },    },
  ],
});

Let’s break down this action handler. Ideally, this handle is the core engine for this entire integration. This action handler receives the query parameter passed from the Copilot client.

Using Pinecone’s query operation, the query is converted into a vector representation, which is then compared to the indexed vectors stored in Pinecone to identify the top three most relevant articles in the index. The result includes the vector values and metadata, which contain the actual matching data for the query.

Since most product knowledge bases have a ton of article posts, some even cover similar ideas, while others are totally different. It’s therefore practical to return multiple relevant pieces of data matching the query. (You can adjust the topK value to return the number of matching data objects based on your intended use case).

Finally, go ahead, and add the following code to define the CopilotKit endpoint as follows:

const anthropic = new Anthropic({ apiKey: ANTHROPIC_API_KEY });
const serviceAdapter = new AnthropicAdapter({ anthropic: anthropic as any });

export const POST = async (req: NextRequest) => {
  const { handleRequest } = copilotRuntimeNextJSAppRouterEndpoint({
    runtime,
    serviceAdapter,
    endpoint: '/api/copilotkit',
  });

    return handleRequest(req);


};

Make sure that the same URL you include in the CopilotKit API endpoint (/api/copilotkit) is specified in your frontend.

To test the entire app, navigate to your main project directory in the terminal and run the following command:

npm run dev

Then, head over to localhost:3000 in your browser and ask a question like “What features does CopilotKit offer?” in the Sidebar input field.

The result, is a fully functioning RAG Knowledge Base application using CopilotKit:

Users can continue asking additional questions; if you have a multitude of articles, instead of sifting through different pages, having the Copilot retrieve the information users need makes the process a whole lot easier.

For reference, or if you want to build on top of what we’ve covered in this article, you can clone this project’s source code from the GitHub repository.

Summary

In this guide, we have walked through the steps of building an Anthropic-powered Copilot using CopilotKit, Anthropic AI API, and Pinecone API for product knowledge bases.

While we’ve explored a couple of features, we have barely scratched the surface of CopilotKit's countless use cases, which range from building interactive AI chatbots to building agentic solutions. In essence, CopilotKit lets you add a ton of useful AI capabilities to your products in minutes.

If you’re looking to integrate AI-powered Copilots into your existing application, consider scheduling a demo, join the Discord Community of developers, and get started with the docs to try it out yourself!

Build a QA Bot for your documentation with LangChain 😻

Nathan Tarbert — Tue, 11 Jun 2024 13:22:48 +0000

TL;DR

In this tutorial, we will build an AI-powered Q&A bot for your website documentation.

🌐 Create a user-friendly Next.js app to accept questions and URLs
🔧 Set up a Wing backend to handle all the requests
💡 Incorporate @langchain for AI-driven answers by scraping and analyzing documentation using RAG
🔄 Complete connection between frontend input and AI-processed responses.

What is Wing?

Wing is an open-source framework for the cloud.

It allows you to create your application's infrastructure and code combined as a single unit and deploy them safely to your preferred cloud providers.

Wing gives you complete control over how your application's infrastructure is configured. In addition to its easy-to-learn programming language, Wing also supports Typescript.

In this tutorial, we'll use TypeScript. So, don't worry—your JavaScript and React knowledge is more than enough to understand this tutorial.

Check out Wing ⭐️

Building the frontend with Next.js

Here, you’ll create a simple form that accepts the documentation URL and the user’s question and then returns a response based on the website's data.

First, create a folder containing two sub-folders - frontend and backend. The frontend folder contains the Next.js app, and the backend folder is for Wing.

mkdir qa-bot && cd qa-bot
mkdir frontend backend

Within the frontend folder, create a Next.js project by running the following code snippet:

cd frontend
npx create-next-app ./

Copy the code snippet below into the app/page.tsx file to create the form that accepts the user’s question and the documentation URL:

"use client";
import { useState } from "react";

export default function Home() {
    const [documentationURL, setDocumentationURL] = useState<string>("");
    const [question, setQuestion] = useState<string>("");
    const [disable, setDisable] = useState<boolean>(false);
    const [response, setResponse] = useState<string | null>(null);

    const handleUserQuery = async (e: React.FormEvent) => {
        e.preventDefault();
        setDisable(true);
        console.log({ question, documentationURL });
    };

    return (
        <main className='w-full md:px-8 px-3 py-8'>
            <h2 className='font-bold text-2xl mb-8 text-center text-blue-600'>
                Documentation Bot with Wing & LangChain
            </h2>

            <form onSubmit={handleUserQuery} className='mb-8'>
                <label className='block mb-2 text-sm text-gray-500'>Webpage URL</label>
                <input
                    type='url'
                    className='w-full mb-4 p-4 rounded-md border text-sm border-gray-300'
                    placeholder='https://www.winglang.io/docs/concepts/why-wing'
                    required
                    value={documentationURL}
                    onChange={(e) => setDocumentationURL(e.target.value)}
                />

                <label className='block mb-2 text-sm text-gray-500'>
                    Ask any questions related to the page URL above
                </label>
                <textarea
                    rows={5}
                    className='w-full mb-4 p-4 text-sm rounded-md border border-gray-300'
                    placeholder='What is Winglang? OR Why should I use Winglang? OR How does Winglang work?'
                    required
                    value={question}
                    onChange={(e) => setQuestion(e.target.value)}
                />

                <button
                    type='submit'
                    disabled={disable}
                    className='bg-blue-500 text-white px-8 py-3 rounded'
                >
                    {disable ? "Loading..." : "Ask Question"}
                </button>
            </form>

            {response && (
                <div className='bg-gray-100 w-full p-8 rounded-sm shadow-md'>
                    <p className='text-gray-600'>{response}</p>
                </div>
            )}
        </main>
    );
}

The code snippet above displays a form that accepts the user’s question and the documentation URL and logs them to the console for now.

Perfect! 🎉You’ve completed the application's user interface. Next, let’s set up the Wing backend.

How to set up Wing on your computer

Wing provides a CLI that enables you to perform various Wing actions within your projects.

It also provides VSCode and IntelliJ extensions that enhance the developer experience with features like syntax highlighting, compiler diagnostics, code completion and snippets, and many others.

Before we proceed, stop your Next.js development server and install the Wing CLI by running the code snippet below in your terminal.

npm install -g winglang@latest

Run the following code snippet to ensure that the Winglang CLI is installed and working as expected:

wing -V

Next, navigate to the backend folder and create an empty Wing Typescript project. Ensure you select the empty template and Typescript as the language.

wing new

Copy the code snippet below into the backend/main.ts file.

import { cloud, inflight, lift, main } from "@wingcloud/framework";

main((root, test) => {
    const fn = new cloud.Function(
        root,
        "Function",
        inflight(async () => {
            return "hello, world";
        })
    );
});

The main() function serves as the entry point to Wing.

It creates a cloud function and executes at compile time. The inflight function, on the other hand, runs at runtime and returns a Hello, world! text.

Start the Wing development server by running the code snippet below. It automatically opens the Wing Console in your browser at http://localhost:3000.

wing it

You've successfully installed Wing on your computer.

How to connect Wing to a Next.js app

From the previous sections, you've created the Next.js frontend app within the frontend folder and the Wing backend within the backend folder.

In this section, you'll learn how to communicate and send data between the Next.js app and the Wing backend.

First, install the Wing React library within the backend folder by running the code below:

npm install @winglibs/react

Next, update the main.ts file as shown below:

import { main, cloud, inflight, lift } from "@wingcloud/framework";
import React from "@winglibs/react";

main((root, test) => {
    const api = new cloud.Api(root, "api", { cors: true })
    ;

    //👇🏻 create an API route
    api.get(
        "/test",
        inflight(async () => {
            return {
                status: 200,
                body: "Hello world",
            };
        })
    );

    //👉🏻 placeholder for the POST request endpoint

    //👇🏻 connects to the Next.js project
    const react = new React.App(root, "react", { projectPath: "../frontend" });
    //👇🏻 an environment variable
    react.addEnvironment("api_url", api.url);
});

The code snippet above creates an API endpoint (/test) that accepts GET requests and returns a Hello world text. The main function also connects to the Next.js project and adds the api_url as an environment variable.

The API URL contained in the environment variable enables us to send requests to the Wing API route. How do we retrieve the API URL within the Next.js app and make these requests?

Update the RootLayout component within the Next.js app/layout.tsx file as done below:

export default function RootLayout({
    children,
}: Readonly<{
    children: React.ReactNode;
}>) {
    return (
        <html lang='en'>
            <head>
                {/** ---👇🏻  Adds this script tag 👇🏻 ---*/}
                <script src='./wing.js' defer />
            </head>
            <body className={inter.className}>{children}</body>
        </html>
    );
}

Re-build the Next.js project by running npm run build.

Finally, start the Wing development server. It automatically starts the Next.js server, which can be accessed at http://localhost:3001 in your browser.

You've successfully connected the Next.js to Wing. You can also access data within the environment variables using window.wingEnv.<attribute_name>.

Processing user's requests with LangChain and Wing

In this section, you'll learn how to send requests to Wing, process these requests with LangChain and OpenAI, and display the results on the Next.js frontend.

First, let's update the Next.js app/page.tsx file to retrieve the API URL and send user's data to a Wing API endpoint.

To do this, extend the JavaScript window object by adding the following code snippet at the top of the page.tsx file.

"use client";
import { useState } from "react";
interface WingEnv {
    api_url: string;
}
declare global {
    interface Window {
        wingEnv: WingEnv;
    }
}

Next, update the handleUserQuery function to send a POST request containing the user's question and website's URL to a Wing API endpoint.

//👇🏻 sends data to the api url
const [response, setResponse] = useState<string | null>(null);

    const handleUserQuery = async (e: React.FormEvent) => {
        e.preventDefault();
        setDisable(true);
        try {
            const request = await fetch(`${window.wingEnv.api_url}/api`, {
                method: "POST",
                headers: {
                    "Content-Type": "application/json",
                },
                body: JSON.stringify({ question, pageURL: documentationURL }),
            });
            const response = await request.text();
            setResponse(response);
            setDisable(false);
        } catch (err) {
            console.error(err);
            setDisable(false);
        }
    };

Before you create the Wing endpoint that accepts the POST request, install the following packages within the backend folder:

npm install @langchain/community @langchain/openai langchain cheerio

Cheerio enables us to scrape the software documentation webpage, while the LangChain packages allow us to access its various functionalities.

The LangChain OpenAI integration package uses the OpenAI language model; therefore, you'll need a valid API key. You can get yours from the OpenAI Developer's Platform.

Next, let’s create the /api endpoint that handle incoming requests.

The endpoint will:

accept the questions and documentation URLs from the Next.js application,
load the documentation page using LangChain document loaders,
split the retrieved documents into chunks,
transform the chunked documents and save them within a LangChain vector store,
and create a retriever function that retrieves the documents from the vector store.

First, import the following into the main.ts file:

import { main, cloud, inflight, lift } from "@wingcloud/framework";
import { ChatOpenAI, OpenAIEmbeddings } from "@langchain/openai";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { createStuffDocumentsChain } from "langchain/chains/combine_documents";
import { CheerioWebBaseLoader } from "@langchain/community/document_loaders/web/cheerio";
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { MemoryVectorStore } from "langchain/vectorstores/memory";
import { createRetrievalChain } from "langchain/chains/retrieval";
import React from "@winglibs/react";

Add the code snippet below within the main() function to create the /api endpoint:

    api.post(
        "/api",
        inflight(async (ctx, request) => {
            //👇🏻 accept user inputs from Next.js
            const { question, pageURL } = JSON.parse(request.body!);

            //👇🏻 initialize OpenAI Chat for LLM interactions
            const chatModel = new ChatOpenAI({
                apiKey: "<YOUR_OPENAI_API_KEY>",
                model: "gpt-3.5-turbo-1106",
            });
            //👇🏻 initialize OpenAI Embeddings for Vector Store data transformation
            const embeddings = new OpenAIEmbeddings({
                apiKey: "<YOUR_OPENAI_API_KEY>",
            });

            //👇🏻 creates a text splitter function that splits the OpenAI result chunk size
            const splitter = new RecursiveCharacterTextSplitter({
                chunkSize: 200, //👉🏻 characters per chunk
                chunkOverlap: 20,
            });

            //👇🏻 creates a document loader, loads, and scraps the page
            const loader = new CheerioWebBaseLoader(pageURL);
            const docs = await loader.load();

            //👇🏻 splits the document into chunks 
            const splitDocs = await splitter.splitDocuments(docs);

            //👇🏻 creates a Vector store containing the split documents
            const vectorStore = await MemoryVectorStore.fromDocuments(
                splitDocs,
                embeddings //👉🏻 transforms the data to the Vector Store format
            );

            //👇🏻 creates a document retriever that retrieves results that answers the user's questions
            const retriever = vectorStore.asRetriever({
                k: 1, //👉🏻  number of documents to retrieve (default is 2)
            });

            //👇🏻 creates a prompt template for the request
            const prompt = ChatPromptTemplate.fromTemplate(`
                Answer this question.
                Context: {context}
                Question: {input}
                `);

            //👇🏻 creates a chain containing the OpenAI chatModel and prompt
            const chain = await createStuffDocumentsChain({
                llm: chatModel,
                prompt: prompt,
            });

            //👇🏻 creates a retrieval chain that combines the documents and the retriever function
            const retrievalChain = await createRetrievalChain({
                combineDocsChain: chain,
                retriever,
            });

            //👇🏻 invokes the retrieval Chain and returns the user's answer
            const response = await retrievalChain.invoke({
                input: `${question}`,
            });

            if (response) {
                return {
                    status: 200,
                    body: response.answer,
                };
            }

            return undefined;
        })
    );

The API endpoint accepts the user’s question and the page URL from the Next.js application, initialises ChatOpenAI and OpenAIEmbeddings, loads the documentation page, and retrieves the answers to the user’s query in the form of documents.

Then, splits the documents into chunks, saves the chunks in the MemoryVectorStore, and enables us to fetch answers to the question using LangChain retrievers.

From the code snippet above, the OpenAI API key is entered directly into the code; this could lead to security breaches, making the API key accessible to attackers. To prevent this data leak, Wing allows you to save private keys and credentials in variables called secrets.

When you create a secret, Wing saves this data in a .env file, ensuring it is secured and accessible.

Update the main() function to fetch the OpenAI API key from the Wing Secret.

main((root, test) => {
    const api = new cloud.Api(root, "api", { cors: true });
    //👇🏻 creates the secret variable
    const secret = new cloud.Secret(root, "OpenAPISecret", {
        name: "open-ai-key",
    });

    api.post(
        "/api",
        lift({ secret })
            .grant({ secret: ["value"] })
            .inflight(async (ctx, request) => {
                const apiKey = await ctx.secret.value();

                const chatModel = new ChatOpenAI({
                    apiKey,
                    model: "gpt-3.5-turbo-1106",
                });

                const embeddings = new OpenAIEmbeddings({
                    apiKey,
                });

                //👉🏻 other code snippets & configurations
    );

    const react = new React.App(root, "react", { projectPath: "../frontend" });
    react.addEnvironment("api_url", api.url);
});

From the code snippet above,
- The secret variable declares a name for the secret (OpenAI API key).
- The lift().grant() grants the API endpoint access to the secret value stored in the Wing Secret.
- The inflight() function accepts the context and request object as parameters, makes a request to LangChain, and returns the result.
- Then, you can access the apiKey using the ctx.secret.value() function.

Finally, save the OpenAI API key as a secret by running this command in your terminal.

Congratulations! You've successfully completed the project for this tutorial.

Here is a brief demo of the application:

Let's dig a little bit deeper into the Wing docs to see what data our AI bot can extract.

Wrapping It Up

So far, we have gone over the following:

What is Wing?
How to use Wing and query data using Langchain,
How to connect Wing to a Next.js application,
How to send data between a Next.js frontend and a Wing backend.

Wing aims to bring back your creative flow and close the gap between imagination and creation. Another great advantage of Wing is that it is open-source. Therefore, if you are looking forward to building distributed systems that leverage cloud services or contribute to the future of cloud development, Wing is your best choice.

Feel free to contribute to the GitHub repository, and share your thoughts with the team and the large community of developrs.

The source code for this tutorial is available here.

Thank you for reading! 🎉

Building your own ChatGPT Graphical Client with NextJS and Wing 🤯

Nathan Tarbert — Thu, 16 May 2024 14:56:39 +0000

TL;DR

By the end of this article, you will build and deploy a ChatGPT Client using Wing and Next.js.

This application can run locally (in a local cloud simulator) or deploy it to your own cloud provider.

Introduction

Building a ChatGPT client and deploying it to your own cloud infrastructure is a good way to ensure control over your data.

Deploying LLMs to your own cloud infrastructure provides you with both privacy and security for your project.

Sometimes, you may have concerns about your data being stored or processed on remote servers when using proprietary LLM platforms like OpenAI’s ChatGPT, either due to the sensitivity of the data being fed into the platform or for other privacy reasons.

In this case, self-hosting an LLM to your cloud infrastructure or running it locally on your machine gives you greater control over the privacy and security of your data.

Wing is a cloud-oriented programming language that lets you build and deploy cloud-based applications without worrying about the underlying infrastructure.
It simplifies the way you build on the cloud by allowing you to define and manage your cloud infrastructure and your application code within the same language.
Wing is cloud agnostic - applications built with it can be compiled and deployed to various cloud platforms.

Check out ⭐ Wing

Let's get started!

To follow along, you need to:

Have some understanding of Next.js
Install Wing on your machine. Not to worry if you don’t know how to. We’ll go over it together in this project.
Get your OpenAI API key.

Create Your Projects

To get started, you need to install Wing on your machine. Run the following command:

npm install -g winglang

Confirm the installation by checking the version:

wing -V

Create your Next.js and Wing apps.

mkdir assistant
cd assistant
npx create-next-app@latest frontend
mkdir backend && cd backend
wing new empty

We have successfully created our Wing and Next.js projects inside the assistant directory. The name of our ChatGPT Client is Assistant. Sounds cool, right?

The frontend and backend directories contain our Next and Wing apps, respectively. wing new empty creates three files: package.json, package-lock.json, and main.w. The latter is the app’s entry point.

Run your application locally in the Wing simulator

The Wing simulator allows you to run your code, write unit tests, and debug your code inside your local machine without needing to deploy to an actual cloud provider, helping you iterate faster.

Use the following command to run your Wing app locally:

wing it

Your Wing app will run on localhost:3000.

Setting Up Your Backend

Let’s install Wing’s OpenAI and React Libraries. The OpenAI library provides a standard interface to interact with the LLM. The React library allows you to connect your Wing backend to your Next app.

npm i @winglibs/openai @winglibs/react

Import these packages in your main.w file. Let's also import all the other libraries we’ll need.

bring openai
bring react
bring cloud
bring ex
bring http

bring is the import statement in Wing. Think of it this way, Wing uses bring to achieve the same functionality as import in JavaScript.

cloud is Wing’s Cloud library. It exposes a standard interface for Cloud API, Bucket, Counter, Domain, Endpoint, Function and many more cloud resources. ex is a standard library for interfacing with Tables and cloud Redis database, and http is for calling different HTTP methods - sending and retrieving information from remote resources.

Get Your OpenAI API Key

We will use gpt-4-turbo for our app but you can use any OpenAI model.

Create an OpenAI account if you don’t have one yet. To create a new API key, Go to platform.openai.com/api-keys and select Create new secret key.

Set the Name, Project, and Permissions, then click Create secret key.

Initializing OpenAI

Create a Class to initialize your OpenAI API. We want this to be reusable.

We will add a personality to our Assistant class so that we can dictate the personality of our AI assistant when passing a prompt to it.

let apiKeySecret = new cloud.Secret(name: "OAIAPIKey") as "OpenAI Secret";

class Assistant {
    personality: str;
    openai: openai.OpenAI;

    new(personality: str) {
        this.openai = new openai.OpenAI(apiKeySecret: apiKeySecret);
        this.personality = personality;
    }

    pub inflight ask(question: str): str {
        let prompt = `you are an assistant with the following personality: ${this.personality}. ${question}`;
        let response = this.openai.createCompletion(prompt, model: "gpt-4-turbo");
        return response.trim();
    }
}

Wing unifies infrastructure definition and application logic using the preflight and inflight concepts respectively.

Preflight code (typically infrastructure definitions) runs once at compile time, while inflight code will run at runtime to implement your app’s behavior.

Cloud storage buckets, queues, and API endpoints are some examples of preflight. You don’t need to add the preflight keyword when defining a preflight, Wing knows this by default. But for an inflight block, you need to add the word “inflight” to it.

We have an inflight block in the code above. Inflight blocks are where you write asynchronous runtime code that can directly interact with resources through their inflight APIs.

Testing and Storing The Cloud Secret

Let's walk through how we will secure our API keys because we definitely want to take security into account.

Let's create a .env file in our backend’s root and pass in our API Key:

OAIAPIKey = Your_OpenAI_API_key

We can test our OpenAI API keys locally referencing our .env file, and then, since we are planning to deploy to AWS, we will walk through setting up the AWS Secrets Manager.

First, let's head over to AWS and sign into the Console. If you don't have an account, you can create one for free.

Navigate to the Secrets Manager, and let's store our API key values.

We have stored our API key in a cloud secret named OAIAPIKey. Copy your key, and we will jump over to the terminal and connect to our secret, which is now stored in the AWS Platform.

wing secrets

Now paste in your API Key as the value in the terminal. Your keys are now properly stored, and we can start interacting with our app.

Storing The AI’s Responses in the Cloud.

Storing your AI's responses in the cloud gives you control over your data. It resides on your own infrastructure, unlike proprietary platforms like ChatGPT, where your data lives on third-party servers that you don’t have control over. You can also retrieve these responses whenever you need them.

Let’s create another class that uses the Assistant class to pass in our AI’s personality and prompt. We would also store each model’s responses as txt files in a cloud bucket.

let counter = new cloud.Counter();

class RespondToQuestions {
    id: cloud.Counter;
    gpt: Assistant;
    store: cloud.Bucket;

    new(store: cloud.Bucket) {
        this.gpt = new Assistant("Respondent");
        this.id = new cloud.Counter() as "NextID";
        this.store = store;
    }

    pub inflight sendPrompt(question: str): str {
        let reply = this.gpt.ask("{question}");
        let n = this.id.inc();
        this.store.put("message-{n}.original.txt", reply);
        return reply;
    }
}

We gave our Assistant the personality “Respondent.” We want it to respond to questions. You could also let the user on the frontend dictate this personality when sending in their prompts.

Every time it generates a response, the counter increments, and the value of the counter is passed into the n variable used to store the model’s responses in the cloud. However, what we really want is to create a database to store both the user prompts coming from the frontend and our model’s responses.

Let's define our database.

Defining Our Database

Wing has ex.Table built-in - a NoSQL database to store and query data.

let db = new ex.Table({
    name: "assistant",
    primaryKey: "id",
    columns: {
        question: ex.ColumnType.STRING,
        answer: ex.ColumnType.STRING
    }
});

We added two columns in our database definition - the first to store user prompts and the second to store the model’s responses.

Creating API Routes and Logic

We want to be able to send and receive data in our backend. Let’s create POST and GET routes.

let api = new cloud.Api({ cors: true });

api.post("/assistant", inflight((request) => {
    // POST request logic goes here
}));

api.get("/assistant", inflight(() => {
    // GET request logic goes here
}));

let myAssistant = new RespondToQuestions(store) as "Helpful Assistant";

api.post("/assistant", inflight((request) => {
    let prompt = request.body;
    let response = myAssistant.sendPrompt(JSON.stringify(prompt)); 
    let id = counter.inc(); 

    // Insert prompt and response in the database
    db.insert(id, { question: prompt, answer: response });

    return cloud.ApiResponse({
        status: 200
    });
}));

In the POST route, we want to pass the user prompt received from the frontend into the model and get a response. Both prompt and response will be stored in the database. cloud.ApiResponse allows you to send a response for a user’s request.

Add the logic to retrieve the database items when the frontend makes a GET request.

api.get("/assistant", inflight(() => {
    let questionsAndAnswers = db.list();

    return cloud.ApiResponse({
        body: JSON.stringify(questionsAndAnswers),
        status: 200
    });
}));

Our backend is ready. Let's test it out in the local cloud simulator.

Run wing it.

Lets go over to localhost:3000 and ask our Assistant a question.

Both our question and the Assistant’s response has been saved to the database. Take a look.

Exposing Your API URL to The Frontend

We need to expose the API URL of our backend to our Next frontend. This is where the react library installed earlier comes in handy.

let website = new react.App({
    projectPath: "../frontend",
    localPort: 4000
});

website.addEnvironment("API_URL", api.url);

Add the following to the layout.js of your Next app.

import { Inter } from "next/font/google";
import "./globals.css";

const inter = Inter({ subsets: ["latin"] });

export const metadata = {
    title: "Create Next App",
    description: "Generated by create next app",
};

export default function RootLayout({ children }) {
    return (
        <html lang="en">
            <head>
            <script src="./wing.js" defer></script>
           </head>
            <body className={inter.className}>{children}</body>
        </html>
    );
}

We now have access to API_URL in our Next application.

Implementing the Frontend Logic

Let’s implement the frontend logic to call our backend.

import { useEffect, useState, useCallback } from 'react';
import axios from 'axios';

function App() {

    const [isThinking, setIsThinking] = useState(false);
    const [input, setInput] = useState("");
    const [allInteractions, setAllInteractions] = useState([]);

    const retrieveAllInteractions = useCallback(async (api_url) => {
            await axios ({
              method: "GET",
              url: `${api_url}/assistant`,
            }).then(res => {
              setAllInteractions(res.data)
            })
  }, [])

    const handleSubmit = useCallback(async (e)=> {
        e.preventDefault()

        setIsThinking(!isThinking)


        if(input.trim() === ""){
          alert("Chat cannot be empty")
          setIsThinking(true)

        }

          await axios({
            method: "POST",
            url: `${window.wingEnv.API_URL}/assistant`,
            headers: {
              "Content-Type": "application/json"
            },
            data: input
          })
          setInput("");
          setIsThinking(false);
          await retrieveAllInteractions(window.wingEnv.API_URL);     

  })

    useEffect(() => {
        if (typeof window !== "undefined") {
            retrieveAllInteractions(window.wingEnv.API_URL);
        }
    }, []);

    // Here you would return your component's JSX
    return (
        // JSX content goes here
    );
}

export default App;

The retrieveAllInteractions function fetches all the questions and answers in the backend’s database. The handSubmit function sends the user’s prompt to the backend.

Let’s add the JSX implementation.

import { useEffect, useState } from 'react';
import axios from 'axios';
import './App.css';

function App() {
    // ...

    return (
        <div className="container">
            <div className="header">
                <h1>My Assistant</h1>
                <p>Ask anything...</p>
            </div>

            <div className="chat-area">
                <div className="chat-area-content">
                    {allInteractions.map((chat) => (
                        <div key={chat.id} className="user-bot-chat">
                            <p className='user-question'>{chat.question}</p>
                            <p className='response'>{chat.answer}</p>
                        </div>
                    ))}
                    <p className={isThinking ? "thinking" : "notThinking"}>Generating response...</p>
                </div>

                <div className="type-area">
                    <input 
                        type="text" 
                        placeholder="Ask me any question" 
                        value={input} 
                        onChange={(e) => setInput(e.target.value)} 
                    />
                    <button onClick={handleSubmit}>Send</button>
                </div>
            </div>
        </div>
    );
}

export default App;

Running Your Project Locally

Navigate to your backend directory and run your Wing app locally using the following command

cd ~assistant/backend
wing it

Also run your Next.js frontend:

cd ~assistant/frontend
npm run dev

Let’s take a look at our application.

Let’s ask our AI Assistant a couple developer questions from our Next App.

Deploying Your Application to AWS

We’ve seen how our app can work locally. Wing also allows you to deploy to any cloud provider including AWS. To deploy to AWS, you need Terraform and AWS CLI configured with your credentials.

Compile to Terraform/AWS using tf-aws. The command instructs the compiler to use Terraform as the provisioning engine to bind all our resources to the default set of AWS resources.

cd ~/assistant/backend
wing compile --platform tf-aws main.w

Run Terraform Init and Apply

cd ./target/main.tfaws
terraform init
terraform apply

Note: terraform apply takes some time to complete.

You can find the complete code for this tutorial here.

Wrapping It Up

As I mentioned earlier, we should all be concerned with our apps security, building your own ChatGPT client and deploying it to your cloud infrastructure gives your app some very good safeguards.

We have demonstrated in this tutorial how Wing provides a straightforward approach to building scalable cloud applications without worrying about the underlying infrastructure.

If you are interested in building more cool stuff, Wing has an active community of developers, partnering in building a vision for the cloud. We'd love to see you there.

Just head over to our Discord and say hi!