This project is a prototype for an AI agent that observes a user transferring data between two web systems, learns the mapping between fields, and then uses browser automation to repeat the process with new data.

The current demo simulates a real business scenario:

• Origin system: external retailer order portal with a database-like table of purchase orders.
• Destination system: internal order registration system with a form and a registered-orders table.
• Agent: records one manual transfer, learns origin-to-destination mappings, and then uses Playwright to transfer remaining orders.

User manually transfers one order
        ↓
Chrome Extension records browser events
        ↓
Backend stores events in events.json
        ↓
Backend learns field mappings
        ↓
Mappings are saved in workflow.json
        ↓
Playwright reads workflow.json
        ↓
Playwright transfers remaining orders automatically

The current version does not use an LLM yet.

Right now, the backend learns using a heuristic:

1. The extension records origin events, such as:

   • click on an origin table cell
   • copy from an origin table cell

2. The extension records destination events, such as:

   • paste into an input
   • input/change event in a form field

3. The backend compares values.

Example:

Origin copied value:
PO-1001 from "PO Number"

Destination pasted value:
PO-1001 into "Internal Order ID"

Learned mapping:
PO Number → Internal Order ID

The same happens for the rest of the fields:

Customer → Client Chain
Store ID → Branch Code
SKU → Material Code
Units → Case Quantity
Delivery Date → Requested Delivery Date
Notes → Logistics Notes

This is useful for proving the basic pipeline, but it is still rule-based. The next step is to replace this heuristic with an LLM-based learner.

agent-demo/
├── apps/
│   ├── origin-todo/
│   │   └── index.html
│   └── destination-todo/
│       └── index.html
│
├── extension/
│   ├── manifest.json
│   ├── content.js
│   ├── background.js
│   ├── popup.html
│   └── popup.js
│
└── agent-backend/
    ├── package.json
    ├── server.js
    ├── workflow.json
    ├── events.json
    └── run-playwright.js

The folder names still say todo, but the current pages are now database/order-transfer demos.

Install:

• Node.js 18+
• npm
• Python 3
• Google Chrome
• Playwright browsers

From:

cd agent-demo/agent-backend

Run:

npm install
npx playwright install chromium

Open three terminals.

cd agent-demo/apps/origin-todo
python -m http.server 3000

Open:

http://localhost:3000

This should show the Origin Retailer Portal.

cd agent-demo/apps/destination-todo
python -m http.server 3001

Open:

http://localhost:3001

This should show the Destination Internal System.

cd agent-demo/agent-backend
npm start

The terminal should show:

[Agent Backend] Server running at http://localhost:4000

1. Open Chrome.
2. Go to:

chrome://extensions

3. Enable Developer mode.
4. Click Load unpacked.
5. Select:

agent-demo/extension

6. Pin the extension if desired.

Important: after editing any extension file, reload the extension from chrome://extensions and refresh both localhost:3000 and localhost:3001.

The content script should run only on:

http://localhost:3000/*
http://localhost:3001/*

It should not record the backend page:

http://localhost:4000/*

The backend is only for API/debugging. If the extension records localhost:4000, the events may show as system: unknown.

Before recording a new example, reset events.json.

In PowerShell:

curl.exe -X POST http://localhost:4000/reset

Then check:

http://localhost:4000/debug-events

Expected result:

{
  "totalEvents": 0,
  "originEvents": 0,
  "destinationEvents": 0,
  "unknownEvents": 0,
  "summary": []
}

The goal is to manually transfer one order from the origin system into the destination system.

Recommended example:

Origin row: PO-1001

PO Number       → Internal Order ID
Customer        → Client Chain
Store ID        → Branch Code
SKU             → Material Code
Units           → Case Quantity
Delivery Date   → Requested Delivery Date
Notes           → Logistics Notes

Steps:

1. Open both systems:

   • localhost:3000
   • localhost:3001

2. Click the Chrome extension.

3. Click:

Start Recording

4. In the origin system, copy one value.

Example:

PO-1001

5. Paste it into the matching destination field.

Example:

Internal Order ID

6. Repeat for all fields in the row.

7. Click Save Order in the destination system.

8. Click the extension again.

9. Click:

Stop + Learn

If successful, the backend will save a learned workflow into:

agent-backend/workflow.json

After recording one full row, workflow.json should contain mappings like:

{
  "workflowName": "database_row_transfer",
  "mappings": [
    {
      "originLabel": "PO Number",
      "originField": "poNumber",
      "destinationLabel": "Internal Order ID",
      "destinationSelector": "#internal-order-id",
      "destinationField": "internalOrderId",
      "observedValue": "PO-1001",
      "confidence": 0.95
    },
    {
      "originLabel": "Customer",
      "originField": "customer",
      "destinationLabel": "Client Chain",
      "destinationSelector": "#client-chain",
      "destinationField": "clientChain",
      "observedValue": "Walmart",
      "confidence": 0.95
    }
  ]
}

Ideally, there should be 7 mappings:

PO Number
Customer
Store ID
SKU
Units
Delivery Date
Notes

If mappings is empty, the recording did not capture matching origin and destination values.

Open:

http://localhost:4000/debug-events

You should see both origin and destination events.

Good example:

{
  "eventType": "copy",
  "system": "origin",
  "selectedText": "PO-1001",
  "tableHeader": "PO Number"
}

Good destination example:

{
  "eventType": "paste",
  "system": "destination",
  "pastedText": "PO-1001",
  "inputLabel": "Internal Order ID"
}

If all events are destination, the origin page was not recorded.

If events are unknown, the content script is probably running on the wrong page.

After recording and learning, click:

Play Agent

The backend runs:

node run-playwright.js

Playwright will:

1. Open the origin system.
2. Read all origin order rows.
3. Open the destination system.
4. Detect which orders already exist.
5. Transfer only unfinished orders.
6. Validate that transferred orders appear in the destination table.

Expected logs:

Loaded workflow
Opening origin...
Opening destination...
Extracting origin rows...
Existing destination order IDs:
["PO-1001"]

Rows to transfer:
PO-1002
PO-1003

Transferring order: PO-1002
PO Number → Internal Order ID: PO-1002
Customer → Client Chain: Soriana
...

Validation:
PASS - PO-1002
PASS - PO-1003

The intended behavior is:

User records PO-1001 manually.
Destination now contains PO-1001.
Agent presses Play.
Agent should NOT redo PO-1001.
Agent should transfer PO-1002 and PO-1003.

This is closer to the challenge because the agent finishes the work the user started.

Cause:

The backend did not capture matching origin and destination values.

Check:

http://localhost:4000/debug-events

You need both:

copy | origin
paste/input/change | destination

Cause:

The extension recorded a page that is not recognized as origin or destination, usually localhost:4000.

Fix:

Make sure the extension content script only runs on ports 3000 and 3001.

Cause:

The workflow did not learn all mappings.

Check workflow.json.

You should see mappings for all fields. If only 3 mappings exist, only 3 fields will be filled.

Cause:

The destination app uses localStorage.

Fix:

Click Clear Destination in the destination app, or run this in the browser console:

localStorage.removeItem("destination_orders");
location.reload();

Currently, the backend learns mappings using value matching. To make it more intelligent, replace the heuristic in /learn with an LLM call.

The LLM should receive:

{
  "task": "Infer origin-to-destination field mappings from an observed browser trace.",
  "originEvents": [],
  "destinationEvents": [],
  "originPageContext": {},
  "destinationPageContext": {}
}

The LLM should return structured JSON:

{
  "workflowName": "database_row_transfer",
  "mappings": [
    {
      "originLabel": "PO Number",
      "originField": "poNumber",
      "destinationLabel": "Internal Order ID",
      "destinationSelector": "#internal-order-id",
      "destinationField": "internalOrderId",
      "reason": "The observed value PO-1001 was copied from PO Number and pasted into Internal Order ID.",
      "confidence": 0.98
    }
  ]
}

The backend would then save this LLM output into workflow.json.

Chrome Extension
    ↓
events.json
    ↓
Backend /learn
    ↓
Build compact trace summary
    ↓
Send trace summary to LLM
    ↓
LLM returns workflow JSON
    ↓
Validate JSON schema
    ↓
Save workflow.json
    ↓
Playwright executes

The LLM should help with:

1. Semantic field matching

Example:

Store ID → Branch Code
Units → Case Quantity
SKU → Material Code
Customer → Client Chain

2. Robustness when field names are different.

3. Explaining why mappings were learned.

4. Handling cases where values are formatted differently.

Example:

2026-06-10 → June 10, 2026
120 → 120 cases

5. Alerting when a mapping is uncertain.

You are an AI workflow learner.

You observe a user transferring one row of data from an origin web system to a destination web system.

Your task:
Infer the reusable field mapping that should be applied to future rows.

Rules:
- Do not memorize only the example values.
- Learn which origin field maps to which destination field.
- Use copied values, pasted values, labels, placeholders, selectors, and nearby text.
- Return only valid JSON.
- Include confidence for each mapping.
- If a mapping is uncertain, include it with lower confidence.

Return this JSON shape:

{
  "workflowName": "database_row_transfer",
  "mappings": [
    {
      "originLabel": "",
      "originField": "",
      "originSelector": "",
      "destinationLabel": "",
      "destinationField": "",
      "destinationSelector": "",
      "observedValue": "",
      "reason": "",
      "confidence": 0.0
    }
  ]
}

Install dotenv:

npm install dotenv

Create .env:

LLM_API_KEY=your_key_here

Important:

Do not commit .env to GitHub.

Example structure:

async function learnWorkflowWithLLM({ originEvents, destinationEvents }) {
  const traceSummary = {
    originEvents,
    destinationEvents
  };

  const prompt = `
You are an AI workflow learner.

Infer reusable origin-to-destination mappings from this trace.

Return only valid JSON.

Trace:
${JSON.stringify(traceSummary, null, 2)}
`;

  // Replace this with the chosen provider:
  // OpenAI, Azure OpenAI, Gemini, Claude, etc.

  const llmResponse = await callLLM(prompt);

  const workflow = JSON.parse(llmResponse);

  return workflow;
}

Then inside /learn:

const workflow = await learnWorkflowWithLLM({
  originEvents,
  destinationEvents
});

writeJson(WORKFLOW_FILE, workflow);

The next step should be:

Keep the current heuristic learner as a fallback.
Add an LLM learner as the primary method.
If the LLM fails or returns invalid JSON, use the heuristic learner.

Recommended logic:

POST /learn
    ↓
Try LLM learner
    ↓
Validate JSON
    ↓
If valid:
      save LLM workflow
    else:
      use heuristic learner

This makes the demo safer because the agent can still work if the LLM API fails during the live demo.

Working:

• Local origin system
• Local destination system
• Chrome Extension recorder
• Backend event storage
• Heuristic workflow learning
• Playwright automation
• Debug endpoint

Still needed:

• Clean final UI/dashboard
• LLM-based /learn
• Better validation table
• Alerts for missing/uncertain mappings
• Better handling of multiple pages/tabs
• Demo script