This example shows how to simulate and evaluate a Rasa Pro assistant built with CALM, Rasa's LLM-powered dialogue engine. CALM replaces traditional NLU pipelines with flow-based conversation management, where an LLM routes user messages to the right flow based on natural language descriptions.

The integration connects to Rasa via its REST webhook channel. Point ArkSim at a running Rasa server, define your scenarios, and get a full evaluation report.

The included CALM assistant handles customer support queries through five flows: greetings, goodbyes, pricing questions, feature questions, and a multi-step order status lookup with slot filling and a custom action.

ArkSim runs four scenarios against it:

Each scenario runs 3 times to test consistency across runs, since CALM's LLM-based routing is non-deterministic.

After the run, ArkSim generates an evaluation report (evaluation/final_report.html) scoring each conversation on goal completion, helpfulness, coherence, relevance, and more. It also surfaces specific failure patterns like repeated responses, missing information, or incorrect behavior, so you know exactly where to improve.

1. Install Rasa Pro (requires Python 3.10+):

   uv pip install rasa-pro

   Or with pip:

   pip install rasa-pro

2. Set your environment variables:

   export RASA_LICENSE="<your-license-key>"
   export OPENAI_API_KEY="<your-key>"

   Get a free Rasa Pro Developer Edition license at rasa.com.

3. Train the CALM assistant:

   cd rasa_project
   rasa train
   cd ..

Start the Rasa server (from rasa_project/). Custom actions run in-process via actions_module, so no separate action server is needed.

cd rasa_project
rasa run --enable-api --cors "*"

In a separate terminal, run the simulation and evaluation:

arksim simulate-evaluate config.yaml

By default the agent connects to http://localhost:5005/webhooks/rest/webhook. Override with the RASA_ENDPOINT environment variable if your server runs elsewhere.