Building CLI Agents: It’s What You Don’t Give Them That Counts

A version of this post originally appeared on Kelson's Twitter.

My team shipped @amplitude/wizard on April 30. One simple command, run from your terminal, and you get a fully instrumented Amplitude project: framework detection, SDK install, custom event tracking written into your code, ingestion verified before the agent says done.

Because the drive was for simplicity, my engineering team actually spent most of our time removing things from the Wizard CLI. Tools the agent shouldn’t have. Context that made it worse. A mega-prompt that felt comprehensive and performed terribly.

At the end of it, the Wizard we shipped is actually fourteen subagents pretending to be one tool. The architecture makes it feel like one tool instead of fourteen. We only got there by rigorously setting constraints.

Here are some of the things we thought about as we worked to make our CLI agent simpler in case you’re building one of your own.

npx @amplitude/wizard

A tracking plan as simple as an SDK install

Our launch post does a good job explaining the problem we were trying to solve with the Wizard CLI and what we built. But to summarize.

Getting started with analytics is hard. The problem isn’t the install; SDKs have been a one-line install for years. It’s everything after that, where you figure out which events and properties you actually need to track. We’ve watched thousands of teams get stuck here.

But if you take a step back, what’s hard about a tracking plan? The whole point of analytics is to answer questions about your product. The question of what to track is a question about your product. Your code is your product, written down. It describes exactly what’s worth tracking, in language more accurate than any docs page or onboarding wizard.

So we built something that reads your code.

npx @amplitude/wizard runs an agent in your terminal that walks your repo, proposes events that match how your product actually works, shows you the plan, and writes the calls inline once you approve. Not generic page_viewed. Actual events, named in your product’s vocabulary, in the files where they belong.

End-to-end, it does auth, framework detection, SDK install, tracking plan, Session Replay + Experiment + Guides and Surveys, API key, verification, and the MCP server. Nothing lands in your tracked code until you approve it, though. Every change is shown as a diff. Revert anytime.

Big things that didn’t make the cut

Letting the agent pick its own tools. It picked Bash for everything. Always. Including things that obviously needed a structured file edit. We narrowed each subagent’s tool set explicitly. Hallucinated git calls dropped to zero.

We started with one model doing it all. That didn’t work. The latest Sonnet models ended up being the right supervisor. Narrow subagents handle classification, file walks, and plan generation. Each one runs hot, dies fast, and returns a structured result. The supervisor never sees the subagent’s reasoning trace, just the validated output.

The Wizard installs an MCP server so users can ask their editor to do tasks with Amplitude, like “Add tracking to the new checkout flow,” or “What events fired on the sign-up page yesterday?” MCP is the only connector standard we tried that didn’t fall apart in testing. We’re going forward with it.

The detail that took weeks: Polling

Oh, the polling loop. This took four rewrites.

When the agent finishes writing track() calls, we don’t trust that the install worked. We hit the Amplitude ingestion API and wait until the first event lands. If it doesn’t, we surface what’s wrong instead of saying “Done” and leaving you to discover at 11 p.m. three days later that nothing was flowing.

The current version is uglier than we’d like. It works.

It’s the kind of thing nobody notices when it works and would be furious about if it didn’t. Most setup tools skip it. We didn’t.

Three things the Wizard taught us

Building good agents is mostly about deciding what NOT to give them. Tools, files, instructions, history. Less is the entire trick.

For instructions, context is a budget, not a buffer. Every token you put into a prompt is a token of attention you’re spending. We removed more instructions than we added in the last month of development.

Controlling context also means constraining scope. Parallelism is great, but subagents really need isolation to be effective. One job, one context, one thing to be good at. When something fails, the supervisor’s context is still clean. Blast radius matters more than throughput.

Finally, don’t give agents too many choices with how to do their job. A little flexibility is fine to reduce instruction, but if you’re too open-ended, they’ll flounder. Give guidance on tools.

Standing on spiky shoulders

Hat tip to PostHog. Their engineering blog showed that writing honestly about what you built, what broke, and what you’d do differently is a better growth channel than any landing page. We took notes. Their open-source CLI was an early inspiration for the Wizard, too. If you’re building agents, their work is worth reading alongside ours.

Where this is heading (headless-ing?)

Today’s Wizard is iteration 1. The obvious flows are great, but the hard taxonomy work isn’t done yet. We shipped because it’s already better than what was there, and real feedback beats waiting for more polish.

The Wizard is one command. What’s next depends on an API-first mindset across the whole product. The CLI is the surface. Everything underneath has to be reachable, scriptable, and agent-driveable.

That’s where my team is going.