What I built

ARYA is four specialized AI agents (Scout, Risk, Executor, Orchestrator) that collaboratively discover yield farming opportunities, evaluate risk, and execute swaps — with a human-in-the-loop approval gate enforced by smart contracts. The user approves or rejects every strategy before funds move.

The stack: LangGraph.js for agent orchestration, five Solidity contracts on 0G Chain (ERC-7857 identity, strategy vault, reputation tracking, ERC-4337 smart accounts with session keys), a Next.js frontend, Uniswap Trading API for swaps, KeeperHub for automated monitoring, and 0G Storage for agent memory.

206 commits over 14 days, solo. Contracts deployed, frontend live on Vercel, all three sponsor integrations wired up. The system worked end-to-end: agents propose a strategy, user reviews it in the dashboard, approves on-chain, executor builds the swap. arya-testnet.com

What the finalists built

Seven projects made the finalist round. Three stood out as instructive comparisons to what I built:

Clan World: Ælder Whispers — Four AI agents compete as rival clan elders in a fully onchain strategy game. They negotiate, betray each other, and consolidate knowledge into ERC-7857 iNFTs that transfer between owners. The key mechanic: a memory-wipe every 10 ticks forces agents to consciously decide what to save. This makes agent skill measurable — you can watch one agent outperform another over time.

Slopstock — A stock exchange for AI agents. Agents are minted as iNFTs with sealed weights in TEEs, ownership is fractionalized into shares, and inference revenue distributes to shareholders. Three live productive agents running right now: a Solidity auditor, a meme-token rugpull detector, and a price oracle. 83 passing tests across 8 contracts. The TEE-sealed weights mean ownership transfer is cryptographically real, not just a database update.

LPlens — Diagnoses underperforming Uniswap V3 LP positions using closed-form impermanent loss math, classifies pool behavior, simulates 1,000 historical swaps against V4 hooks, and proposes one-click migrations with Permit2. Every numeric output has an honesty label (VERIFIED/COMPUTED/ESTIMATED) — a trust mechanism I haven’t seen before in DeFi tooling.

The other four finalists (Mnemosyne, Aegis402, DAIO, Common OS) followed similar patterns — each had a single novel mechanic explored deeply rather than a broad integration of many components.

Where the gap was

1. I built infrastructure. They built experiences.

ARYA is plumbing. It’s a pipeline that moves data between agents and presents recommendations in a dashboard. The user experience is: look at numbers, click approve. Functional, but not compelling in a 3-minute demo.

Clan World has AI agents publicly trash-talking each other and forming alliances. Slopstock lets you buy shares in an AI agent and watch your dividends roll in. These create moments that stick in a judge’s memory. My demo was “look at this well-architected system working correctly.” Their demos were “watch this happen.”

2. Novel mechanics vs. sensible architecture

I spent significant time on architecture: clean agent separation, proper graph-based orchestration, session keys with spend limits, reputation tracking. Good engineering, but none of it is new.

The winners introduced mechanics that don’t exist elsewhere. Clan World’s memory-wipe-and-save cycle. Slopstock’s fractional ownership of TEE-sealed model weights. LPlens’s honesty labels that explicitly distinguish verified on-chain data from estimated projections. Each finalist had at least one thing where you’d say “I’ve never seen that before.”

ARYA’s novelty was supposed to be the multi-agent collaboration with human-in-the-loop approval. But that’s a well-known pattern in AI safety research — it’s responsible engineering, not a new idea.

3. Depth over breadth

ARYA integrates three sponsors across five contracts, four agents, a full frontend, and multiple external APIs. It’s wide. But each integration is shallow — the Uniswap integration calls the Trading API for quotes, the KeeperHub integration sets up basic monitoring, the 0G integration stores state.

LPlens does one thing — analyze LP positions — but goes absurdly deep: closed-form IL math, pool behavior classification, 1,000-swap Monte Carlo simulations, V4 hook compatibility analysis. Slopstock has 8 contracts with 83 passing tests and three agents that produce value today. Depth reads as conviction. Breadth reads as a checklist.

What I’d change

Pick one agent, make it exceptional. Instead of four agents doing the full pipeline, I’d build one agent that does something no human can do efficiently — like LPlens analyzing every LP position in a user’s wallet and explaining exactly why each one is bleeding money.

Demo-driven development. Start with the 3-minute demo script on day 1. Every feature exists only if it creates a visible moment in that demo. I built bottom-up (contracts → agents → frontend) instead of top-down (demo → what do I need to make this moment work). The bottom-up approach produces sound architecture. The top-down approach produces a compelling story.

Ship a live agent. Slopstock’s agents are running and producing value right now. My agents run in response to user requests. There’s a difference between “this agent can do things” and “this agent is doing things right now, watch it.” The latter is dramatically more impressive when you have three minutes to convince a judge.

The project works. The architecture is sound. But hackathons reward memorable demos and novel ideas over clean engineering — and that’s the right call for a 2-week sprint where the goal is to show what’s possible.

This is what went right, what went wrong, and what I’d do differently.

Timeline

The jam ran mid-to-late April. I started April 17 with a blank Defold project and submitted April 26. Rough breakdown:

• Day 1 (Apr 17): Project scaffolding, first working prototype of the hook mechanic, basic collision setup.
• Days 2–3 (Apr 18–19): Cable auto-release, mouse aiming, initial level geometry via Wavedash (Defold’s tilemap tool).
• Days 4–6 (Apr 22–23): Level loader with collection proxies, two playable levels, sprite assets, camera panning.
• Days 7–8 (Apr 24–25): Level select screen, splash screen, third level shell, spike hazards.
• Days 9–10 (Apr 25–26): Camera clamping, backgrounds, crate walls, exit gate, speed gates, final polish, “thank you” screen.

The core mechanic: pull, don’t swing

My first attempt used a pendulum-swing approach — raycast to an anchor, create a rope constraint, let the player orbit around it. In zero-gravity top-down, this felt lifeless. There’s no “down” to swing from, so the rope just made the player orbit aimlessly.

I scrapped it and went with a pull-toward system. When you fire a cable, the player is pulled directly toward the anchor point. Multiple cables (up to 3) pull toward the bisector of the last two. The cable auto-releases when you arrive close enough. This created the momentum-based gameplay I wanted — fire at a distant anchor, get pulled at speed, release at the right moment to coast, fire the next cable to redirect.

The implementation is pure Lua math on every frame — no physics joints:

local PULL_FORCE = 500
local DAMPING_HOOKED = 0.02  -- near-zero friction while pulled

-- Each frame: compute pull direction from active cables
-- Apply force toward bisector of last two anchor positions
-- Auto-release when within AUTO_RELEASE_DIST (120px)

This gave me full control over the feel without fighting Box2D’s joint solver in a zero-gravity environment.

What went right

Choosing Defold. The engine’s collection proxy system made level loading trivial — each level is a self-contained .collection file, loaded/unloaded by a central loader script. The HTML5 build pipeline (via bob.jar) just works. Zero-gravity was a single line in game.project:

[physics]
gravity_y = 0.0
gravity_x = 0.0

Speed gates as a late mechanic. On day 9, I added the speed aura — a particle effect that activates above 700 px/s — and made some checkpoints require that speed to pass. This added skill expression to what was otherwise just “hook and go.” Players have to chain hooks correctly to build enough speed.

Keeping scope tiny. Three short levels. No enemies. No powerups. One mechanic, explored through level design.

What went wrong

Art. I’m not an artist. The final game uses colored squares for most geometry and a spider sprite for the player. The cable rendering is literally Defold’s draw_line debug primitive — single-pixel white lines. It works mechanically but looks like a prototype.

Sound. I added a background music track on day 8 and never got to sound effects. Hook fire, cable snap, speed aura activation, checkpoint hit — all silent. This hurt game feel significantly.

Level design iteration time. Defold’s tile editor is functional but not fast for iterating on level layouts. I’d place anchors, build the game, test the route, realize the spacing was wrong, and repeat. A hot-reload workflow or an in-editor play mode would have saved time.

Late camera work. I didn’t add camera clamping until day 9. For the first week, the camera would happily follow the player off into void space. This wasn’t a huge problem during development but made the game feel unpolished whenever you overshot a section.

Technical decisions I’d revisit

Single-file player script. All player logic — movement, cable management, chain tracking, speed aura, checkpoint handling, restart — lives in one ~250-line script. For a jam this was fine; for anything longer-lived I’d split cable management into its own module.

Manual rope math instead of physics joints. The right call for this game, but it means the cable has no elasticity, no visual sag, no physical interaction with obstacles. A hybrid approach — manual pull force but a visual catenary curve — would look better without changing the feel.

No automated testing. Defold doesn’t have a built-in test runner for game scripts. Every change meant building and manually playing through. For a 10-day jam this is acceptable, but I caught at least two regressions late (camera breaking after restart, checkpoint ordering off by one).

Tools and workflow

• Defold Editor for scene composition, tilemap painting, and builds.
• Git + GitHub with feature branches for major additions (controls rework, level one, assets).
• build.sh wrapping bob.jar for HTML5 bundling — avoids the Defold Editor’s bundle GUI.
• Python HTTP server for local testing (WASM requires HTTP, not file://).

Numbers

• 10 days, ~40 commits across 5 PRs
• 3 playable levels
• ~800 lines of Lua (player, camera, level, checkpoint, HUD, loader, cable, utilities)
• 1 background music track, 0 sound effects
• Final HTML5 bundle: ~3MB

What I’d do next

If I picked this back up: proper cable visuals (sprite-based with catenary math), sound effects for every action, 2–3 more levels that explore the multi-cable mechanic more deeply, and a global leaderboard via a simple REST API. The speed gate mechanic has room to grow — gates that require specific chain counts, or anchors that only activate at certain speeds.

The game works. It feels good to swing through a level at full speed, chaining hooks and barely clearing a speed gate. For 10 days and a first game jam, I’ll take it.

I built this for an internal documentation assistant — engineers asking questions about proprietary system docs that couldn’t be indexed by public LLMs. The corpus was ~2,000 pages of technical documentation across PDFs, markdown, and HTML. Traffic was modest (a few hundred queries per day) but bursty — entire teams would hit it during incidents, which is exactly when you can’t afford it to fall over.

The stack: FastAPI, a columnar database with vector search capabilities, an embedding model behind an API gateway, and an LLM for generation. Everything async, everything behind rate limits.

Adaptive retrieval depth

Not every query needs the same number of documents. “What port does the connector use?” needs 3 chunks. “Explain all the differences between mode A and mode B” might need 10.

I started with a fixed k=5 and quickly saw two failure modes: simple lookups returned irrelevant padding documents that confused the LLM, and complex questions missed critical context because 5 chunks weren’t enough.

def determine_k(query: str) -> int:
    length = len(query)
    if length < 30:
        k = 3
    elif length <= 100:
        k = 5
    else:
        k = 7
    if COMPLEXITY_PATTERN.search(query):
        k += 2
    return max(2, min(k, 10))

The complexity pattern matches terms like “compare”, “list all”, “comprehensive”, “every”. It’s a crude heuristic — query length is a weak proxy for complexity. But it eliminated the worst cases of over-retrieval and under-retrieval without adding an LLM call to classify the query (which would double latency for every request).

Multi-query retrieval with early exit

Single-query retrieval has a blind spot: if the user’s phrasing doesn’t match how the source documents express the concept, you miss relevant chunks. Multi-query generates rephrased variants and merges results.

The problem: multi-query is expensive. An extra LLM call for rephrasing plus N additional vector searches. For most queries the original phrasing works fine — you don’t want to pay that cost unconditionally.

original_results = await self._search_async(query, k)

if high_confidence(original_results, threshold=0.7):
    rewrite_task.cancel()
    docs = [doc for doc, score in original_results if score >= min_similarity]
else:
    variants = await rewrite_task
    docs = merge_results(result_sets, min_similarity)

The query rewrite and original retrieval run concurrently from the start. If the original results score above threshold on at least 3 documents, we cancel the rewrite and skip additional searches. In practice, ~70% of requests take the fast path. The remaining 30% — usually jargon-heavy or vaguely phrased queries — benefit measurably from the rephrased variants.

The early-exit pattern means multi-query adds zero latency to the majority path while still catching the long tail of poorly-phrased queries.

Concurrency control

The vector database and the LLM gateway both had hard connection limits. During an incident, 15 engineers would simultaneously ask questions about the same system, and without backpressure the service would exhaust connection pools and cascade into 429s from the LLM gateway.

self._hana_semaphore = asyncio.Semaphore(4)   # max concurrent DB queries
self._aicore_semaphore = asyncio.Semaphore(3)  # max concurrent LLM calls

async def _search_async(self, query: str, k: int):
    async with self._hana_semaphore, self._aicore_semaphore:
        return await with_retry(self._search, query, k)

Deliberately conservative. Under load, requests queue at the semaphore rather than hammering upstream. The alternative — letting all requests through and handling 429s reactively — creates worse tail latency because retries compound with each other.

The retry layer underneath handles rate limits with exponential backoff, respecting Retry-After headers:

async def with_retry(fn, *args, max_retries=3):
    for attempt in range(max_retries + 1):
        try:
            return await asyncio.to_thread(fn, *args)
        except RateLimitError as exc:
            if attempt == max_retries:
                raise
            retry_after = float(
                exc.response.headers.get("Retry-After", 0)
            ) if exc.response else 0
            delay = max(retry_after, 0.5 * (2 ** attempt))
            await asyncio.sleep(delay)

The asyncio.to_thread is important — the SDK clients are synchronous, so blocking calls go to the thread pool to keep the event loop responsive for other requests.

Structured citations

A RAG answer without citations is just a hallucination with extra steps. Early user feedback was clear: “I don’t trust this answer unless I can verify it.” So the system injects source headers that the LLM can reference:

def format_docs(docs: list[Document]) -> str:
    parts = []
    for doc in docs:
        meta = doc.metadata
        name = meta.get("document_name", "Unknown")
        page = meta.get("page")
        chapter = meta.get("chapter")

        if page is not None:
            ref = f"Chapter: {chapter}, Page {page}" if chapter else f"Page {page}"
            parts.append(f"[Source: {name}, {ref}]\n{doc.page_content}")
        else:
            parts.append(doc.page_content)
    return "\n\n".join(parts)

Documents without pagination metadata (parsed markdown files) are included without a source header. The prompt explicitly instructs the LLM not to cite them — this prevents fabricated page numbers. The trade-off: some answers lack citations even when the information is correct. We accepted this over the alternative of the LLM inventing “Page 47” for a markdown file.

Streaming with error boundaries

The response streams token-by-token. This creates an error handling problem: once you’ve started streaming, you can’t return a JSON error response. The HTTP status is already 200.

async def retrieve_stream(self, query: str) -> AsyncGenerator[str, None]:
    try:
        docs = await self._retrieve(query)
    except DatabaseError:
        yield "Sorry, an error occurred while searching."
        return

    try:
        async for chunk in self.document_chain.astream({...}):
            yield chunk
    except Exception:
        yield "\n\nSorry, an error occurred while generating the answer."

Retrieval errors fail fast — if the database is unreachable, the user gets an immediate message rather than waiting for a timeout. Generation errors degrade gracefully: partial answers are already on the wire, so we append an error notice. Users preferred seeing a partial answer with an error notice over getting nothing.

Timeouts as graceful degradation

Each sub-operation has an independent timeout tuned to its importance:

• Query rewrite: 500ms. If the LLM takes longer to rephrase, proceed with the original phrasing. The rewrite is an optimization, not a requirement.
• Variant retrieval: 1 second per variant. If one rephrased query’s retrieval is slow, merge results from whichever variants completed.

async def _search_async_with_timeout(self, query: str, k: int):
    return await asyncio.wait_for(
        self._search_async(query, k), timeout=self.retrieval_timeout
    )

Under normal load, everything completes within budget. Under burst load (incident-driven traffic), the enhancements gracefully shed while the core path — original query retrieval + generation — always completes. Users get a slightly less optimized answer rather than a timeout error.

What I learned

The production gap in RAG isn’t retrieval quality — it’s operational resilience. The entire service is under 300 lines. Most of that is plumbing: semaphores, retries, timeouts, error boundaries. The actual RAG logic is maybe 40 lines.

Things I’d do differently next time: add a response cache keyed on query embedding similarity (many incident-driven questions are near-duplicates), instrument retrieval quality metrics (track how often multi-query actually changes the result set), and add a fallback path that returns raw document snippets without LLM generation when the gateway is fully saturated. An imperfect answer fast beats a perfect answer never.

This post covers how we wired up distributed tracing: correlation IDs first, then OpenTelemetry, then structured logging that actually made the whole thing queryable.

The problem

A user places an order. The request hits an API gateway, fans out to an inventory service, a payment service, a notification service, and eventually writes to a ledger via Kafka. Each service logs independently. When checkout latency spikes to 12 seconds, you’re searching five different log streams by timestamp, hoping the clocks are synced and the log formats are consistent enough to piece together what happened.

We tried this for three months. The worst incident took 45 minutes to resolve — not because the fix was hard, but because finding the slow service required manually jumping between Kibana indices and guessing at timing overlaps.

Correlation IDs: the quick win

Before going full OpenTelemetry, we started with the simplest possible thing: a UUID generated at the edge, propagated everywhere.

At the gateway (Spring Boot filter):

@Component
public class CorrelationIdFilter extends OncePerRequestFilter {

    @Override
    protected void doFilterInternal(HttpServletRequest request,
                                    HttpServletResponse response,
                                    FilterChain chain) throws ServletException, IOException {
        String correlationId = request.getHeader("X-Correlation-ID");
        if (correlationId == null) {
            correlationId = UUID.randomUUID().toString();
        }
        MDC.put("correlationId", correlationId);
        response.setHeader("X-Correlation-ID", correlationId);
        try {
            chain.doFilter(request, response);
        } finally {
            MDC.remove("correlationId");
        }
    }
}

Propagation rules we settled on:

• HTTP calls: pass as X-Correlation-ID header (added to our shared RestTemplate config)
• Kafka messages: set in record headers, not the payload (avoids schema changes)
• Async workers: extract from message metadata before processing, put in MDC

This alone cut our mean-time-to-identify from ~30 minutes to ~8 minutes. One search by correlation ID across all indices would surface every log line for a transaction. The limitation: no timing information, no parent-child relationships, no visualization.

OpenTelemetry: when correlation IDs aren’t enough

We evaluated three options: Zipkin (lighter, less active development), Jaeger (mature, but self-hosted complexity), and OpenTelemetry with a managed backend. We went with OTel exporting to Grafana Tempo — mainly because we already had Grafana for metrics and didn’t want another UI.

Basic setup (Spring Boot with OTel SDK):

@Configuration
public class TracingConfig {

    @Bean
    public OpenTelemetry openTelemetry() {
        SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
            .addSpanProcessor(BatchSpanProcessor.builder(
                OtlpGrpcSpanExporter.builder().build()
            ).build())
            .build();

        return OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .setPropagators(ContextPropagators.create(
                W3CTraceContextPropagator.getInstance()
            ))
            .build();
    }
}

Instrumenting a service call:

@Service
public class PaymentService {

    private final Tracer tracer;

    public PaymentService(OpenTelemetry openTelemetry) {
        this.tracer = openTelemetry.getTracer("payment-service");
    }

    public PaymentResult processPayment(String orderId, String method, long amount) {
        Span span = tracer.spanBuilder("process_payment").startSpan();
        try (Scope scope = span.makeCurrent()) {
            span.setAttribute("order.id", orderId);
            span.setAttribute("payment.method", method);
            PaymentResult result = paymentClient.charge(orderId, amount);
            span.setAttribute("payment.status", result.getStatus());
            return result;
        } finally {
            span.end();
        }
    }
}

Context propagation is handled by W3C Trace Context headers (traceparent, tracestate). The OTel Java agent auto-instruments most HTTP clients and Kafka producers/consumers, so we only wrote manual spans for business-critical paths where we wanted custom attributes.

The trade-off we hit: the OTel Java agent adds ~50ms to startup and a small per-request overhead (~2ms in our measurements). For our latency budget this was fine. We considered the manual SDK-only approach (no agent) but decided the auto-instrumentation coverage was worth the overhead.

Where we drew the line: instrument at service boundaries only — incoming request, outgoing HTTP call, message publish, message consume. We explicitly decided not to instrument internal method calls. The signal-to-noise ratio drops fast when you trace everything.

Structured logging

We had structured logging before tracing, but it wasn’t connected. The missing piece was injecting trace context into every log line automatically.

{
  "timestamp": "2024-07-15T02:13:47Z",
  "level": "info",
  "service": "payment-service",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "correlation_id": "ord-18473-a8f2",
  "message": "payment processed",
  "order_id": "18473",
  "amount_cents": 4500,
  "duration_ms": 230
}

We kept both the correlation ID (business identifier — the order ID) and the trace ID (infrastructure identifier). This lets you query from either direction: “show me everything for order 18473” or “show me everything in this trace.” Different people ask different questions — support engineers think in orders, on-call engineers think in traces.

Implementation (Logback with MDC):

<!-- logback-spring.xml -->
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
        <includeMdcKeyName>correlationId</includeMdcKeyName>
        <includeMdcKeyName>traceId</includeMdcKeyName>
        <includeMdcKeyName>spanId</includeMdcKeyName>
    </encoder>
</appender>

@Component
public class TraceContextLogger implements HandlerInterceptor {

    @Override
    public boolean preHandle(HttpServletRequest request,
                             HttpServletResponse response,
                             Object handler) {
        Span span = Span.current();
        SpanContext ctx = span.getSpanContext();
        if (ctx.isValid()) {
            MDC.put("traceId", ctx.getTraceId());
            MDC.put("spanId", ctx.getSpanId());
        }
        return true;
    }
}

What actually changed

The first incident after full rollout: alert fires for checkout latency at 2 AM. The alert itself now includes a trace ID (we configured AlertManager to attach it from the exemplar). Open Tempo, paste the trace ID, see the full waterfall. A downstream ledger service is taking 8 seconds on a database query — connection pool exhausted because a batch job was running during peak hours. Four minutes from alert to root cause.

Before tracing, that same incident pattern took 30-45 minutes. The fix was always simple once you found the slow service. The cost was in the finding.

The less obvious win: support tickets. “What happened to order #18473?” used to mean 20 minutes of log archaeology. Now it’s one search. We built a small internal tool that takes an order ID, finds the correlation ID, pulls the trace, and renders the timeline. Support engineers use it directly without paging on-call.

What I’d skip if doing it again: we spent two weeks building custom dashboards for trace metrics (p99 per service, error rates by span). Grafana Tempo’s built-in service graph and RED metrics dashboard gave us 90% of that for free. Should have started there.

What I wouldn’t skip: keeping the correlation ID separate from the trace ID. Some teams just use the trace ID for everything. But traces are infrastructure-scoped — they break across async boundaries, batch jobs, retry queues. The business correlation ID survives all of that because it’s just a string you propagate manually. When the Kafka consumer picks up a failed message three hours later, the trace is new but the correlation ID is the same.