Background

Hi team, I'm working with @chunyu3 on TypeSpec Authoring feature for inner loop project. And I'm trying to use waza to test the skill. Basically our skill is to help Azure service team user to generate TypeSpec code that follow our guidelines.

Questions

1. When waza run completes, is there a way for me to check the fixture workspace in each test session? Because when test run fails, I'd like to debug to see how the generated code look like.

   

   In waza --debug logs, I saw above dir: C:\Users\haolingdong\AppData\Local\Temp\waza-1926329688, it exists, but the content is empty.

   

   I guess there are some shut down process to clean up the session workspace. But based on my research, I did not find Waza currently have a built-in flag to disable session data cleanup. Would be really appreciated if you could let me know how to debug and investigate the generated code for each running test session.

2. After I run npm install in before test hook, I noticed that node_modules are created inside fixture folder. This is a bit surprised to be because I thought node_modules will be created inside the test workspace for each test session. Is this expected?

hooks:
before_run:
  - command: "npm install"
    working_directory: "./fixtures/Microsoft.Widget/Widget"
    error_on_fail: true

If that's expected, should I put node_modules to the file path in task spec like below?

id: 001006-version-add-preview-after-stable
name: "Versioning: Add preview after stable"
description: |
  Test that the skill handles adding a new preview version after a stable version.
tags:
  - basic
  - happy-path
  - versioning
inputs:
  prompt: "add a new preview version `2025-05-04-preview` to my service widget resource manager. Carry over all the features from the latest stable version"
  files:
    - path: Microsoft.Widget/Widget/employee.tsp
    - path: Microsoft.Widget/Widget/main.tsp
    - path: Microsoft.Widget/Widget/shared.tsp
    - path: Microsoft.Widget/Widget/tspconfig.yaml
    - path: Microsoft.Widget/Widget/package.json
++    - path: Microsoft.Widget/Widget/node_modules/*

3. Is there any suggestions on starting the azsdk MCP server before test?
   code here: https://github.com/Azure/azure-rest-api-specs/blob/main/.vscode/mcp.json

Reproduce

My code can be found here: Azure/azure-rest-api-specs#41195
One can use this example test case to reproduce: https://github.com/Azure/azure-rest-api-specs/blob/cfd4bd38ac21a6b19f24aff318cc674e07fc6a29/.github/skills/evals/azure-typespec-author/tasks/001006-version-add-preview-after-stable.yaml

Environment Info:

1. OS: Windows 11
2. waza version: the .exe @richardpark-msft shared to me last week (to solve session idel error). :)

Thank you so much for your advice in advance!

/cc @ArthurMa1978 @lirenhe @lmazuel