Summary

• Date: 2026-04-06
• Pages visited:
  • https://github.github.com/gh-aw/ (Home)
  • https://github.github.com/gh-aw/setup/quick-start/ (Quick Start)
  • https://github.github.com/gh-aw/setup/cli/ (CLI Commands)
• Overall impression: As a first-time user, the home page clearly communicates the project's purpose (AI-powered repo automation in GitHub Actions). The Quick Start is well-structured and guides you through a concrete goal. However, several friction points slow down getting started for true beginners.

⚠️ Note: Visual screenshots were unavailable due to network isolation between the Playwright container and the agent container. All analysis was performed via curl + HTML parsing.

🔴 Critical Issues

1. Search is broken in dev mode

On every page, a banner reads: "Search is only available in production builds. Try building and previewing the site to test it out locally."

As a noob, I have no idea what a "production build" is. The search bar appears but doesn't work — it's the first thing I tried to find "how to install." This is actively misleading for testing/demo environments.

Recommendation: Hide the search input entirely when search is unavailable, or replace with a clear message like "Search unavailable — use the sidebar instead."

2. COPILOT_GITHUB_TOKEN — confusing secret explained too briefly

Quick Start Step 2 mentions:

"Set up the required secret — COPILOT_GITHUB_TOKEN (a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN)"

As a beginner, "a separate GitHub token" raises many questions: How do I create it? What scopes does it need? Where do I add it? The inline note is minimal and defers to the Authentication page with no context. New users are likely to get stuck or misconfigure this.

Recommendation: Add a one-sentence inline summary like "You'll need a PAT (Personal Access Token) with Copilot access — see Authentication for exact steps." or embed the key steps inline.

🟡 Confusing Areas

3. "Frontmatter" undefined on first mention

The About Workflows page explains the YAML block at the top of a .md file as "frontmatter" without defining it for a general audience. The term appears on the Overview page, Quick Start, and Creating Workflows — always assumed knowledge.

Recommendation: Add a short tooltip/callout on first use: "Frontmatter is the YAML block between --- markers at the top of a markdown file that configures workflow settings." The Glossary has a definition but isn't linked from these early pages.

4. Video "Your browser doesn't support HTML5 video" — no fallback text

The Quick Start page embeds a video with <video> element. The fallback text is: "Your browser doesn't support HTML5 video. Download the video here."

While there is a download link, new users may be confused if the video doesn't load (e.g., in a Codespace browser or corporate proxy). The video itself is a key part of understanding the install flow.

Recommendation: Add a brief textual summary of what the video shows (e.g., a numbered list of the 3 actions performed).

5. The gh aw add-wizard vs gh aw add distinction unclear

The CLI Commands page lists both add-wizard ("Add workflows with interactive guided setup") and add ("Add workflows from other repositories (non-interactive)"). As a noob, I don't understand when to use one vs. the other. Quick Start uses add-wizard but never explains why it's preferred for beginners.

Recommendation: Add a one-line "When to use each" note on the CLI page, e.g., "Use add-wizard for guided setup; use add for scripting/automation."

6. The note about early development is prominent but lacks guidance

The home page displays a large ⓘ Note: GitHub Agentic Workflows is in early development... warning. For a new user, this raises anxiety: Is it safe to try? Will it break my repository?

Recommendation: Pair the warning with a reassurance sentence, e.g., "For a safe first try, use a test repository." or link directly to the security architecture page.

7. Sidebar is very long — hard to navigate

The sidebar contains 70+ items across ~15 categories (Setup, Guides, Design Patterns, Reference, Troubleshooting). For a beginner, the cognitive load is high. There's no "Beginner path" or "Start here" highlight.

Recommendation: Consider adding a "Getting Started" group that only shows: Overview → Quick Start → Creating Workflows → CLI Commands → FAQ, collapsed by default for experienced users.

8. gh aw compile requirement not prominent in Quick Start

In "Step 4 - Customize your workflow (optional)", there's a note: "If you have changed the frontmatter... regenerate the workflow YAML by running: gh aw compile". This is buried in an optional step.

But beginners editing a workflow for the first time will often not know they need to recompile. The link between .md → .lock.yml is non-obvious.

Recommendation: Add a callout earlier: "Every time you edit the YAML frontmatter of a workflow, run gh aw compile to regenerate the Actions YAML."

🟢 What Worked Well

1. Home page is clear and compelling — the headline "repository automation, running the coding agents you know and love, with strong guardrails" immediately communicates the value proposition.

2. Security architecture section on home page is excellent — the 5-layer security model with clear explanations builds trust. The flowchart is a nice visual touch.

3. Prerequisites are explicitly listed — Quick Start clearly lists AI Account, GitHub Repository, GitHub Actions enabled, gh CLI version, and OS. No guessing required.

4. Estimated time is shown — "Estimated time: 10 minutes" at the top of Quick Start sets expectations well.

5. "Most Common Commands" table on CLI page — the prominent table at the top of the CLI page is exactly what a new user needs: a scannable cheat sheet.

6. add-wizard is the recommended path — guiding beginners through an interactive setup wizard is the right UX choice.

7. Troubleshooting tip in Step 2 — the "Tip: Having trouble? Check your repository secrets, see the FAQ and Common Issues" is well-placed.

8. "What's next?" section — the three links at the bottom of Quick Start (Creating Workflows, How Workflows Work, FAQ) provide a clear continuation path.

Recommendations (Prioritized)

Quick Wins

1. 🔴 Hide or replace non-functional search with a visible "Use the sidebar to navigate" hint
2. 🟡 Define "frontmatter" inline on first use in Quick Start and Overview pages
3. 🟡 Expand the COPILOT_GITHUB_TOKEN inline explanation with scopes/creation steps
4. 🟡 Add text summary alongside videos for users where video doesn't load

Medium-Term

5. 🟡 Add "When to use add vs add-wizard" comparison on CLI page
6. 🟡 Add a gh aw compile reminder callout earlier in Quick Start
7. 🟡 Add "test repository" recommendation next to the early-development warning

Longer-Term

8. 🟢 Create a "Beginner path" sidebar group with the 5 essential pages
9. 🟢 Add inline Glossary tooltips for jargon terms (frontmatter, lock file, safe outputs) on first use

References:

• §24033154428

Generated by Documentation Noob Tester · ● 3.5M · ◷

• expires on Apr 7, 2026, 1:39 PM UTC