Skip to content

Fix (Proxy resolution): use proxied /api/health and sanitize Vite API URL#409

Closed
armand0e wants to merge 2 commits into
coleam00:mainfrom
armand0e:fix/ui-health-proxy-and-api-url
Closed

Fix (Proxy resolution): use proxied /api/health and sanitize Vite API URL#409
armand0e wants to merge 2 commits into
coleam00:mainfrom
armand0e:fix/ui-health-proxy-and-api-url

Conversation

@armand0e

@armand0e armand0e commented Aug 21, 2025

Copy link
Copy Markdown

— Summary —

The UI showed a “Backend Service Startup Failure” modal even when the backend was healthy. Root cause was the frontend calling an absolute API URL derived from HOST=0.0.0.0 (unreachable from the browser on a different machine, same network). We now consistently use the proxied endpoint and sanitize VITE_API_URL.

— Root Cause —

  • docker-compose.yml sets VITE_API_URL=http://${HOST}:${ARCHON_SERVER_PORT}. With HOST=0.0.0.0 (valid for binding, not for browsers), the frontend attempted requests to http://0.0.0.0:8181, which fails.
  • Previous MainLayout.tsx used ${baseUrl}/health for initial readiness checks, so it inherited the unreachable URL.

TypeScript env typings didn’t declare variables we referenced, leading to lints and friction.

— What Changed —

archon-ui-main/src/components/layouts/MainLayout.tsx

  • Health check now calls /api/health directly (proxied through Vite), removing reliance on absolute URLs during startup.

archon-ui-main/src/config/api.ts

  • If VITE_API_URL parses to hostname 0.0.0.0, fallback to relative base '' so the proxy handles routing.

In development, construct the API base using window.location + VITE_PORT (with a safe 8181 fallback).

archon-ui-main/src/env.d.ts

Added VITE_API_URL, VITE_PORT, DEV, PROD, MODE typings to satisfy Typescript

— Why This Fix Is Safe —

Vite proxy (vite.config.ts) already routes /api to archon-server, so switching to relative /api/* is the intended path in dev/Docker.

Sanitizing VITE_API_URL=0.0.0.0 only affects unusable configurations in browsers and falls back to the proxy path.

— Testing/Validation —

Open the UI and verify the startup modal no longer appears once the backend completes init.

Visit http://localhost:${ARCHON_UI_PORT:-3737}/api/health and confirm JSON includes ready: true post-initialization.

Verify credentials pages function (calls under /api/* work normally).

Confirm no TypeScript lint errors for env usage.

— Potential Follow-ups —

Consider removing VITE_API_URL from docker-compose.yml to always rely on the proxy in dev.

— Risks —

Minimal risk. Simply aligning calls to the Vite proxy and hardening env handling.

Summary by CodeRabbit

  • Bug Fixes

    • Health check now uses a proxied relative endpoint for consistent connectivity across environments.
    • API URL resolution improved to respect environment settings and avoid 0.0.0.0 host issues.
    • Several API routes now accept IDs containing slashes for more flexible resource addressing.
  • New Features

    • RAG settings now expose an OpenAI base URL field and saving shows success/error toasts.
    • Crawl option to restrict crawling to the start-path scope for finer control.
    • Ability to explicitly clear saved API keys.
  • Chores

    • Expanded environment variable typings and example .env entries; large documentation refresh for RAG and API docs.

…_URL=0.0.0.0

- Fixed frontend “Backend Service Startup Failure” caused by absolute API URL resolving to http://0.0.0.0:8181 in the browser.
- MainLayout health check now calls '/api/health' via Vite proxy, avoiding unreachable hosts and CORS issues.
- getApiUrl() falls back to relative base when VITE_API_URL hostname is 0.0.0.0 and builds dev URL using VITE_PORT (with 8181 fallback).
- Added typings for VITE_API_URL, VITE_PORT and Vite runtime flags to resolve TS lints.

Impact:
- Fixed self-hosted Supabase backend resolution.
- More robust in Docker, local dev, and varied HOST settings.
- Reduces risk of UI regressions where absolute URLs misroute traffic.

Refs:
- docker-compose.yml sets VITE_API_URL from HOST and ARCHON_SERVER_PORT.
- vite.config.ts proxies '/api' to the backend service.
@coderabbitai

coderabbitai Bot commented Aug 21, 2025

Copy link
Copy Markdown

Walkthrough

Renames and centralizes LLM base-URL keys to OPENAI_BASE_URL across UI, services, docs, and migrations; improves API URL detection (VITE_API_URL/VITE_PORT), switches UI health check to proxied /api/health, tightens provider base_url resolution with credentials/env fallbacks, adds crawling scoping and propagated scoped source IDs, and updates several route/path and test behaviors.

Changes

Cohort / File(s) Summary
UI: health check & API config
archon-ui-main/src/components/layouts/MainLayout.tsx, archon-ui-main/src/config/api.ts, archon-ui-main/src/env.d.ts, .env.example
Replace absolute health-check call with proxied "/api/health"; enhance getApiUrl to parse VITE_API_URL, treat 0.0.0.0 as relative (use proxy), prefer VITE_PORT (default 8181); add typings for VITE_API_URL, DEV, PROD, MODE; add OPENAI_BASE_URL example and comments in .env.example.
RAG / credentials rename (UI & service layer)
archon-ui-main/src/components/settings/RAGSettings.tsx, archon-ui-main/src/components/settings/APIKeysSection.tsx, archon-ui-main/src/services/credentialsService.ts, migration/complete_setup.sql, docs files (docs/..., docs/docs/rag.mdx, docs/docs/api-reference.mdx)
Rename LLM_BASE_URL → OPENAI_BASE_URL in props, credentials, defaults, migration SQL, and documentation examples; extend API keys filter to include keys containing BASE_URL; RAG settings Save now persists via credentialsService.updateRagSettings with async try/catch and toasts; docs updated for variable naming and many editorial improvements.
Provider resolution & cache
python/src/server/services/credential_service.py, python/src/server/services/llm_provider_service.py, python/src/server/services/source_management_service.py
Add OpenAI base_url resolution from credentials/env (OPENAI_BASE_URL / OPENAI_API_BASE), centralize provider config resolution via new helper, conditionally pass base_url when present, add invalidate_provider_cache(keys_changed) and call invalidation on credential set/delete for provider-related keys.
Source/source-id propagation (crawling & storage)
python/src/server/services/crawling/crawling_service.py, python/src/server/services/crawling/document_storage_operations.py, python/src/server/services/crawling/code_extraction_service.py
Introduce scoped_source_id derived from URL path, pass it as provided_source_id into storage and code-extraction flows; add provided_source_id optional params so extracted code examples use provided scoped ID when given.
Crawl scoping strategy
python/src/server/services/crawling/strategies/recursive.py
Add restrict_to_start_path: bool = True param, build per-netloc allowed base paths and apply within_scope checks to limit enqueued URLs to the start-path scope when enabled (default preserved unless explicitly changed).
API route matching (path params)
python/src/server/api_routes/knowledge_api.py
Allow slashes in source_id by changing routes to {source_id:path} for several knowledge-item endpoints (PUT, DELETE, GET code-examples, POST refresh).
UI tests / simulation change
python/src/server/api_routes/tests_api.py
Remove real Docker-based UI test execution stream; replace with simulation endpoint path (streamed output simulation only); removed process wait, exit code handling, and coverage copying logic.
Streamlit pages / environment inputs
original_archon/.../environment.py (multiple iterations)
Rename local LLM base URL variables to OPENAI_BASE_URL across streamlit environment pages, allow saving empty API key values to clear keys, add minor UI/CSS tweaks and messages.
Tests
python/tests/test_async_llm_provider_service.py
Add AsyncMock support and a test verifying OpenAI client creation with custom base_url; adjust mocks to reflect OPENAI_BASE_URL override flow.
Docs & editorial
docs/docs/api-reference.mdx, docs/docs/rag.mdx
Extensive formatting, table/quote normalization, added presets and examples, provider examples updated to use OPENAI_BASE_URL for non-OpenAI providers; many editorial improvements without behavioral changes.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant UI as UI / RAG Settings
    participant CS as CredentialsService
    participant LS as LLMProviderService
    participant C as CredentialsStore/Env

    UI->>CS: updateRagSettings(ragSettings with OPENAI_BASE_URL)
    CS->>C: persist credential (OPENAI_BASE_URL)
    CS->>LS: invalidate_provider_cache(keys_changed=[...OPENAI_BASE_URL...])
    LS-->>CS: ack
Loading
sequenceDiagram
    autonumber
    participant Crawler as CrawlingService
    participant Strategy as RecursiveCrawlStrategy
    participant Storage as DocumentStorageOperations
    participant CodeExt as CodeExtractionService

    Crawler->>Strategy: crawl_recursive_with_progress(start_urls, restrict_to_start_path=true)
    Strategy->>Strategy: build scopes_by_netloc from start_urls
    Strategy->>Strategy: within_scope(candidate_url)?
    alt within scope
        Strategy->>Crawler: enqueue url
    else out of scope
        Strategy-->>Strategy: skip
    end
    Crawler->>Storage: process_and_store_documents(..., original_source_id=scoped_source_id)
    Storage->>CodeExt: extract_and_store_code_examples(..., provided_source_id=scoped_source_id)
    CodeExt->>Storage: store code examples with source_id=scoped_source_id
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

I nudge the proxy, twitch my nose with cheer,
OPENAI_BASE_URL now hops quite near.
Scoped paths and creds, neatly aligned,
Health checked through tunnels, no string left behind.
A rabbit's thump: the codebase feels clear. 🐇✨

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
archon-ui-main/src/config/api.ts (1)

34-34: Consider removing the any type cast.

With the proper TypeScript definitions now in place in env.d.ts, the any cast is no longer necessary.

-  const port = (import.meta.env as any).VITE_PORT as string | undefined;
+  const port = import.meta.env.VITE_PORT;
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between 8b29d20 and 894d8de.

📒 Files selected for processing (3)
  • archon-ui-main/src/components/layouts/MainLayout.tsx (1 hunks)
  • archon-ui-main/src/config/api.ts (2 hunks)
  • archon-ui-main/src/env.d.ts (1 hunks)
🔇 Additional comments (4)
archon-ui-main/src/env.d.ts (1)

6-9: LGTM! Environment variable types properly defined.

The added TypeScript definitions provide proper typing for the Vite environment variables used throughout the codebase, including the optional VITE_API_URL and standard Vite environment flags.

archon-ui-main/src/components/layouts/MainLayout.tsx (1)

48-49: Good fix for the proxy routing issue.

The change from absolute URL to relative /api/health properly leverages the Vite proxy configuration and resolves the browser reachability issue with 0.0.0.0 hostnames. The comment clearly explains the reasoning.

archon-ui-main/src/config/api.ts (2)

12-24: Robust URL sanitization logic for 0.0.0.0 fallback.

The implementation correctly identifies unreachable 0.0.0.0 hostnames and falls back to relative URLs for Vite proxy routing. The try-catch block properly handles malformed URLs by preserving the original value.


34-40: Clean port resolution with proper fallback.

The type casting to string | undefined aligns with the updated TypeScript definitions, and the fallback to port 8181 provides a sensible default for development environments.

@Wirasm Wirasm requested a review from tazmon95 August 22, 2025 06:03
@Wirasm

Wirasm commented Aug 22, 2025

Copy link
Copy Markdown
Collaborator

@tazmon95 Could you please take a look at this pr?

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (10)
.env.example (1)

62-63: Align DISPATCHER_CHECK_INTERVAL defaults between docs and UI

  • In .env.example (lines ~62) the default for DISPATCHER_CHECK_INTERVAL is documented as 0.5 seconds.
  • In archon-ui-main/src/services/credentialsService.ts (line 169) the code still defaults it to 30 seconds.
  • Moreover, the current parsing logic uses parseInt, so any fetched value of 0.5 would be truncated to 0 and ignored.

Action items:

  • Change the hard-coded default in credentialsService.ts:
     // Advanced Settings defaults
  • DISPATCHER_CHECK_INTERVAL: 30,
  • DISPATCHER_CHECK_INTERVAL: 0.5,
- Update the parsing step for numeric settings (including DISPATCHER_CHECK_INTERVAL) to use parseFloat:
```diff
 // Number fields
-  (settings as any)[cred.key] = parseInt(cred.value, 10) ?? (settings as any)[cred.key];
+  (settings as any)[cred.key] = parseFloat(cred.value) ?? (settings as any)[cred.key];
  • Add or update tests/validation to ensure fractional interval values are accepted and applied correctly.
archon-ui-main/src/services/credentialsService.ts (2)

146-172: Fix default and parsing for DISPATCHER_CHECK_INTERVAL; honor fractional seconds.

Two issues:

  • Default is 30 here, but migrations/docs use 0.5s.
  • The parser later treats DISPATCHER_CHECK_INTERVAL as an integer, so a stored value of "0.5" becomes 0 and falls back to 30.

Result: UI displays and persists an incorrect interval. Align default to 0.5 and parse as float.

Apply this diff to fix defaults:

       // Advanced Settings defaults
       MEMORY_THRESHOLD_PERCENT: 80,
-      DISPATCHER_CHECK_INTERVAL: 30,
+      DISPATCHER_CHECK_INTERVAL: 0.5,
       CODE_EXTRACTION_BATCH_SIZE: 50,
       CODE_SUMMARY_MAX_WORKERS: 3,

And update numeric parsing (see next comment) to handle floats.


174-211: Robust numeric parsing: don’t discard valid 0 or fractional values.

  • Integer fields currently use parseInt(...) || default, which treats 0 as falsy and reverts to the default.
  • DISPATCHER_CHECK_INTERVAL should be parsed as float (see above). CRAWL_DELAY_BEFORE_HTML already is, but also incorrectly uses || 0.5, which discards legitimate 0.0 if ever allowed.

Use NaN checks instead of falsy checks; move DISPATCHER_CHECK_INTERVAL to the float parsing branch.

Apply this diff:

-        else if (
+        else if (
           [
             "CONTEXTUAL_EMBEDDINGS_MAX_WORKERS",
             "CRAWL_BATCH_SIZE",
             "CRAWL_MAX_CONCURRENT",
             "CRAWL_PAGE_TIMEOUT",
             "DOCUMENT_STORAGE_BATCH_SIZE",
             "EMBEDDING_BATCH_SIZE",
             "DELETE_BATCH_SIZE",
             "MEMORY_THRESHOLD_PERCENT",
-            "DISPATCHER_CHECK_INTERVAL",
             "CODE_EXTRACTION_BATCH_SIZE",
             "CODE_SUMMARY_MAX_WORKERS",
           ].includes(cred.key)
         ) {
-          (settings as any)[cred.key] =
-            parseInt(cred.value || "0", 10) || (settings as any)[cred.key];
+          const parsed = parseInt(cred.value ?? "", 10);
+          if (!Number.isNaN(parsed)) {
+            (settings as any)[cred.key] = parsed;
+          }
         }
-        // Float fields
-        else if (cred.key === "CRAWL_DELAY_BEFORE_HTML") {
-          settings[cred.key] = parseFloat(cred.value || "0.5") || 0.5;
+        // Float fields
+        else if (["CRAWL_DELAY_BEFORE_HTML", "DISPATCHER_CHECK_INTERVAL"].includes(cred.key)) {
+          const parsed = parseFloat(cred.value ?? "");
+          if (!Number.isNaN(parsed)) {
+            (settings as any)[cred.key] = parsed;
+          }
         }
archon-ui-main/src/components/settings/RAGSettings.tsx (1)

161-169: Fix NaN risk when incrementing/decrementing CONTEXTUAL_EMBEDDINGS_MAX_WORKERS

If CONTEXTUAL_EMBEDDINGS_MAX_WORKERS is undefined, the +/- handlers produce NaN (e.g., undefined + 1). Also, value binding should default.

Apply defaults on read and in handlers:

- value={ragSettings.CONTEXTUAL_EMBEDDINGS_MAX_WORKERS}
+ value={ragSettings.CONTEXTUAL_EMBEDDINGS_MAX_WORKERS ?? 3}
@@
- onClick={() => setRagSettings({
-   ...ragSettings,
-   CONTEXTUAL_EMBEDDINGS_MAX_WORKERS: Math.min(ragSettings.CONTEXTUAL_EMBEDDINGS_MAX_WORKERS + 1, 10)
- })}
+ onClick={() => {
+   const curr = ragSettings.CONTEXTUAL_EMBEDDINGS_MAX_WORKERS ?? 3;
+   setRagSettings({
+     ...ragSettings,
+     CONTEXTUAL_EMBEDDINGS_MAX_WORKERS: Math.min(curr + 1, 10),
+   });
+ }}
@@
- onClick={() => setRagSettings({
-   ...ragSettings,
-   CONTEXTUAL_EMBEDDINGS_MAX_WORKERS: Math.max(ragSettings.CONTEXTUAL_EMBEDDINGS_MAX_WORKERS - 1, 1)
- })}
+ onClick={() => {
+   const curr = ragSettings.CONTEXTUAL_EMBEDDINGS_MAX_WORKERS ?? 3;
+   setRagSettings({
+     ...ragSettings,
+     CONTEXTUAL_EMBEDDINGS_MAX_WORKERS: Math.max(curr - 1, 1),
+   });
+ }}

Also applies to: 182-187, 199-204

docs/docs/api-reference.mdx (3)

16-24: Authentication description contradicts current system behavior

Docs say “API key-based… Future JWT”, but project-wide pattern is no authentication on endpoints.

Update the section to reflect current reality and, if applicable, add a forward-looking note:

- **Authentication**: Currently API key-based through settings. Future versions will support JWT tokens.
+ **Authentication**: No authentication is enforced on API endpoints at present. Future versions may introduce optional authentication.

If auth is in-progress elsewhere, reference that explicitly to avoid confusion.


2011-2022: Fix event name and syntax error in Socket.IO example

Example currently listens for crawl_completed and has a connect_error handler with a stray brace.

-    socket.on('progress_update', (data) => {
+    socket.on('progress_update', (data) => {
       onMessage(data);
     });
-
-    socket.on('crawl_completed', (data) => {
+    socket.on('progress_complete', (data) => {
       onMessage(data);
     });
 
     socket.on('connect_error', (error) => {
       console.error('Socket.IO error:', error);
-    };
+    });

370-401: Update API health check documentation to reflect all existing endpoints

Please revise the System Information and API Health Check sections (around lines 370–401 in docs/docs/api-reference.mdx) to accurately document every health endpoint exposed by the server:

  • GET /health
    • System readiness check (includes credential loading)
  • GET /api/health
    • Alias for /health used by the UI proxy
  • GET /api/internal/health
    • Internal API health check
  • GET api/knowledge/health
    • Knowledge API health check
  • GET api/mcp/health
    • MCP API health check
  • GET api/bug-report/health
    • Bug reporting service health check

Ensure each path’s purpose is clearly stated. If the /api/health alias is intended to be the primary UI-facing endpoint and /health will be deprecated, mark /health as deprecated; otherwise leave both supported.

Applies also to the duplicate documentation in lines 1801–1816.

python/src/server/services/source_management_service.py (1)

181-209: External API call lacks retry/backoff; add a minimal, bounded retry

Guidelines require retries with exponential backoff for external calls. A small loop improves resiliency to transient 429/5xx.

-        response = client.chat.completions.create(
-            model=model_choice,
-            messages=[
-                {
-                    "role": "system",
-                    "content": "You are a helpful assistant that provides concise library/tool/framework summaries.",
-                },
-                {"role": "user", "content": prompt},
-            ],
-        )
+        # Minimal retry with exponential backoff (1s, 2s, 4s)
+        import time  # at top-level if preferred
+        last_exc = None
+        for attempt in range(3):
+            try:
+                response = client.chat.completions.create(
+                    model=model_choice,
+                    messages=[
+                        {
+                            "role": "system",
+                            "content": "You are a helpful assistant that provides concise library/tool/framework summaries.",
+                        },
+                        {"role": "user", "content": prompt},
+                    ],
+                )
+                break
+            except Exception as e:
+                last_exc = e
+                if attempt < 2:
+                    time.sleep(2**attempt)
+                else:
+                    raise

Note: add import time near other imports.

python/src/server/services/credential_service.py (1)

524-528: Update Streamlit pages to use “LLM_PROVIDER” or provide an alias for “llm_provider”

The search shows that the legacy lowercase key is still being referenced in multiple Streamlit environment files. These will silently break if the lowercase session key isn’t populated from the new uppercase‐based provider setting. To fully switch over:

• original_archon/streamlit_pages/environment.py: lines 35–36, 66–67, 81–82, 118–119, 133–134, 138–139
• original_archon/iterations/v5-parallel-specialized-agents/streamlit_pages/environment.py: same blocks at lines 35–36, 66–67, 81–82, 118–119, 133–134, 138–139
• original_archon/iterations/v6-tool-library-integration/streamlit_pages/environment.py: same blocks at lines 35–36, 66–67, 81–82, 118–119, 133–134, 138–139

Action items:

  1. Replace all occurrences of st.session_state.llm_provider key checks ("llm_provider") with "LLM_PROVIDER", matching the new env var naming.
  2. When initializing session state, read from profile_env_vars.get("LLM_PROVIDER", ...) into a unified key (either the new uppercase key or maintain lowercase as an alias for backward compatibility).
  3. Update any selectors or rerun logic that references the old key to use the unified key.

This ensures a clean migration off the legacy lowercase key without runtime errors or unexpected defaults.

python/src/server/services/crawling/code_extraction_service.py (1)

142-173: Add a test to verify provided_source_id is persisted
There are currently no existing tests that exercise the provided_source_id parameter in CodeExtractionService.extract_and_store_code_examples. To prevent regressions, please add a unit (or integration) test that:

  • Instantiates or mocks a CodeExtractionService
  • Calls extract_and_store_code_examples(..., provided_source_id="my-source-id")
  • Intercepts the payload sent to your storage layer (e.g. by mocking the write client)
  • Asserts that in each stored code‐example record, metadata.source_id == "my-source-id"

Suggested location and test scaffold:

• File: python/tests/test_code_extraction_service.py
• Example test:

import pytest
from unittest.mock import AsyncMock
from server.services.crawling.code_extraction_service import CodeExtractionService

@pytest.mark.asyncio
async def test_extract_and_store_code_examples_preserves_provided_source_id(mocker):
    # Arrange
    service = CodeExtractionService(...)
    # Mock out the lower‐level storage call
    mock_store = AsyncMock()
    service._store_code_examples = mock_store
    crawl_results = [{"url": "https://example.com", "markdown": "```py\nprint('hi')\n```"}]
    url_to_full = {"https://example.com": "```py\nprint('hi')\n```"}

    # Act
    await service.extract_and_store_code_examples(
        crawl_results,
        url_to_full,
        provided_source_id="test-source-123"
    )

    # Assert: ensure _store_code_examples was called and incoming payload has the provided_source_id
    assert mock_store.called, "Expected storage call"
    _, kwargs = mock_store.call_args
    for record in kwargs["code_examples"]:
        assert record.metadata.source_id == "test-source-123"

Also update or add similar assertions around lines 213–214 where provided_source_id is threaded through your storage logic.

🧹 Nitpick comments (30)
.env.example (1)

43-50: Good addition: OPENAI_BASE_URL override is clear and actionable.

This section helps local devs route via a proxy/self-hosted OpenAI-compatible endpoint. One small enhancement: given this PR adds Vite env typings and proxy behavior, include commented Vite overrides to reduce confusion during local setup and make the UI/server behavior explicit.

Apply this diff to add optional Vite overrides (commented) for clarity:

 OPENAI_BASE_URL=http://localhost:8080/v1
 # OPENAI_API_BASE=
 
+# Frontend (Vite) overrides — optional (proxy is used by default)
+# If you need to bypass the Vite proxy for debugging, uncomment:
+# VITE_PORT=3737
+# VITE_API_URL=http://localhost:8181
+
migration/complete_setup.sql (1)

69-75: Optional: keep categories stable across UIs.

The seeding places OPENAI_BASE_URL in 'rag_strategy'. Note the UI’s generic API Keys section currently saves edits under the 'api_keys' category for any key it touches (see APIKeysSection.tsx comment). While the table enforces uniqueness by key (so category can change on update without duplication), consider preserving the original category for consistency and simpler querying.

Also applies to: 89-99

archon-ui-main/src/services/credentialsService.ts (1)

288-307: Minor: avoid PUT-ing unchanged settings; preserve server/category semantics.

updateRagSettings blindly issues a PUT for every key. It works but generates unnecessary writes and log noise. Consider diffing against current values to only update changed keys. Also, ensure server treats category as idempotent for these updates.

I can provide a small helper to fetch current values, diff, and batch only the changes if you’d like.

archon-ui-main/src/components/settings/APIKeysSection.tsx (2)

47-56: Broader filter will include BASE_URL keys (good), but handle category on save.

Expanding the filter to include BASE_URL makes OPENAI_BASE_URL show up here. However, the save path always writes with category 'api_keys' (see below), which can inadvertently move settings seeded under 'rag_strategy'.

Apply the following changes to preserve the original category:

 interface CustomCredential {
   key: string;
   value: string;
   description: string;
   originalValue?: string;
   originalKey?: string; // Track original key for renaming
+  originalCategory?: string; // Preserve category for correct writes
   hasChanges?: boolean;
   is_encrypted?: boolean;
   showValue?: boolean; // Track per-credential visibility
   isNew?: boolean; // Track if this is a new unsaved credential
 }

In loadCredentials mapping:

 const uiCredentials = apiKeys.map(cred => ({
   key: cred.key,
   value: cred.value || '',
   description: cred.description || '',
   originalValue: cred.value || '',
   originalKey: cred.key, // Track original key for updates
+  originalCategory: cred.category,
   hasChanges: false,
   is_encrypted: cred.is_encrypted || false,
   showValue: false,
   isNew: false
 }));

In save paths (both create and update), use the preserved category:

-  category: 'api_keys'
+  category: cred.originalCategory || 'api_keys'

149-176: Guard against data loss when toggling encryption off without a new value.

If a credential was encrypted and the user toggles to unencrypted but leaves value empty, the update may clear the stored secret (since plaintext isn’t provided). Consider blocking save for that row until a new plaintext is entered, or require re-entry when flipping encryption.

I can wire a minimal UI validation that tracks original encryption and enforces new plaintext on de-encryption if you want a quick patch.

python/src/server/api_routes/tests_api.py (1)

101-102: Remove unused global active_websockets.

WebSocket connections are fully managed by TestWebSocketManager.connections. The standalone active_websockets dict is never used and can be deleted.

Apply this diff:

-test_executions: dict[str, TestExecution] = {}
-active_websockets: dict[str, list[WebSocket]] = {}
+test_executions: dict[str, TestExecution] = {}
archon-ui-main/src/components/settings/RAGSettings.tsx (2)

76-86: Provider-agnostic naming vs. "Ollama Base URL" label

The "Ollama Base URL" input persists into ragSettings.OPENAI_BASE_URL. This can confuse future readers since the key looks OpenAI-specific but the label and default are Ollama.

Option A (minimal): Update label/help to clarify it's the provider base URL stored in OPENAI_BASE_URL for now.
Option B (cleaner): Rename the prop to BASE_URL across UI/services/docs and have backend map it per provider. If you want, I can prep a follow-up PR plan that minimizes churn.

Also applies to: 79-82


89-111: Surface error reason in toast for easier debugging

Currently the error toast is generic. Include error details (safe message) to reduce support loops.

- showToast('Failed to save settings', 'error');
+ const message = (err as any)?.message ?? 'Unknown error';
+ showToast(`Failed to save settings: ${message}`, 'error');
docs/docs/api-reference.mdx (1)

1856-1869: Minor: import placement and API surface in SDK examples

  • Python: fine, but consider timeouts/retries in real clients.
  • JS: dynamic import inside method is atypical; import at module scope, and prefer socket.disconnect() over close().

Update the JS example import placement to the top and replace any close() with disconnect().

Also applies to: 1988-2021

python/src/server/api_routes/knowledge_api.py (4)

179-216: DELETE handler: message vs. behavior mismatch (item vs. source)

Handler delegates to SourceManagementService.delete_source, then returns “Successfully deleted knowledge item”. Consider aligning the message (“source” vs. “knowledge item”) to reduce confusion when both endpoints exist.

- return {"success": True, "message": f"Successfully deleted knowledge item {source_id}"}
+ return {"success": True, "message": f"Successfully deleted source {source_id}"}

If this path is specifically for items (not the broader source), flip the service call instead to the item-level delete API for consistency.


228-259: Code examples lookup: OK — add pagination if this grows

Directly returning full contents can be heavy. If this endpoint sees large payloads, consider limit/offset in the query and a fields selector.

No action required for this PR; just a note for scalability.


28-35: Duplicate imports: remove redundant lines

There are repeated imports for get_crawler, RAGService, DocumentStorageService, get_supabase_client.

-from ..services.crawler_manager import get_crawler
-from ..services.search.rag_service import RAGService
-from ..services.storage import DocumentStorageService
-from ..utils import get_supabase_client
+# (duplicates removed; imports already present above)

867-877: Prefer timezone-aware timestamps in health response

Use UTC timezone-aware timestamps to avoid ambiguity.

-from datetime import datetime
+from datetime import datetime, timezone
@@
-        "timestamp": datetime.now().isoformat(),
+        "timestamp": datetime.now(timezone.utc).isoformat(),
original_archon/iterations/v5-parallel-specialized-agents/streamlit_pages/environment.py (1)

188-195: Local var named OPENAI_BASE_URL but persisted as BASE_URL

This is just a local name, but it can mislead since the value is provider-agnostic.

Consider renaming the local variable to baseUrl or providerBaseUrl to reflect its usage:

- OPENAI_BASE_URL = st.text_input(
+ base_url = st.text_input(
@@
- updated_values["BASE_URL"] = OPENAI_BASE_URL
+ updated_values["BASE_URL"] = base_url
original_archon/iterations/v6-tool-library-integration/streamlit_pages/environment.py (1)

188-195: Same naming nit: provider-agnostic value stored under BASE_URL

Mirror suggestion from v5 file: rename local OPENAI_BASE_URL to base_url for clarity.

See diff in the v5 comment; apply the same change here.

python/src/server/services/source_management_service.py (5)

19-34: Default model mismatch vs docs; please align MODEL_CHOICE default

Code defaults to "gpt-4.1-nano", while docs (docs/docs/rag.mdx) show "gpt-4o-mini" as the default. Pick one source of truth to avoid operator confusion and flaky expectations. Given tests already reference "gpt-4.1-nano", the quickest fix is to update the docs’ table default. See my docs suggestion on that file.


36-118: Duplicate provider/base_url resolution logic; centralize and de-risk Google base URL

This helper largely reimplements logic already present in credential_service/llm_provider_service (e.g., provider/base_url mapping and Google’s OpenAI-compatible endpoint). Risks:

  • Divergence over time across places that decide base_url.
  • Inconsistent Google base URL path if one site changes; tests expect ".../v1beta/openai/" while docs currently suggest otherwise.

Consider delegating to a small synchronous accessor in credential_service (read from cache only, no I/O) for provider/base_url, so both this module and llm_provider_service share the same single source of truth. At minimum, reuse credential_service._get_provider_base_url to avoid hardcoding URLs here.

Apply at least this logging improvement when falling back to OpenAI on unknown providers:

@@
-        else:
-            # Unknown provider, default to OpenAI without base_url
+        else:
+            # Unknown provider, default to OpenAI without base_url
+            logger.warning(f"Unknown LLM provider '{provider}', defaulting to OpenAI")
             if credential_service._cache_initialized and "OPENAI_API_KEY" in credential_service._cache:

162-176: LLM client creation: log with traceback; avoid leaking secrets in messages

Use exc_info=True so we don’t lose the stack on failures.

-            search_logger.error(f"Failed to create LLM client for summary generation: {e}")
+            search_logger.error("Failed to create LLM client for summary generation", exc_info=True)

And similarly in title generation:

-                search_logger.error(
-                    f"Failed to create LLM client for title generation: {e}"
-                )
+                search_logger.error("Failed to create LLM client for title generation", exc_info=True)

Also applies to: 178-179


212-215: Keep full stack trace on failure path

-        search_logger.error(
-            f"Error generating summary with LLM for {source_id}: {e}. Using default summary."
-        )
+        search_logger.error(
+            f"Error generating summary with LLM for {source_id}. Using default summary.",
+            exc_info=True,
+        )

293-295: Add traceback to title-generation error for postmortem debugging

-            search_logger.error(f"Error generating title for {source_id}: {e}")
+            search_logger.error(f"Error generating title for {source_id}", exc_info=True)
python/src/server/services/crawling/strategies/recursive.py (1)

234-244: Avoid duplicate binary checks and minor micro-optimization

is_binary_file is called twice. Cache it once per link.

-                            if within_scope(next_url):
-                                # Skip binary files and already visited URLs
-                                if next_url not in visited and not self.url_handler.is_binary_file(next_url):
-                                    next_level_urls.add(next_url)
-                                elif self.url_handler.is_binary_file(next_url):
-                                    logger.debug(f"Skipping binary file from crawl queue: {next_url}")
+                            if within_scope(next_url):
+                                is_bin = self.url_handler.is_binary_file(next_url)
+                                # Skip binary files and already visited URLs
+                                if next_url not in visited and not is_bin:
+                                    next_level_urls.add(next_url)
+                                elif is_bin:
+                                    logger.debug(f"Skipping binary file from crawl queue: {next_url}")
python/src/server/services/crawling/crawling_service.py (1)

570-577: Pass restrict_to_start_path explicitly to document intent

Given the strategy now supports path scoping, explicitly opt-in from the orchestrator to avoid surprises if defaults change again.

             crawl_results = await self.crawl_recursive_with_progress(
                 [url],
                 max_depth=max_depth,
                 max_concurrent=None,  # Let strategy use settings
                 progress_callback=await self._create_crawl_progress_callback("crawling"),
                 start_progress=10,
-                end_progress=20,
+                end_progress=20,
+                restrict_to_start_path=True,
             )
python/tests/test_async_llm_provider_service.py (1)

160-183: Optional: add a targeted test for Google’s compatibility base_url

Given the fragile nature of the Google OpenAI-compatible path, consider a small test mirroring the new OpenAI base_url test to lock “.../v1beta/openai/” and prevent regressions.

python/src/server/services/llm_provider_service.py (2)

95-103: OpenAI base_url fallback from credentials is correct; consider adding env fallback here too for symmetry.

In the explicit provider override path, we only consult credentials. In practice, deployments sometimes rely on environment-only overrides (e.g., OPENAI_BASE_URL in Docker). Mirroring the env fallback you added in get_active_provider makes behavior consistent across both paths.

Apply this diff and import:

@@
-            if provider == "openai" and not base_url:
+            if provider == "openai" and not base_url:
                 try:
                     base_url = await credential_service.get_credential("OPENAI_BASE_URL", decrypt=True)
                     if not base_url:
                         base_url = await credential_service.get_credential("OPENAI_API_BASE", decrypt=True)
                 except Exception:
                     base_url = None
+                if not base_url:
+                    # Mirror credential_service.get_active_provider fallbacks
+                    import os
+                    base_url = os.getenv("OPENAI_BASE_URL") or os.getenv("OPENAI_API_BASE")

Optionally, normalize to an OpenAI-compatible path if absent (e.g., append "/v1" when missing). I can provide a small helper if you want that.


127-131: Conditional base_url kwarg avoids breaking tests; add richer error logging below.

This change is good. Also, consider emitting the full stack trace in the except block for easier debugging.

Apply within the same function’s except:

-        logger.error(
-            f"Error creating LLM client for provider {provider_name if 'provider_name' in locals() else 'unknown'}: {e}"
-        )
+        logger.error(
+            f"Error creating LLM client for provider {provider_name if 'provider_name' in locals() else 'unknown'}: {e}",
+            exc_info=True,
+        )
python/src/server/services/crawling/document_storage_operations.py (1)

90-93: Using a provided, scoped source_id consistently per crawl is the right call.

The logging line is useful during bring-up. If logs get noisy in large crawls, consider downgrading to debug.

python/src/server/services/credential_service.py (3)

242-263: Great: non-fatal provider cache invalidation on set; deduplicate key lists to avoid drift.

The duplicated provider_related_keys set appears again in delete_credential. Centralize it to a module-level constant to keep both sites in sync.

Apply this refactor:

@@
-from .llm_provider_service import invalidate_provider_cache
+from .llm_provider_service import invalidate_provider_cache
@@
-                provider_related_keys = {
-                    "LLM_PROVIDER",
-                    "MODEL_CHOICE",
-                    "EMBEDDING_MODEL",
-                    "OPENAI_API_KEY",
-                    "OPENAI_BASE_URL",
-                    "OPENAI_API_BASE",
-                    "GOOGLE_API_KEY",
-                    "OLLAMA_BASE_URL",
-                }
-                if category == "rag_strategy" or key in provider_related_keys:
+                # Keep provider-related keys in one place to avoid drift
+                from typing import Final
+                PROVIDER_RELATED_KEYS: Final[set[str]] = {
+                    "LLM_PROVIDER",
+                    "MODEL_CHOICE",
+                    "EMBEDDING_MODEL",
+                    "OPENAI_API_KEY",
+                    "OPENAI_BASE_URL",
+                    "OPENAI_API_BASE",
+                    "GOOGLE_API_KEY",
+                    "OLLAMA_BASE_URL",
+                }
+                if category == "rag_strategy" or key in PROVIDER_RELATED_KEYS:
                     invalidate_provider_cache(keys_changed=[key])

You can then reuse PROVIDER_RELATED_KEYS in delete_credential below.


292-310: Same note: reuse the centralized PROVIDER_RELATED_KEYS set here.

Avoid maintaining two copies of the same set.

Apply this diff:

@@
-                provider_related_keys = {
-                    "LLM_PROVIDER",
-                    "MODEL_CHOICE",
-                    "EMBEDDING_MODEL",
-                    "OPENAI_API_KEY",
-                    "OPENAI_BASE_URL",
-                    "OPENAI_API_BASE",
-                    "GOOGLE_API_KEY",
-                    "OLLAMA_BASE_URL",
-                }
-                if key in provider_related_keys:
+                from typing import Final
+                PROVIDER_RELATED_KEYS: Final[set[str]] = {
+                    "LLM_PROVIDER",
+                    "MODEL_CHOICE",
+                    "EMBEDDING_MODEL",
+                    "OPENAI_API_KEY",
+                    "OPENAI_BASE_URL",
+                    "OPENAI_API_BASE",
+                    "GOOGLE_API_KEY",
+                    "OLLAMA_BASE_URL",
+                }
+                if key in PROVIDER_RELATED_KEYS:
                     invalidate_provider_cache(keys_changed=[key])

459-473: OpenAI base_url resolution path is solid; consider normalizing the URL.

Some gateways expect a trailing "/v1". Normalizing avoids subtle 404s when callers omit it.

Example normalization (before returning the config):

@@
                 if not base_url:
                     base_url = os.getenv("OPENAI_BASE_URL") or os.getenv("OPENAI_API_BASE")
+            # Optional: normalize base_url to include /v1 when talking to OpenAI-compatible endpoints
+            if provider == "openai" and base_url and not re.search(r'/v1/?$', base_url):
+                base_url = base_url.rstrip('/') + '/v1'

Note: you’d need import re at top. If you don’t want implicit changes, logging a warning when /v1 is missing is a lighter alternative.

python/src/server/services/crawling/code_extraction_service.py (1)

316-321: Fallback source_id derivation is fine; consider light normalization.

URLs with trailing slashes or default ports can lead to subtly different source_ids across runs. Optional normalization (lowercasing host, stripping trailing slashes) improves deduplication.

Example:

-                    if not source_id:
-                        parsed_url = urlparse(source_url)
-                        source_id = parsed_url.netloc or parsed_url.path
+                    if not source_id:
+                        parsed_url = urlparse(source_url)
+                        host_or_path = parsed_url.netloc or parsed_url.path
+                        source_id = host_or_path.rstrip('/').lower()

Comment on lines +119 to 122
| Parameter | Type | Required | Description |
| ----------- | ------ | -------- | ----------------- |
| `source_id` | string | ✅ | Source identifier |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Route param should accept slashes: document {source_id:path}

Backend now uses {source_id:path} for knowledge-item routes to allow slashes in IDs. The tables/examples should show that and mention URL-encoding in curl examples.

  • Update path placeholders in the docs (PUT/DELETE knowledge-items, GET code-examples, POST refresh) to {source_id:path}.
  • Add a short note: “source_id may contain slashes; URL-encode when using curl.”

Also applies to: 153-156

🧰 Tools
🪛 LanguageTool

[grammar] ~119-~119: There might be a mistake here.
Context: ... Type | Required | Description | | ----------- | ------ | -------- | ----...

(QB_NEW_EN)


[grammar] ~120-~120: There might be a mistake here.
Context: ... ------ | -------- | ----------------- | | source_id | string | ✅ | Sourc...

(QB_NEW_EN)

🤖 Prompt for AI Agents
In docs/docs/api-reference.mdx around lines 119-122 (and similarly 153-156),
update the path parameter placeholder from `{source_id}` to `{source_id:path}`
in the PUT/DELETE knowledge-items, GET code examples, and POST refresh examples
so docs match backend routing, and add a short note near the parameter table or
examples: "source_id may contain slashes; URL-encode when using curl." Ensure
all examples and tables reflect the `{source_id:path}` placeholder.

Comment on lines +1590 to 1615
import { io } from "socket.io-client";

// Connect to chat namespace
const socket = io('/chat');
const socket = io("/chat");

// Join session
const sessionId = 'session-uuid';
socket.emit('join_session', { session_id: sessionId });
const sessionId = "session-uuid";
socket.emit("join_session", { session_id: sessionId });

ws.onmessage = function(event) {
ws.onmessage = function (event) {
const data = JSON.parse(event.data);
if (data.type === 'message') {
console.log('Agent:', data.content);
} else if (data.type === 'thinking') {
console.log('Agent is thinking...');
if (data.type === "message") {
console.log("Agent:", data.content);
} else if (data.type === "thinking") {
console.log("Agent is thinking...");
}
};

// Send message
ws.send(JSON.stringify({
type: 'message',
content: 'Help me with authentication'
}));
ws.send(
JSON.stringify({
type: "message",
content: "Help me with authentication",
})
);
```

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Chat example mixes Socket.IO and raw WebSocket APIs

The snippet uses io("/chat") but then switches to ws.onmessage/ws.send, which are WebSocket APIs. Keep it consistently Socket.IO.

Replace with:

-// Connect to chat namespace
-const socket = io("/chat");
-
-// Join session
-const sessionId = "session-uuid";
-socket.emit("join_session", { session_id: sessionId });
-
-ws.onmessage = function (event) {
-  const data = JSON.parse(event.data);
-  if (data.type === "message") {
-    console.log("Agent:", data.content);
-  } else if (data.type === "thinking") {
-    console.log("Agent is thinking...");
-  }
-};
-
-// Send message
-ws.send(
-  JSON.stringify({
-    type: "message",
-    content: "Help me with authentication",
-  })
-);
+// Connect to chat namespace
+const socket = io("/chat");
+
+// Join session
+const sessionId = "session-uuid";
+socket.emit("join_session", { session_id: sessionId });
+
+// Receive assistant messages
+socket.on("message", (payload) => {
+  console.log("Agent:", payload.content);
+});
+
+// Thinking updates (if emitted)
+socket.on("thinking", () => {
+  console.log("Agent is thinking...");
+});
+
+// Send a message
+socket.emit("message", { content: "Help me with authentication" });
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
import { io } from "socket.io-client";
// Connect to chat namespace
const socket = io('/chat');
const socket = io("/chat");
// Join session
const sessionId = 'session-uuid';
socket.emit('join_session', { session_id: sessionId });
const sessionId = "session-uuid";
socket.emit("join_session", { session_id: sessionId });
ws.onmessage = function(event) {
ws.onmessage = function (event) {
const data = JSON.parse(event.data);
if (data.type === 'message') {
console.log('Agent:', data.content);
} else if (data.type === 'thinking') {
console.log('Agent is thinking...');
if (data.type === "message") {
console.log("Agent:", data.content);
} else if (data.type === "thinking") {
console.log("Agent is thinking...");
}
};
// Send message
ws.send(JSON.stringify({
type: 'message',
content: 'Help me with authentication'
}));
ws.send(
JSON.stringify({
type: "message",
content: "Help me with authentication",
})
);
```
import { io } from "socket.io-client";
// Connect to chat namespace
const socket = io("/chat");
// Join session
const sessionId = "session-uuid";
socket.emit("join_session", { session_id: sessionId });
// Receive assistant messages
socket.on("message", (payload) => {
console.log("Agent:", payload.content);
});
// Thinking updates (if emitted)
socket.on("thinking", () => {
console.log("Agent is thinking...");
});
// Send a message
socket.emit("message", { content: "Help me with authentication" });
🤖 Prompt for AI Agents
In docs/docs/api-reference.mdx around lines 1590 to 1615, the example mixes
Socket.IO and raw WebSocket APIs (uses io(...) but then ws.onmessage/ws.send);
replace the raw WebSocket usage with Socket.IO calls: use socket.on(...) to
handle incoming events (handle the message and thinking events via socket.on)
and use socket.emit(...) or socket.send(...) on the same socket instance to send
the message payload, keeping the join_session emit as-is so the example
consistently uses the Socket.IO client API.

Comment thread docs/docs/rag.mdx
Comment on lines +36 to +38
| **MODEL_CHOICE** | Chat model for query enhancement | `gpt-4o-mini` | Response quality |
| **EMBEDDING_MODEL** | Model for vector embeddings | `text-embedding-3-small` | Search accuracy |
| **LLM_PROVIDER** | Provider (openai/google/ollama/) | `openai` | Model availability |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Default model in table should match backend default

The backend defaults to MODEL_CHOICE="gpt-4.1-nano" (see source_management_service._get_model_choice). Update docs to prevent misconfiguration.

-| **MODEL_CHOICE**    | Chat model for query enhancement | `gpt-4o-mini`            | Response quality   |
+| **MODEL_CHOICE**    | Chat model for query enhancement | `gpt-4.1-nano`           | Response quality   |
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
| **MODEL_CHOICE** | Chat model for query enhancement | `gpt-4o-mini` | Response quality |
| **EMBEDDING_MODEL** | Model for vector embeddings | `text-embedding-3-small` | Search accuracy |
| **LLM_PROVIDER** | Provider (openai/google/ollama/) | `openai` | Model availability |
| **MODEL_CHOICE** | Chat model for query enhancement | `gpt-4.1-nano` | Response quality |
| **EMBEDDING_MODEL** | Model for vector embeddings | `text-embedding-3-small` | Search accuracy |
| **LLM_PROVIDER** | Provider (openai/google/ollama/) | `openai` | Model availability |
🧰 Tools
🪛 LanguageTool

[grammar] ~36-~36: There might be a mistake here.
Context: ...-mini` | Response quality | | EMBEDDING_MODEL | Model for vector...

(QB_NEW_EN)


[grammar] ~37-~37: There might be a mistake here.
Context: ...mbedding-3-small` | Search accuracy | | LLM_PROVIDER | Provider (openai...

(QB_NEW_EN)

🤖 Prompt for AI Agents
In docs/docs/rag.mdx around lines 36 to 38 the default MODEL_CHOICE value shown
(`gpt-4o-mini`) does not match the backend default (`gpt-4.1-nano`); update the
table entry for MODEL_CHOICE to use `gpt-4.1-nano` so docs and backend defaults
are consistent and avoid misconfiguration.

Comment on lines +43 to 44
restrict_to_start_path: bool = True,
) -> List[Dict[str, Any]]:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Verification agent

🧩 Analysis chain

Behavioral change: default now restricts crawl to start path; likely breaking

Setting restrict_to_start_path: bool = True flips previous behavior from domain-wide to path-scoped by default. CrawlingService does not pass this arg, so everything using recursive crawling will suddenly scope to path. If that’s intended system-wide, great—document it. If not, keep the default False and have callers opt in explicitly.

Recommended change:

-        end_progress: int = 60,
-        restrict_to_start_path: bool = True,
+        end_progress: int = 60,
+        restrict_to_start_path: bool = False,

🏁 Script executed:

#!/bin/bash
# Locate the parameter definition in the recursive crawling strategy
grep -n "restrict_to_start_path" python/src/server/services/crawling/strategies/recursive.py

Length of output: 226


🏁 Script executed:

#!/bin/bash
# Display the recursive crawling strategy implementation for context
sed -n '1,200p' python/src/server/services/crawling/strategies/recursive.py

Length of output: 9242


#!/bin/bash
# Find all calls to crawl_recursive_with_progress in the codebase
grep -R "crawl_recursive_with_progress" -n .

🏁 Script executed:

#!/bin/bash
# Find all invocations of crawl_recursive_with_progress in the repository
grep -R "crawl_recursive_with_progress" -n .

Length of output: 706


🏁 Script executed:

#!/bin/bash
# Show the definition of crawl_recursive_with_progress in CrawlingService
sed -n '200,260p' python/src/server/services/crawling/crawling_service.py

Length of output: 2099


🏁 Script executed:

#!/bin/bash
# Inspect orchestration usage of crawl_recursive_with_progress
sed -n '540,620p' python/src/server/services/crawling/crawling_service.py

Length of output: 1950


Action Required: Restore restrict_to_start_path=False to Preserve Previous Domain-Wide Crawling

The recent change sets the default of restrict_to_start_path to True, which means every call to crawl_recursive_with_progress (including those from CrawlingService) will now be limited to the start path rather than the full domain. Because CrawlingService.crawl_recursive_with_progress does not pass this parameter, all existing recursive crawls will silently switch to path-scoped behavior—likely a breaking change. If the intent is to make path-scoping the new global default, please update the public docs; otherwise, revert the default and let callers opt in explicitly.

• File: python/src/server/services/crawling/strategies/recursive.py
– Line 43: change default back to False
• No other callers override this flag, so restoring the default prevents unintended behavior shifts

Recommended diff:

@@ async def crawl_recursive_with_progress(
-        restrict_to_start_path: bool = True,
+        restrict_to_start_path: bool = False,
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
restrict_to_start_path: bool = True,
) -> List[Dict[str, Any]]:
@@ async def crawl_recursive_with_progress(
restrict_to_start_path: bool = False,
) -> List[Dict[str, Any]]:
🤖 Prompt for AI Agents
In python/src/server/services/crawling/strategies/recursive.py around line 43,
the function signature changed the parameter restrict_to_start_path default to
True which alters global behavior; change the default back to False
(restrict_to_start_path: bool = False) so existing callers retain domain-wide
crawling, and if path-scoping should be the new global default instead, update
the public docs and explicitly opt callers in rather than changing the default
silently.

Comment on lines +142 to +157
# Build scope constraints from the provided start URLs so we only crawl within
# the same domain AND within the starting path (not the whole domain).
# Example: start https://github.com/owner/repo -> only follow links under /owner/repo
scopes_by_netloc = {}
if restrict_to_start_path:
try:
for su in start_urls:
ps = urlparse(su)
netloc = ps.netloc
path = ps.path or "/"
if netloc:
scopes_by_netloc.setdefault(netloc, set()).add(path)
except Exception as e:
logger.warning(f"Failed building crawl scopes from start URLs: {e}")
scopes_by_netloc = {}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

⚠️ Potential issue

Overly strict path scoping when start URL points to a file (e.g., /docs/index.html)

Current logic uses the start URL’s full path as the allowed base. If the path is a file, almost all internal links will be marked out-of-scope. Scope to the directory instead when the last segment looks like a file.

-                for su in start_urls:
-                    ps = urlparse(su)
-                    netloc = ps.netloc
-                    path = ps.path or "/"
-                    if netloc:
-                        scopes_by_netloc.setdefault(netloc, set()).add(path)
+                for su in start_urls:
+                    ps = urlparse(su)
+                    netloc = ps.netloc
+                    raw_path = ps.path or "/"
+                    # If last segment has a dot and no trailing slash, treat as file and scope to its directory
+                    if raw_path and not raw_path.endswith("/") and "." in raw_path.rsplit("/", 1)[-1]:
+                        path = "/" if "/" not in raw_path else raw_path.rsplit("/", 1)[0]
+                    else:
+                        path = raw_path
+                    if netloc:
+                        scopes_by_netloc.setdefault(netloc, set()).add(path)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Build scope constraints from the provided start URLs so we only crawl within
# the same domain AND within the starting path (not the whole domain).
# Example: start https://github.com/owner/repo -> only follow links under /owner/repo
scopes_by_netloc = {}
if restrict_to_start_path:
try:
for su in start_urls:
ps = urlparse(su)
netloc = ps.netloc
path = ps.path or "/"
if netloc:
scopes_by_netloc.setdefault(netloc, set()).add(path)
except Exception as e:
logger.warning(f"Failed building crawl scopes from start URLs: {e}")
scopes_by_netloc = {}
# Build scope constraints from the provided start URLs so we only crawl within
# the same domain AND within the starting path (not the whole domain).
# Example: start https://github.com/owner/repo -> only follow links under /owner/repo
scopes_by_netloc = {}
if restrict_to_start_path:
try:
for su in start_urls:
ps = urlparse(su)
netloc = ps.netloc
raw_path = ps.path or "/"
# If last segment has a dot and no trailing slash, treat as file and scope to its directory
if raw_path and not raw_path.endswith("/") and "." in raw_path.rsplit("/", 1)[-1]:
path = "/" if "/" not in raw_path else raw_path.rsplit("/", 1)[0]
else:
path = raw_path
if netloc:
scopes_by_netloc.setdefault(netloc, set()).add(path)
except Exception as e:
logger.warning(f"Failed building crawl scopes from start URLs: {e}")
scopes_by_netloc = {}
🤖 Prompt for AI Agents
In python/src/server/services/crawling/strategies/recursive.py around lines
142-157, the scope-building logic currently uses the full start URL path which
makes scopes too strict when the start URL points to a file (e.g.,
/docs/index.html). Modify the loop that computes `path` so that if the last path
segment looks like a file (contains a dot and does not end with '/'), replace
`path` with its parent directory instead; ensure the resulting path is non-empty
and begins with '/' (use '/' when parent is empty) and canonicalize to include a
trailing '/' if your scope-matching expects directory prefixes. Keep the rest of
the try/except and logging the same.

Comment on lines +514 to 517
return rag_settings.get("OPENAI_BASE_URL", "http://localhost:11434/v1")
elif provider == "google":
return "https://generativelanguage.googleapis.com/v1beta/openai/"
return None # Use default for OpenAI

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Bug: Ollama base URL reads OPENAI_BASE_URL; should prefer OLLAMA_BASE_URL (with sensible fallbacks).

Using the OpenAI key for Ollama will misroute requests when both providers are configured. You already treat OLLAMA_BASE_URL as a provider-related key elsewhere; make resolution consistent here.

Apply this fix:

 def _get_provider_base_url(self, provider: str, rag_settings: dict) -> str | None:
     """Get base URL for provider."""
     if provider == "ollama":
-        return rag_settings.get("OPENAI_BASE_URL", "http://localhost:11434/v1")
+        # Prefer explicit Ollama base, then allow compatibility key for transitional setups
+        return (
+            rag_settings.get("OLLAMA_BASE_URL")
+            or os.getenv("OLLAMA_BASE_URL")
+            or rag_settings.get("OPENAI_BASE_URL")  # compatibility fallback
+            or "http://localhost:11434/v1"
+        )
     elif provider == "google":
         return "https://generativelanguage.googleapis.com/v1beta/openai/"
     return None  # Use default for OpenAI
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
return rag_settings.get("OPENAI_BASE_URL", "http://localhost:11434/v1")
elif provider == "google":
return "https://generativelanguage.googleapis.com/v1beta/openai/"
return None # Use default for OpenAI
def _get_provider_base_url(self, provider: str, rag_settings: dict) -> str | None:
"""Get base URL for provider."""
if provider == "ollama":
# Prefer explicit Ollama base, then allow compatibility key for transitional setups
return (
rag_settings.get("OLLAMA_BASE_URL")
or os.getenv("OLLAMA_BASE_URL")
or rag_settings.get("OPENAI_BASE_URL") # compatibility fallback
or "http://localhost:11434/v1"
)
elif provider == "google":
return "https://generativelanguage.googleapis.com/v1beta/openai/"
return None # Use default for OpenAI
🤖 Prompt for AI Agents
In python/src/server/services/credential_service.py around lines 514 to 517, the
provider-specific base URL resolution incorrectly uses OPENAI_BASE_URL for the
Ollama provider; update the branch so that when provider == "ollama" it first
reads rag_settings.get("OLLAMA_BASE_URL"), then falls back to
rag_settings.get("OPENAI_BASE_URL") if present, and finally to the local default
"http://localhost:11434/v1"; keep the existing google branch and return None
otherwise so OpenAI uses the default.

Comment on lines +569 to 572
"OPENAI_BASE_URL", # Ollama base URL
"EMBEDDING_MODEL", # Custom embedding model
"MODEL_CHOICE", # Chat model for sync contexts
]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Comment/key mismatch: this lists OPENAI_BASE_URL as “Ollama base URL”. Include OLLAMA_BASE_URL explicitly.

To avoid confusion and support per-provider bases, include both. Keep OPENAI_BASE_URL for OpenAI-compatible gateways (OpenRouter, Azure), and OLLAMA_BASE_URL for Ollama.

Apply:

-        "OPENAI_BASE_URL",  # Ollama base URL
+        # Base URLs for OpenAI-compatible providers
+        "OPENAI_BASE_URL",   # e.g., OpenAI, Azure OpenAI, OpenRouter
+        "OLLAMA_BASE_URL",   # Ollama base URL
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"OPENAI_BASE_URL", # Ollama base URL
"EMBEDDING_MODEL", # Custom embedding model
"MODEL_CHOICE", # Chat model for sync contexts
]
# Base URLs for OpenAI-compatible providers
"OPENAI_BASE_URL", # e.g., OpenAI, Azure OpenAI, OpenRouter
"OLLAMA_BASE_URL", # Ollama base URL
"EMBEDDING_MODEL", # Custom embedding model
"MODEL_CHOICE", # Chat model for sync contexts
]

@Wirasm

Wirasm commented Sep 4, 2025

Copy link
Copy Markdown
Collaborator

Thank you for this contribution @armand0e, this is partially solved and this pr has a lot of conflicts, feel free to adapt your solution and resolve the conflicts and reopen it.

Really appreciate the contribution!

@Wirasm Wirasm closed this Sep 4, 2025
@armand0e

armand0e commented Sep 4, 2025

Copy link
Copy Markdown
Author

understood. I threw this together with very little actual effort, just got it half-working on my machine and called it a day lol. Will revisit in the future when I have more time and build it out proper.

coleam00 pushed a commit that referenced this pull request Apr 7, 2026
…378) (#409)

* Investigate issues #252 and #378: test infrastructure failures

Root cause for #378: github-context.test.ts globally mocks fs/promises
with readFile returning empty string, which leaks to version.test.ts
causing JSON.parse('') to fail.

Issue #252 (test hang) no longer reproduces but preventive cleanup
should be added to executor.test.ts.

* Fix test infrastructure: mock.module leak and executor cleanup (#252, #378)

Remove global mock.module('fs/promises') from github-context.test.ts that was
leaking a readFile mock across all test files, causing 3 version.test.ts failures
in CI. The mock was redundant since ensureRepoReady and autoDetectAndLoadCommands
are already mocked at the adapter level.

Add afterAll mock.restore() to executor.test.ts to prevent module mocks and
pending timers from leaking to other test files.

Changes:
- Remove mock.module('fs/promises') from github-context.test.ts
- Add afterAll cleanup to executor.test.ts

Fixes #252, Fixes #378
Tyone88 pushed a commit to Tyone88/Archon that referenced this pull request Apr 16, 2026
…m00#252, coleam00#378) (coleam00#409)

* Investigate issues coleam00#252 and coleam00#378: test infrastructure failures

Root cause for coleam00#378: github-context.test.ts globally mocks fs/promises
with readFile returning empty string, which leaks to version.test.ts
causing JSON.parse('') to fail.

Issue coleam00#252 (test hang) no longer reproduces but preventive cleanup
should be added to executor.test.ts.

* Fix test infrastructure: mock.module leak and executor cleanup (coleam00#252, coleam00#378)

Remove global mock.module('fs/promises') from github-context.test.ts that was
leaking a readFile mock across all test files, causing 3 version.test.ts failures
in CI. The mock was redundant since ensureRepoReady and autoDetectAndLoadCommands
are already mocked at the adapter level.

Add afterAll mock.restore() to executor.test.ts to prevent module mocks and
pending timers from leaking to other test files.

Changes:
- Remove mock.module('fs/promises') from github-context.test.ts
- Add afterAll cleanup to executor.test.ts

Fixes coleam00#252, Fixes coleam00#378
joaobmonteiro pushed a commit to joaobmonteiro/Archon that referenced this pull request Apr 26, 2026
…m00#252, coleam00#378) (coleam00#409)

* Investigate issues coleam00#252 and coleam00#378: test infrastructure failures

Root cause for coleam00#378: github-context.test.ts globally mocks fs/promises
with readFile returning empty string, which leaks to version.test.ts
causing JSON.parse('') to fail.

Issue coleam00#252 (test hang) no longer reproduces but preventive cleanup
should be added to executor.test.ts.

* Fix test infrastructure: mock.module leak and executor cleanup (coleam00#252, coleam00#378)

Remove global mock.module('fs/promises') from github-context.test.ts that was
leaking a readFile mock across all test files, causing 3 version.test.ts failures
in CI. The mock was redundant since ensureRepoReady and autoDetectAndLoadCommands
are already mocked at the adapter level.

Add afterAll mock.restore() to executor.test.ts to prevent module mocks and
pending timers from leaking to other test files.

Changes:
- Remove mock.module('fs/promises') from github-context.test.ts
- Add afterAll cleanup to executor.test.ts

Fixes coleam00#252, Fixes coleam00#378
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants