migration/add_source_url_display_name.sql (2)

17-19: Index choices are fine; consider a case-insensitive index for display lookups.

The plain btree indexes will help equality filters. If the UI often performs case-insensitive searches (ILIKE) on source_display_name, add a functional index to avoid sequential scans.

Apply this additional DDL (outside the changed hunk) if case-insensitive lookups are common:

CREATE INDEX IF NOT EXISTS idx_archon_sources_display_name_lower
  ON archon_sources (lower(source_display_name));

---

35-36: 16-hex hash is probably sufficient; consider 20+ hex chars or per-URL uniqueness guard for extra safety.

64-bit IDs have extremely low collision probability, but widening to 80–96 bits (20–24 hex chars) further reduces risk with negligible storage cost. Alternatively, enforce uniqueness on source_url to protect against theoretical collisions and accidental canonicalization drift.

Would you like me to propose a safe widening plan (forward-compatible code + non-blocking online migration)?

migration/complete_setup.sql (1)

190-193: Tighten the column comment to reflect non-URL sources.

source_id is a 16-char SHA-256 prefix for URL sources, but can be file_* for file sources. Clarify to avoid future confusion.

Apply:

-COMMENT ON COLUMN archon_sources.source_id IS 'Unique hash identifier for the source (16-char SHA256 hash of URL)';
+COMMENT ON COLUMN archon_sources.source_id IS 'Identifier: 16-char SHA-256 prefix for URL sources, or file_* for file uploads';

python/src/server/services/crawling/helpers/url_handler.py (2)

244-247: Consistent brand capitalization: GitHub.

Minor UX polish: use “GitHub API,” not “Github API.”

-                    return "Github API"
+                    return "GitHub API"

---

100-106: Consider treating .svg as crawlable text.

SVG is XML; excluding it as binary skips potentially useful diagrams/code examples. If bandwidth isn’t a concern, allow SVG.

-                ".svg",

If you prefer keeping it excluded globally, consider a site-config override to include SVG only for documentation domains.

python/src/server/services/source_management_service.py (6)

18-33: Preserve stack traces for diagnostics.

Log warnings about model selection with full context.

-    except Exception as e:
-        logger.warning(f"Error getting model choice: {e}, using default")
+    except Exception as e:
+        logger.warning(f"Error getting model choice: {e}, using default", exc_info=True)

---

43-51: Docstring still mentions “domain”; update to new ID semantics.

Align docs with the 16-char SHA-256 prefix and file_* IDs to reduce confusion.

-        source_id: The source ID (domain)
+        source_id: The source ID (16-char SHA-256 prefix for URL sources, or file_* for file uploads)

---

101-104: Add exc_info to error logs around LLM calls.

This improves debuggability in production without changing behavior.

-            search_logger.error(f"Failed to create LLM client fallback: {e}")
+            search_logger.error(f"Failed to create LLM client fallback: {e}", exc_info=True)
@@
-            search_logger.error(f"Empty or invalid response from LLM for {source_id}")
+            search_logger.error(f"Empty or invalid response from LLM for {source_id}", exc_info=True)
@@
-        search_logger.error(
-            f"Error generating summary with LLM for {source_id}: {e}. Using default summary."
-        )
+        search_logger.error(
+            f"Error generating summary with LLM for {source_id}: {e}. Using default summary.",
+            exc_info=True,
+        )

Also applies to: 118-125, 135-139

---

148-149: Default title should prefer display name when provided.

Avoid surfacing the hash as a title if content is short and a display name exists.

-    # Default title is the source ID
-    title = source_id
+    # Default title prefers human-readable display name
+    title = source_display_name or source_id

Also applies to: 163-167

---

365-366: Bubble up full stack traces on failure.

Include exc_info to aid postmortems; re-raise is good.

-        search_logger.error(f"Error updating source {source_id}: {e}")
+        search_logger.error(f"Error updating source {source_id}: {e}", exc_info=True)

---

572-585: Consider retries with backoff for external LLM calls.

Guidelines recommend retries for dependency calls. A lightweight tenacity retry on client.chat.completions.create would improve resilience without complicating the flow. Happy to draft this if desired.

python/src/server/services/crawling/crawling_service.py (2)

307-313: Sanitize URLs in logs; IDs and display names look good.

Generate IDs/display names here (nice), but avoid logging raw query strings/tokens. Log a redacted URL (strip ?query and #fragment).

-            safe_logfire_info(
-                f"Generated unique source_id '{original_source_id}' and display name '{source_display_name}' from URL '{url}'"
-            )
+            redacted = url.split("?", 1)[0].split("#", 1)[0]
+            safe_logfire_info(
+                f"Generated unique source_id '{original_source_id}' and display name '{source_display_name}' from URL '{redacted}'"
+            )

---

446-456: Include display name in completion payload for immediate UI updates.

Frontends can show the friendly name without extra fetch.

             await complete_crawl_progress(
                 task_id,
                 {
                     "chunks_stored": storage_results["chunk_count"],
                     "code_examples_found": code_examples_count,
                     "processed_pages": len(crawl_results),
                     "total_pages": len(crawl_results),
                     "sourceId": storage_results.get("source_id", ""),
+                    "sourceDisplayName": source_display_name,
                     "log": "Crawl completed successfully!",
                 },
             )

python/tests/test_source_id_refactor.py (2)

8-11: Remove unused imports to satisfy Ruff and keep tests clean.

pytest, Mock, patch, and hashlib are not used.

-import pytest
-from unittest.mock import Mock, patch
-import hashlib
 from concurrent.futures import ThreadPoolExecutor
 import time

---

204-220: Edge-case handling looks solid; consider one more case to avoid false API positives.

Current URLHandler logic treats any path containing "/api" as an API. This can misclassify paths like "/apiary". Consider tightening the check (e.g., segment match r'(^|/)api(/|$)') and adding a test like "https://example.com/apiary" → "Example".

Would you like me to propose a patch for URLHandler to use a segment-aware check and add the corresponding test?

python/src/server/services/crawling/document_storage_operations.py (4)

67-68: Reuse the initialized DocumentStorageService instead of creating a new instance.

self.doc_storage_service is already constructed in init. Reuse it to keep configuration consistent and reduce allocations.

-        # Initialize storage service for chunking
-        storage_service = DocumentStorageService(self.supabase_client)
+        # Reuse initialized storage service for chunking
+        storage_service = self.doc_storage_service
@@
-            chunks = storage_service.smart_chunk_text(markdown_content, chunk_size=5000)
+            chunks = storage_service.smart_chunk_text(markdown_content, chunk_size=5000)

Also applies to: 93-94

---

10-18: Remove unused imports (Ruff/Mypy hygiene).

urlparse, generate_code_summaries_batch, and add_code_examples_to_supabase are not used in this module.

-from urllib.parse import urlparse
@@
-from ..storage.code_storage_service import (
-    generate_code_summaries_batch,
-    add_code_examples_to_supabase,
-)
+# (removed unused code_storage_service imports)

---

110-123: Optional: include display metadata per chunk for better traceability.

If downstream analytics benefit from it, consider adding source_display_name to each chunk’s metadata (it’s currently only stored at the source level). This helps with ad-hoc queries without joining back to sources.

Example:

                 metadata = {
                     "url": source_url,
                     "title": doc.get("title", ""),
                     "description": doc.get("description", ""),
                     "source_id": source_id,
+                    "source_display_name": source_display_name,
                     "knowledge_type": request.get("knowledge_type", "documentation"),
                     "crawl_type": crawl_type,
                     "word_count": word_count,

---

220-224: Nit: avoid leading space and repeated string concatenation when building combined content.

Use a list-append + join pattern; it’s cleaner and avoids a leading space for the first chunk.

-            combined_content = ""
-            for chunk in source_contents[:3]:  # First 3 chunks for this source
-                if len(combined_content) + len(chunk) < 15000:
-                    combined_content += " " + chunk
-                else:
-                    break
+            combined_parts = []
+            total_len = 0
+            for chunk in source_contents[:3]:  # First 3 chunks for this source
+                if total_len + len(chunk) < 15000:
+                    combined_parts.append(chunk)
+                    total_len += len(chunk)
+                else:
+                    break
+            combined_content = " ".join(combined_parts)

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

python/tests/test_source_id_refactor.py (1)

176-181: Fix expected display name for path-based API URLs ("Example API" vs "Example").

extract_display_name returns " API" when "/api" appears in the path. For "https://example.com/api/v2/" the expected value should be "Example API", not "Example".

Apply this diff:

         test_cases = [
             ("https://api.github.com/", "GitHub API"),
             ("https://api.openai.com/v1/", "Openai API"),
-            ("https://example.com/api/v2/", "Example"),
+            ("https://example.com/api/v2/", "Example API"),
         ]

python/src/server/services/source_management_service.py (1)

302-307: Do not set updated_at to the string literal "now()"; rely on DB default/trigger.

PostgREST/Supabase will store "now()" as a string, corrupting your timestamp. Use a DB trigger/default instead and remove this field from application updates.

Apply this diff:

             update_data = {
                 "summary": summary,
                 "total_word_count": word_count,
                 "metadata": metadata,
-                "updated_at": "now()",
             }

python/src/server/services/crawling/document_storage_operations.py (1)

64-71: Avoid ZeroDivisionError and report accurate averages.

len(all_contents)/len(crawl_results) can divide by zero and overstates averages when documents are skipped. Track processed docs.

         all_metadatas = []
         source_word_counts = {}
         url_to_full_document = {}
+        processed_docs = 0
@@
-            if not markdown_content:
+            if not markdown_content:
                 continue
+            processed_docs += 1
@@
-        safe_logfire_info(
-            f"Document storage | documents={len(crawl_results)} | chunks={len(all_contents)} | avg_chunks_per_doc={len(all_contents) / len(crawl_results):.1f}"
-        )
+        avg_chunks = (len(all_contents) / processed_docs) if processed_docs else 0.0
+        safe_logfire_info(
+            f"Document storage | documents={processed_docs} | chunks={len(all_contents)} | avg_chunks_per_doc={avg_chunks:.1f}"
+        )

Also applies to: 141-143

python/tests/test_source_id_refactor.py (1)

71-86: Normalization tests only cover casing; AI summary mentions scheme normalization.

The current tests verify case-only normalization. The AI summary claims "URL normalization yields identical IDs for variations (case, scheme, host casing)," but generate_unique_source_id only lowercases the URL. http vs https (and trailing slashes, default ports) will produce different IDs today. Either:

• add tests to assert the current behavior (distinct IDs for different schemes), or
• enhance normalization to canonicalize scheme/ports/trailing slashes and then add tests accordingly.

python/src/server/services/source_management_service.py (5)

365-367: Preserve stack traces in logs for diagnostics.

Include exc_info=True so logs retain full tracebacks per guidelines.

-        search_logger.error(f"Error updating source {source_id}: {e}")
+        search_logger.error(f"Error updating source {source_id}: {e}", exc_info=True)

---

170-176: Redundant imports inside function.

import os is already at module top; remove the inner import to satisfy Ruff.

-                import os
-
                 import openai

---

45-51: Docstring references to "domain" are outdated post-refactor.

source_id is now a 16-char hash; docstrings should reflect that to avoid confusion in prompts/logs.

-        source_id: The source ID (domain)
+        source_id: The unique source identifier (16-char hash or file_* id)

Also applies to: 154-161

---

235-237: Include exception context in title-generation error logs.

Add exc_info for easier debugging.

-            search_logger.error(f"Error generating title for {source_id}: {e}")
+            search_logger.error(f"Error generating title for {source_id}: {e}", exc_info=True)

---

59-74: Consider using source_display_name in the summary prompt as context.

extract_source_summary currently references source_id in the prompt, which will be a hash. Passing a human-friendly name (when available) will improve summaries. This requires threading a display name through callers; optional but recommended.

python/src/server/services/crawling/document_storage_operations.py (3)

223-229: Avoid blocking the event loop with synchronous LLM calls.

extract_source_summary performs a network call synchronously. Consider offloading to a thread or using an async client to prevent event loop stalls.

---

111-112: Default knowledge_type inconsistency ("documentation" vs "technical").

Chunk metadata defaults to "documentation" while update_source_info uses "technical". Unify defaults to avoid mixed data.

-                    "knowledge_type": request.get("knowledge_type", "documentation"),
+                    "knowledge_type": request.get("knowledge_type", "technical"),

Also applies to: 243-246

---

256-277: Prefer display name as title in fallback upsert path.

When the primary update fails, you upsert with title=source_id, which is a hash and not user-friendly. Use source_display_name when available.

-                    fallback_data = {
+                    fallback_data = {
                         "source_id": source_id,
-                        "title": source_id,  # Use source_id as title fallback
+                        "title": source_display_name or source_id,  # Prefer display name if available
                         "summary": summary,
                         "total_word_count": source_id_word_counts[source_id],
                         "metadata": {
                             "knowledge_type": request.get("knowledge_type", "technical"),
                             "tags": request.get("tags", []),
                             "auto_generated": True,
                             "fallback_creation": True,
                             "original_url": request.get("url"),
                         },
                     }

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

python/src/server/services/source_management_service.py (1)

302-307: Do not set updated_at to the string literal "now()"; rely on DB defaults/triggers.

PostgREST treats "now()" as a string and will corrupt the timestamp. Let the database set updated_at.

             update_data = {
                 "summary": summary,
                 "total_word_count": word_count,
                 "metadata": metadata,
-                "updated_at": "now()",
             }

python/src/server/services/source_management_service.py (4)

8-8: Remove redundant in-function import os statements; keep the single top-level import.

There are additional import os statements inside functions that are unnecessary now that Line 8 imports it at module scope.

Outside the selected range, remove the inner imports:

-            import os
             import openai

-                import os
                 import openai

---

208-214: Sanitize source_display_name before injecting into the LLM prompt.

Prevent prompt injection/formatting issues by stripping newlines and constraining length before interpolation.

-            # Use display name if available for better context
-            source_context = source_display_name if source_display_name else source_id
+            # Use display name if available for better context; sanitize to avoid prompt injection
+            source_context = (source_display_name or source_id)
+            source_context = source_context.replace("\n", " ").replace("\r", " ").strip()[:100]

---

315-321: Remove unused result variable from the update call.

The assignment is unused and can be dropped.

-            result = (
-                client.table("archon_sources")
-                .update(update_data)
-                .eq("source_id", source_id)
-                .execute()
-            )
+            client.table("archon_sources").update(update_data).eq("source_id", source_id).execute()

---

326-335: Good: prefer source_display_name as title; also sanitize and ensure fallback.

This aligns with the PR goal to avoid unnecessary AI title generation. Minor hardening: ensure the title is single-line and non-empty.

-            if source_display_name:
-                # Use the display name directly as the title (truncated to prevent DB issues)
-                title = source_display_name[:100].strip()
+            if source_display_name:
+                # Sanitize to a single line and truncate
+                sanitized = source_display_name.replace("\n", " ").replace("\r", " ").strip()
+                title = (sanitized[:100] if sanitized else source_id)
                 metadata = {
                     "knowledge_type": knowledge_type,
                     "tags": tags or [],
                     "auto_generated": False,
                 }

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

python/src/server/services/crawling/helpers/url_handler.py (1)

185-215: Canonicalize URLs before hashing; redact sensitive data in error logs.

Lowercasing alone will produce different IDs for equivalent URLs (trailing slash, fragments, default ports, reordered/tracking query params). This can reintroduce collisions/duplicates across “equivalent” URLs and undermine the PR’s objective. Also, logging the raw URL can leak tokens.

     @staticmethod
     def generate_unique_source_id(url: str) -> str:
         """
-        Generate a unique source ID from URL using hash.
+        Generate a deterministic source ID from a canonicalized URL using SHA-256.
@@
-        This creates a 16-character hash that is guaranteed to be unique
-        for each URL, solving race condition issues when multiple crawls
-        target the same domain.
+        This creates a 16-character hash that is extremely unlikely to collide
+        for distinct canonical URLs, solving race conditions when multiple crawls
+        target the same domain.
@@
-        try:
-            # Normalize URL for consistent hashing
-            normalized_url = url.strip().lower()
-            # Generate SHA256 hash and take first 16 characters
-            # 16 hex chars = 64 bits = ~18 quintillion unique values
-            return hashlib.sha256(normalized_url.encode("utf-8")).hexdigest()[:16]
-        except Exception as e:
-            logger.error(f"Error generating unique source ID for {url}: {e}")
-            # Fallback: use a hash of the error message + url to still get something unique
-            fallback = f"error_{url}_{str(e)}"
-            return hashlib.sha256(fallback.encode("utf-8")).hexdigest()[:16]
+        try:
+            # Canonicalize deterministically: lower-case scheme/host, drop default ports,
+            # drop fragment, normalize path (strip trailing slash except root),
+            # sort query params and drop common tracking params.
+            from urllib.parse import urlparse, urlunparse, parse_qsl, urlencode
+
+            parsed = urlparse(url.strip())
+            scheme = (parsed.scheme or "").lower()
+            netloc = (parsed.netloc or "").lower()
+            if netloc.endswith(":80") and scheme == "http":
+                netloc = netloc[:-3]
+            if netloc.endswith(":443") and scheme == "https":
+                netloc = netloc[:-4]
+
+            path = parsed.path or ""
+            if path.endswith("/") and path != "/":
+                path = path.rstrip("/")
+
+            tracking = {"utm_source","utm_medium","utm_campaign","utm_term","utm_content","gclid","fbclid"}
+            query_items = [(k, v) for k, v in parse_qsl(parsed.query, keep_blank_values=True) if k not in tracking]
+            query = urlencode(sorted(query_items))
+
+            canonical = urlunparse((scheme, netloc, path, "", query, ""))  # fragment dropped
+            return hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:16]
+        except Exception as e:
+            # Avoid logging raw query params/tokens; include stack trace
+            try:
+                redacted = url.split("?", 1)[0]
+            except Exception:
+                redacted = "<unparseable-url>"
+            logger.error(f"Error generating unique source ID for {redacted}: {e}", exc_info=True)
+            # Fallback: hash a safe concatenation
+            fallback = f"error_{redacted}_{str(e)}"
+            return hashlib.sha256(fallback.encode("utf-8")).hexdigest()[:16]

Optionally consider 24 hex chars (96 bits) to further reduce birthday-bound collision risk, while keeping IDs compact. Happy to wire tests if you want.

python/src/server/services/crawling/document_storage_operations.py (2)

78-86: Bug: local variable source_url shadows function parameter; corrupts source records and logs.

The loop assigns source_url = doc.get("url", ""), overwriting the function parameter meant to carry the original crawl URL. This flows into _create_source_records and can store an arbitrary doc URL as the source_url.

-            source_url = doc.get("url", "")
-            markdown_content = doc.get("markdown", "")
+            doc_url = doc.get("url", "")
+            markdown_content = doc.get("markdown", "")
@@
-            url_to_full_document[source_url] = markdown_content
+            url_to_full_document[doc_url] = markdown_content
@@
-            source_id = original_source_id
-            safe_logfire_info(f"Using original source_id '{source_id}' for URL '{source_url}'")
+            source_id = original_source_id
+            safe_logfire_info(f"Using original source_id '{source_id}' for URL '{doc_url}'")
@@
-                all_urls.append(source_url)
+                all_urls.append(doc_url)
@@
-                    "url": source_url,
+                    "url": doc_url,
@@
-            await self._create_source_records(
-                all_metadatas, all_contents, source_word_counts, request,
-                source_url, source_display_name
-            )
+            await self._create_source_records(
+                all_metadatas, all_contents, source_word_counts, request,
+                source_url, source_display_name
+            )

Also applies to: 90-103, 104-122, 131-137, 311-338

---

64-71: Fix avg_chunks_per_doc and avoid ZeroDivisionError; log processed count.

Average uses total requested docs (crawl_results) and will divide by zero when no docs are processed. Track processed docs instead.

-        all_metadatas = []
+        all_metadatas = []
         source_word_counts = {}
         url_to_full_document = {}
+        processed_docs = 0
@@
-            if not markdown_content:
+            if not markdown_content:
                 continue
+            processed_docs += 1
@@
-        safe_logfire_info(
-            f"Document storage | documents={len(crawl_results)} | chunks={len(all_contents)} | avg_chunks_per_doc={len(all_contents) / len(crawl_results):.1f}"
-        )
+        avg_chunks = (len(all_contents) / processed_docs) if processed_docs else 0.0
+        safe_logfire_info(
+            f"Document storage | documents={processed_docs} | chunks={len(all_contents)} | avg_chunks_per_doc={avg_chunks:.1f}"
+        )

Also applies to: 140-143

python/src/server/services/crawling/helpers/url_handler.py (4)

31-31: Make sitemap/txt checks robust and case-insensitive.

Parse the URL and lower-case the path to avoid false negatives (e.g., uppercase extensions, query strings). Also prefer avoiding repeated parsing work in hot paths.

-            return url.endswith("sitemap.xml") or "sitemap" in urlparse(url).path
+            parsed = urlparse(url)
+            path = (parsed.path or "").lower()
+            return path.endswith("/sitemap.xml") or "sitemap" in path
@@
-            return url.endswith(".txt")
+            parsed = urlparse(url)
+            path = (parsed.path or "").lower()
+            return path.endswith(".txt")

Also applies to: 48-48

---

165-184: Redact query strings in logs for GitHub URL transformation.

Avoid leaking tokens/params in logs; log a redacted URL (no query/fragment). This also keeps logs cleaner.

-            logger.info(f"Transformed GitHub file URL to raw: {url} -> {raw_url}")
+            redacted_in = url.split("?", 1)[0]
+            redacted_out = raw_url.split("?", 1)[0]
+            logger.info(f"Transformed GitHub file URL to raw: {redacted_in} -> {redacted_out}")
@@
-            logger.warning(
-                f"GitHub directory URL detected: {url} - consider using specific file URLs or GitHub API"
-            )
+            redacted = url.split("?", 1)[0]
+            logger.warning(
+                f"GitHub directory URL detected: {redacted} - consider using specific file URLs or GitHub API"
+            )

---

360-363: Redact URL and include stack trace on display-name extraction failures.

Follow logging guidelines: avoid leaking tokens and preserve stack traces.

-            logger.warning(f"Error extracting display name for {url}: {e}, using URL")
+            redacted = url.split("?", 1)[0]
+            logger.warning(f"Error extracting display name for {redacted}: {e}, using URL", exc_info=True)

---

268-279: Polish display-name casing for special files.

Use conventional casing (“LLMs.txt”, “Sitemap.xml”) for nicer UI output.

-                        return f"{base_name} - Llms.Txt"
+                        return f"{base_name} - LLMs.txt"
@@
-                        return f"{base_name} - Sitemap.Xml"
+                        return f"{base_name} - Sitemap.xml"
@@
-                    return f"{formatted} - Sitemap.Xml"
+                    return f"{formatted} - Sitemap.xml"
@@
-                    return f"{formatted} - Llms.Txt"
+                    return f"{formatted} - LLMs.txt"

Also applies to: 320-341

python/src/server/services/crawling/crawling_service.py (3)

259-259: Redact query strings in info logs to avoid token leakage.

Several info logs include the full URL; redact queries/fragments in logs while keeping IDs and context.

-        safe_logfire_info(f"Starting background crawl orchestration | url={url}")
+        safe_logfire_info(f"Starting background crawl orchestration | url={url.split('?', 1)[0]}")
@@
-            safe_logfire_info(f"Starting async crawl orchestration | url={url} | task_id={task_id}")
+            safe_logfire_info(f"Starting async crawl orchestration | url={url.split('?', 1)[0]} | task_id={task_id}")
@@
-            safe_logfire_info(
-                f"Generated unique source_id '{original_source_id}' and display name '{source_display_name}' from URL '{url}'"
-            )
+            safe_logfire_info(
+                f"Generated unique source_id '{original_source_id}' and display name '{source_display_name}' from URL '{url.split('?', 1)[0]}'"
+            )

Also applies to: 305-305, 311-312

---

395-401: Sanity check: validate source_id round-trip.

Ensure the source_id returned from storage matches the one derived at the start; log a warning if not.

             storage_results = await self.doc_storage_ops.process_and_store_documents(
                 crawl_results,
                 request,
                 crawl_type,
                 original_source_id,
                 doc_storage_callback,
                 self._check_cancellation,
                 source_url=url,
                 source_display_name=source_display_name,
             )
+            if storage_results.get("source_id") != original_source_id:
+                safe_logfire_error(
+                    f"source_id mismatch after storage | expected={original_source_id} | actual={storage_results.get('source_id','')}"
+                )

---

483-486: Preserve stack trace on orchestration errors.

safe_logfire_error records the message, but we also want a full traceback for postmortems.

-        except Exception as e:
-            safe_logfire_error(f"Async crawl orchestration failed | error={str(e)}")
+        except Exception as e:
+            safe_logfire_error(f"Async crawl orchestration failed | error={str(e)}")
+            # Also emit stack trace via standard logger
+            logger.exception("Async crawl orchestration failed")

python/tests/test_code_extraction_source_id.py (1)

40-46: Fix mocked storage payload shape; avoid referencing late-populated list.

The mocked _prepare_code_examples_for_storage currently returns a list (not the dict the service expects) and closes over extracted_blocks before it’s populated, risking None/KeyErrors if the implementation changes. Either remove the mock or return a dict with required keys.

-        code_service._generate_code_summaries = AsyncMock(return_value=[{"summary": "Test code"}])
-        code_service._prepare_code_examples_for_storage = Mock(return_value=[
-            {"source_id": extracted_blocks[0]["source_id"] if extracted_blocks else None}
-        ])
+        code_service._generate_code_summaries = AsyncMock(return_value=[{"summary": "Test code"}])
+        # Return a minimal-but-correct storage payload shape
+        code_service._prepare_code_examples_for_storage = Mock(return_value={
+            "urls": [],
+            "chunk_numbers": [],
+            "examples": [],
+            "summaries": [{"summary": "Test code"}],
+            "metadatas": [{"source_id": "placeholder"}],
+        })

Alternative: drop the _prepare_code_examples_for_storage mock entirely—since _store_code_examples is already mocked, you can exercise the real preparation logic safely.

Also applies to: 59-70

python/src/server/services/crawling/code_extraction_service.py (2)

339-342: Include stack traces on errors in extraction/storage.

Add a standard logger with exception stack traces alongside safe_logfire_error for better diagnostics.

-from ...config.logfire_config import safe_logfire_error, safe_logfire_info
+from ...config.logfire_config import safe_logfire_error, safe_logfire_info, get_logger
+logger = get_logger(__name__)
@@
-            safe_logfire_error(
-                    f"Error processing code from document | url={doc.get('url')} | error={str(e)}"
-                )
+            safe_logfire_error(
+                f"Error processing code from document | url={doc.get('url')} | error={str(e)}"
+            )
+            logger.exception("Error processing code from document")
@@
-        except Exception as e:
-            safe_logfire_error(f"Error storing code examples | error={str(e)}")
+        except Exception as e:
+            safe_logfire_error(f"Error storing code examples | error={str(e)}")
+            logger.exception("Error storing code examples")
             return 0

Also applies to: 1526-1529, 11-11

---

346-357: Docstring drift: remove unused parameter from _extract_html_code_blocks docs.

The signature no longer has a min_length argument; update the docstring to match.

-        Args:
-            content: The content to search for HTML code patterns
-            min_length: Minimum length for code blocks
+        Args:
+            content: The content to search for HTML code patterns

python/src/server/services/crawling/document_storage_operations.py (1)

138-139: Redact URL keys when logging url_to_full_document to avoid leaking tokens.

Log redacted keys (no query/fragment) to keep PII/credentials out of logs.

-        safe_logfire_info(f"url_to_full_document keys: {list(url_to_full_document.keys())[:5]}")
+        keys = list(url_to_full_document.keys())[:5]
+        redacted_keys = [k.split('?', 1)[0] for k in keys]
+        safe_logfire_info(f"url_to_full_document keys: {redacted_keys}")

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

python/src/server/services/source_management_service.py (4)

36-38: Fix Optional typing for provider to satisfy mypy/3.12

Annotate provider as optional; provider: str = None violates strict typing. Keep signatures consistent across the module.

-def extract_source_summary(
-    source_id: str, content: str, max_length: int = 500, provider: str = None
-) -> str:
+def extract_source_summary(
+    source_id: str, content: str, max_length: int = 500, provider: str | None = None
+) -> str:

---

333-365: Good: AI source_type is overridden by URL inference

This addresses the earlier correctness concern with AI metadata carrying an incorrect source_type.

---

149-151: Make provider optional type; keep signature consistent with docstring

Same Optional typing fix as above for generate_source_title_and_metadata.

 def generate_source_title_and_metadata(
     source_id: str,
     content: str,
     knowledge_type: str = "technical",
     tags: list[str] | None = None,
-    provider: str = None,
+    provider: str | None = None,
     source_display_name: str | None = None,
 ) -> tuple[str, dict[str, Any]]:

---

309-314: Do not set updated_at to string "now()" — rely on DB trigger

This writes a literal string to the column and corrupts timestamps via PostgREST. Remove it and rely on your migration trigger.

             update_data = {
                 "summary": summary,
                 "total_word_count": word_count,
                 "metadata": metadata,
-                "updated_at": "now()",
             }

If you need to force a write to bump updated_at, touch a benign field or use a SQL function on the server side. I can help produce the trigger if needed.

python/src/server/services/crawling/document_storage_operations.py (2)

78-86: Correct fix: avoid shadowing the source_url parameter

Renaming the local variable to doc_url prevents clobbering the function argument. This unblocks correct propagation and logging.

---

141-143: Division by zero and incorrect average when no docs are processed

len(all_contents) / len(crawl_results) can raise ZeroDivisionError and uses requested docs rather than processed docs. Track processed docs or derive from url_to_full_document.

Apply this focused diff for the logging lines:

-        safe_logfire_info(
-            f"Document storage | documents={len(crawl_results)} | chunks={len(all_contents)} | avg_chunks_per_doc={len(all_contents) / len(crawl_results):.1f}"
-        )
+        processed_docs = len(url_to_full_document)
+        avg_chunks = (len(all_contents) / processed_docs) if processed_docs else 0.0
+        safe_logfire_info(
+            f"Document storage | documents={processed_docs} | chunks={len(all_contents)} | avg_chunks_per_doc={avg_chunks:.1f}"
+        )

If you prefer an explicit counter, add the following outside this range:

@@
-        url_to_full_document = {}
+        url_to_full_document = {}
+        processed_docs = 0
@@
-            url_to_full_document[doc_url] = markdown_content
+            url_to_full_document[doc_url] = markdown_content
+            processed_docs += 1

python/src/server/services/source_management_service.py (4)

152-162: Docstring missing new parameters

Please document provider and source_display_name to match the signature.

     """
     Generate a user-friendly title and metadata for a source based on its content.
 
     Args:
         source_id: The source ID (domain)
         content: Sample content from the source
         knowledge_type: Type of knowledge (default: "technical")
         tags: Optional list of tags
+        provider: Optional provider override for LLM selection
+        source_display_name: Optional human-friendly name to guide title generation

---

238-245: Clarify “caller-determined” source_type; ensure it never leaks as the default

You default source_type to "url" here and then rely on callers to override. That’s fine, but it can be easy to forget. Consider setting it only at the call site (or set to None here) to avoid accidental writes with a wrong default. If you keep the default, add a brief inline comment stating which call sites override it (e.g., update_source_info).

---

266-275: Docstring for update_source_info lacks source_url and source_display_name

Add these to the Args section to keep docs and signature aligned.

     Args:
         client: Supabase client
         source_id: The source ID (domain)
         summary: Summary of the source
         word_count: Total word count for the source
         content: Sample content for title generation
         knowledge_type: Type of knowledge
         tags: List of tags
         update_frequency: Update frequency in days
+        original_url: The crawl entry URL (kept for historical context)
+        source_url: The canonical source URL for this source_id
+        source_display_name: Human-friendly name for UI and title generation

---

289-296: Deduplicate source_type inference logic

You compute source_type from source_url/original_url in two places. Consider extracting a tiny helper to avoid drift.

@@
-            # Determine source_type based on source_url or original_url
-            if source_url and source_url.startswith("file://"):
-                source_type = "file"
-            elif original_url and original_url.startswith("file://"):
-                source_type = "file"
-            else:
-                source_type = "url"
+            def _infer_source_type(source_url: str | None, original_url: str | None) -> str:
+                for u in (source_url, original_url):
+                    if u and u.startswith("file://"):
+                        return "file"
+                return "url"
+            source_type = _infer_source_type(source_url, original_url)
@@
-                # Determine source_type based on source_url or original_url
-                if source_url and source_url.startswith("file://"):
-                    source_type = "file"
-                elif original_url and original_url.startswith("file://"):
-                    source_type = "file"
-                else:
-                    source_type = "url"
+                source_type = _infer_source_type(source_url, original_url)

Also applies to: 338-345

python/src/server/services/crawling/document_storage_operations.py (1)

232-236: Consider retries around DB writes to harden against transient failures

Supabase/PostgREST can intermittently fail. A light exponential backoff around update_source_info and the fallback upsert would improve resilience. Optional but recommended.

Also applies to: 250-286

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

python/src/server/services/crawling/document_storage_operations.py (2)

145-148: Accurate averages and div-by-zero guard — nice fix

Switching to processed_docs fixes both accuracy and ZeroDivisionError risks flagged earlier. LGTM.

---

245-257: Offload DB upsert/update to thread to keep async loop responsive

update_source_info performs network I/O; run it in a worker thread. This aligns with how extract_source_summary is offloaded.

-            try:
-                update_source_info(
+            try:
+                await asyncio.to_thread(
+                    update_source_info,
                     client=self.supabase_client,
                     source_id=source_id,
                     summary=summary,
                     word_count=source_id_word_counts[source_id],
                     content=combined_content,
-                    knowledge_type=request.get("knowledge_type", "technical"),
+                    knowledge_type=request.get("knowledge_type", "documentation"),
                     tags=request.get("tags", []),
                     update_frequency=0,  # Set to 0 since we're using manual refresh
                     original_url=request.get("url"),  # Store the original crawl URL
                     source_url=source_url,
                     source_display_name=source_display_name,
-                )
+                )

python/src/server/services/crawling/document_storage_operations.py (5)

91-93: Avoid blocking the event loop during chunking

smart_chunk_text is synchronous; offload to a worker thread to prevent event-loop starvation on large documents.

-            chunks = storage_service.smart_chunk_text(markdown_content, chunk_size=5000)
+            chunks = await asyncio.to_thread(
+                storage_service.smart_chunk_text, markdown_content, chunk_size=5000
+            )

---

75-79: Support both sync and async cancellation checks

cancellation_check may be a coroutine. Detect and await it when needed to avoid unhandled coroutine warnings or no-ops.

-            if cancellation_check:
-                cancellation_check()
+            if cancellation_check:
+                if asyncio.iscoroutinefunction(cancellation_check):
+                    await cancellation_check()
+                else:
+                    cancellation_check()
@@
-                if cancellation_check and i % 10 == 0:  # Check every 10 chunks
-                    cancellation_check()
+                if cancellation_check and i % 10 == 0:  # Check every 10 chunks
+                    if asyncio.iscoroutinefunction(cancellation_check):
+                        await cancellation_check()
+                    else:
+                        cancellation_check()

Also applies to: 100-103

---

234-236: Preserve stack traces in logs; include structured context

Logging only str(e) loses the traceback. If safe_logfire_error supports it, pass exc_info=True and include structured context (source_id, url, counts).

Example (adjust to your logging helper signature):

safe_logfire_error(
    f"Failed to generate AI summary for '{source_id}'",
    exc_info=True,
)
# Similarly for source upsert/update failures and double-failure raise

Also applies to: 260-262, 289-295

---

297-317: Reduce DB round-trips: verify sources in bulk

Querying per source_id is N round-trips. Use a single in_ filter and compute missing IDs.

-        if unique_source_ids:
-            for source_id in unique_source_ids:
-                try:
-                    source_check = (
-                        self.supabase_client.table("archon_sources")
-                        .select("source_id")
-                        .eq("source_id", source_id)
-                        .execute()
-                    )
-                    if not source_check.data:
-                        raise Exception(
-                            f"Source record verification failed - '{source_id}' does not exist in sources table"
-                        )
-                    safe_logfire_info(f"Source record verified for '{source_id}'")
-                except Exception as e:
-                    safe_logfire_error(f"Source verification failed for '{source_id}': {str(e)}")
-                    raise
-            safe_logfire_info(
-                f"All {len(unique_source_ids)} source records verified - proceeding with document storage"
-            )
+        if unique_source_ids:
+            resp = (
+                self.supabase_client.table("archon_sources")
+                .select("source_id")
+                .in_("source_id", list(unique_source_ids))
+                .execute()
+            )
+            existing = {row["source_id"] for row in (resp.data or [])}
+            missing = set(unique_source_ids) - existing
+            if missing:
+                raise Exception(f"Source record verification failed - missing: {sorted(missing)}")
+            safe_logfire_info(
+                f"All {len(unique_source_ids)} source records verified - proceeding with document storage"
+            )

---

109-121: Unify Knowledge Type Default Across the Pipeline

There are scattered literal defaults for knowledge_type—some components default to "documentation", others to "technical". This inconsistency makes it harder to reason about analytics, filters, and test expectations. To clean this up, I recommend introducing a single constant and using it everywhere:

• Define in a central module (e.g. python/src/server/constants.py):

DEFAULT_KNOWLEDGE_TYPE = "documentation"  # or "technical", per team consensus

• Replace all literal defaults and metadata assignments with this constant:
– Pydantic model in api_routes/knowledge_api.py (lines ~64, 74)
– MCP client in services/mcp_service_client.py (line ~61)
– Source management service in services/source_management_service.py (lines ~146, 256, 576)
– Crawling storage operations in services/crawling/document_storage_operations.py (lines ~115, 251, 272)
– Storage services in services/storage/storage_services.py (lines ~27, 212)
– Knowledge item service in services/knowledge/knowledge_item_service.py (lines ~156, 374)

• In each processing method, compute once at entry:

kt = request.get("knowledge_type", DEFAULT_KNOWLEDGE_TYPE)

then pass knowledge_type=kt and set metadata["knowledge_type"] = kt.

• Update all tests to reference the constant or to expect the unified default:
– python/tests/test_source_url_shadowing.py (line 62)
– python/tests/test_async_source_summary.py (line 56)
– python/tests/test_crawl_orchestration_isolated.py (lines 200, 266, 313, 429)

This refactor ensures one source of truth for the default and removes repeated string literals, reducing future drift.

python/tests/test_async_source_summary.py (3)

19-83: Solid non-blocking test; also assert new context propagation

Great coverage of asyncio.to_thread behavior. Add assertions to verify source_url and source_display_name are forwarded to update_source_info.

-            with patch('src.server.services.crawling.document_storage_operations.update_source_info'):
+            with patch('src.server.services.crawling.document_storage_operations.update_source_info') as mock_update:
@@
-                        assert total_time < 1.0, "Should complete in reasonable time"
+                        assert total_time < 1.0, "Should complete in reasonable time"
+
+                        # Ensure new context fields are propagated
+                        call_kwargs = mock_update.call_args.kwargs
+                        assert call_kwargs["source_url"] == "https://example.com"
+                        assert call_kwargs["source_display_name"] == "Example Site"

---

40-42: Remove unused mocks of smart_chunk_text

_These tests call create_source_records directly; chunking isn’t invoked here. Cleaning these lines reduces noise.

-        # Mock the storage service
-        doc_storage.doc_storage_service.smart_chunk_text = Mock(
-            return_value=["chunk1", "chunk2"]
-        )

Apply similar removals in the other two tests.

Also applies to: 97-99, 151-153, 213-216

---

135-150: Test name/docstring imply concurrency that isn’t present

current implementation awaits each to_thread call sequentially; sources aren’t processed concurrently. Either rename the test or parallelize summary generation with gather + a semaphore.

Option A — rename:

-    async def test_multiple_sources_concurrent_summaries(self):
-        """Test that multiple source summaries are generated concurrently."""
+    async def test_multiple_sources_threaded_execution(self):
+        """Test that each source summary runs in a worker thread without blocking the loop."""

Option B — make code concurrent (then update test to assert overlaps):

• In _create_source_records, build tasks per source_id with asyncio.to_thread and await asyncio.gather with a bounded concurrency to avoid API/db overload.

Also applies to: 182-193

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

python/src/server/services/source_management_service.py (3)

36-38: Use Optional typing for provider to satisfy mypy under Python 3.12.

provider: str = None violates strict typing.

-def extract_source_summary(
-    source_id: str, content: str, max_length: int = 500, provider: str = None
-) -> str:
+def extract_source_summary(
+    source_id: str, content: str, max_length: int = 500, provider: str | None = None
+) -> str:

---

143-151: Type correctness + docstring completeness for generate_source_title_and_metadata.

Mirror the Optional typing for provider and document both provider and source_display_name in Args.

-def generate_source_title_and_metadata(
+def generate_source_title_and_metadata(
     source_id: str,
     content: str,
     knowledge_type: str = "technical",
     tags: list[str] | None = None,
-    provider: str = None,
+    provider: str | None = None,
     source_display_name: str | None = None,
 ) -> tuple[str, dict[str, Any]]:
@@
-    Args:
+    Args:
         source_id: The source ID (domain)
         content: Sample content from the source
         knowledge_type: Type of knowledge (default: "technical")
         tags: Optional list of tags
+        provider: Optional provider override
+        source_display_name: Optional human-friendly name to guide title generation

---

309-314: Do not set updated_at to the string "now()"; rely on a DB trigger.

PostgREST treats "now()" as a literal; this will corrupt timestamps. Remove updated_at from update payload and let the DB trigger populate it.

             update_data = {
                 "summary": summary,
                 "total_word_count": word_count,
                 "metadata": metadata,
-                "updated_at": "now()",
             }

python/src/server/services/crawling/document_storage_operations.py (1)

79-87: Guard against missing URL; don’t persist chunks with empty doc_url.

Empty URLs will corrupt url_to_full_document and break downstream joins. Add a strict guard and log context before continuing.

-            doc_url = doc.get("url", "")
-            markdown_content = doc.get("markdown", "")
+            doc_url = doc.get("url", "")
+            markdown_content = doc.get("markdown", "")
+
+            # Validate required fields
+            if not doc_url:
+                safe_logfire_error("Skipping document with missing URL; cannot store or map content")
+                continue
             if not markdown_content:
                 continue

python/tests/test_source_race_condition.py (1)

30-43: Also assert new columns are propagated in upserts.

Since this PR adds source_url and source_display_name, verifying they are present in upsert payloads would harden the regression surface for this class of bugs.

Apply:

-        def track_upsert(data):
-            """Track upsert calls."""
-            successful_creates.append(data["source_id"])
+        def track_upsert(data):
+            """Track upsert calls."""
+            successful_creates.append(data)  # capture full payload for validation
             return Mock(execute=mock_execute)
@@
-        assert len(successful_creates) == 5, "All 5 attempts should succeed"
-        assert all(sid == "test_source_123" for sid in successful_creates)
+        assert len(successful_creates) == 5, "All 5 attempts should succeed"
+        assert all(item["source_id"] == "test_source_123" for item in successful_creates)
+        assert all(item["source_url"] == "https://example.com" for item in successful_creates)
+        # Display name is accepted but title preservation happens server-side; we still verify propagation
+        assert all("source_display_name" in item for item in successful_creates)

python/tests/test_async_source_summary.py (3)

74-85: Relax timing threshold to avoid CI flakiness.

The function is async but still does real work and scheduling; <1.0s can intermittently fail under load.

-                        assert total_time < 1.0, "Should complete in reasonable time"
+                        assert total_time < 2.0, "Should complete in reasonable time"

---

136-141: Docstring overpromises concurrency; clarify behavior.

Summaries run in threads but sequentially per source (awaited in a loop). Adjust wording to reduce confusion.

-        """Test that multiple source summaries are generated concurrently."""
+        """Test that multiple source summaries are generated in threads without blocking the event loop (sequential per source)."""

---

298-304: Relax timing threshold to reduce false negatives.

Similar rationale as earlier test: <1.0s can be too strict for some runners.

-                        assert total_time < 1.0, "Should complete in reasonable time"
+                        assert total_time < 2.0, "Should complete in reasonable time"

python/src/server/services/source_management_service.py (3)

289-296: Normalize source_type inference via URL scheme; avoid ad hoc checks.

file:// is handled, but other schemes (s3://, gs://, data:, ftp) will be misclassified as "url" or "file". Use urllib.parse to infer a stable source_type and keep logic in one helper.

Example:

+from urllib.parse import urlparse
+
+def _infer_source_type(source_url: str | None, original_url: str | None) -> str:
+    url = source_url or original_url or ""
+    scheme = urlparse(url).scheme.lower()
+    if scheme == "file":
+        return "file"
+    # treat known remote schemes as url
+    return "url"

@@
-            if source_url and source_url.startswith("file://"):
-                source_type = "file"
-            elif original_url and original_url.startswith("file://"):
-                source_type = "file"
-            else:
-                source_type = "url"
+            source_type = _infer_source_type(source_url, original_url)
@@
-                if source_url and source_url.startswith("file://"):
-                    source_type = "file"
-                elif original_url and original_url.startswith("file://"):
-                    source_type = "file"
-                else:
-                    source_type = "url"
+                source_type = _infer_source_type(source_url, original_url)
@@
-                if source_url and source_url.startswith("file://"):
-                    metadata["source_type"] = "file"
-                elif original_url and original_url.startswith("file://"):
-                    metadata["source_type"] = "file"
-                else:
-                    metadata["source_type"] = "url"
+                metadata["source_type"] = _infer_source_type(source_url, original_url)

Also applies to: 339-345, 359-364

---

100-116: Add retries with backoff for LLM calls; fail with contextual errors.

External API calls should use retries and surface context on failure per coding guidelines. Wrap chat completions with a bounded backoff (e.g., tenacity).

Illustrative change:

from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

@retry(
    reraise=True,
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=0.5, min=0.5, max=4),
    retry=retry_if_exception_type(Exception),
)
def _create_summary_with_llm(client, model, messages):
    return client.chat.completions.create(model=model, messages=messages)

# then use:
response = _create_summary_with_llm(client, model_choice, messages)

Log the source_id and content length in the exception path for debugging.

Also applies to: 218-227

---

266-275: Docstring should include new args: original_url, source_url, source_display_name.

Keep the function contract self-documenting.

     Args:
         client: Supabase client
         source_id: The source ID (domain)
         summary: Summary of the source
         word_count: Total word count for the source
         content: Sample content for title generation
         knowledge_type: Type of knowledge
         tags: List of tags
         update_frequency: Update frequency in days
+        original_url: Original crawl URL for provenance
+        source_url: Canonical URL of the source (used for type inference and UI)
+        source_display_name: Human-friendly name to store as title when available

python/src/server/services/crawling/document_storage_operations.py (1)

220-224: Avoid leading space when building combined_content.

Join chunks via list accumulation for cleaner content.

-            combined_content = ""
-            for chunk in source_contents[:3]:  # First 3 chunks for this source
-                if len(combined_content) + len(chunk) < 15000:
-                    combined_content += " " + chunk
-                else:
-                    break
+            parts = []
+            for chunk in source_contents[:3]:  # First 3 chunks for this source
+                if sum(len(p) for p in parts) + len(chunk) < 15000:
+                    parts.append(chunk)
+                else:
+                    break
+            combined_content = " ".join(parts)

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

Learnt from: Chillbruhhh
PR: coleam00/Archon#378
File: python/src/server/services/storage/document_storage_service.py:304-306
Timestamp: 2025-08-20T19:38:04.050Z
Learning: The archon_crawled_pages table in the Archon project has a table-level unique constraint on (url, chunk_number) defined inline in the CREATE TABLE statement in migration/complete_setup.sql at line 202, which allows upsert operations with on_conflict="url,chunk_number" to work properly without requiring additional migrations.