Great project!
When trying to crawl knowledge sources across the local docker network, the frontend URL validity checks in KnowlegeBasePage.tsx do well to validate publicly accessible URLs, but unfortunately block URLs that would otherwise be locally accessible to the stack and want to be ingested. For instance, scraping a local api instance for its openapi spec output. In such cases, a local TLD may be used (or no TLD at all) and so the checks for FQDN and google resolution would block its use.
Submitting the same to /api/knowledge-items/crawl directly does work fine.
cheers!
Great project!
When trying to crawl knowledge sources across the local docker network, the frontend URL validity checks in KnowlegeBasePage.tsx do well to validate publicly accessible URLs, but unfortunately block URLs that would otherwise be locally accessible to the stack and want to be ingested. For instance, scraping a local api instance for its openapi spec output. In such cases, a local TLD may be used (or no TLD at all) and so the checks for FQDN and google resolution would block its use.
Submitting the same to /api/knowledge-items/crawl directly does work fine.
cheers!