Documentation overhaul + examples

Summary

Split docs into a small docs site with quickstarts, configuration reference, API reference, and an ops guide (metrics/pprof).

Motivation

• Faster onboarding
• Clear runbooks for operating at scale

Scope

• /docs with mkdocs or mdBook
• Quickstart: local & Docker/Tor
• Example: crawl a tiny local site; Prometheus+Grafana sample dashboard
• Architecture diagrams

Acceptance Criteria

• A new user can run a crawl and see metrics in <10 minutes
• README becomes a short entry point linking to docs

Tasks

• [ ] Create docs framework and structure
• [ ] Add examples and diagrams
• [ ] Update README to point to docs

Assign it to me if you don't mind

@Ayush3603 Feel free to open a PR.