CitationSleuth is an advanced Retrieval-Augmented Generation (RAG) system designed to detect and prevent hallucinations in Large Language Models. It employs a "Trust but Verify" architecture, combining Vector Search with Knowledge Graphs to audit AI answers against source documents.

• CitationSleuth_final_project.ipynb: The complete project pipeline (Ingestion, RAG, Verification, and UI).
• demo-file.pdf: The research paper (SayNav) used to demonstrate and test the system.
• Secret Keys.txt: Contains the necessary API credentials for testing.

Follow these steps to set up and run the project.

Upload CitationSleuth_final_project.ipynb to Google Colab.

Navigate to the code cell labeled Step 2: Imports, Auth, & Model Initialization.

1. Open the Secret Keys.txt file provided in this repository.
2. Copy ALL the values (HF_TOKEN, GOOGLE_API_KEY, NEO4J_URI, NEO4J_PASSWORD, and NGROK_TOKEN).
3. Paste them directly into the corresponding variables in the notebook code.

Model Selection: The system is configured to use gemini-2.5-flash by default for high speed and efficiency.

• Quota Limits: If you encounter a 429 Resource Exhausted error, it means the free tier quota for this model has been reached.
• Fix: You can generate a new api key and update it in step 2, or modify the model name in Step 2 to use other variants such as gemini-1.5-flash or gemini-2.5-flash-exp .

Troubleshooting HF_TOKEN: If you receive a 401 Unauthorized error, the provided Hugging Face token may have expired. You can generate a free "Read" token at Hugging Face Settings and use that instead.

Run Step 3: PDF Ingestion.

• The cell will display a "Choose Files" button.
• Upload the demo-file.pdf provided in this repository.
• The system will extract text and chunk the document for the database.

Run the cells from Step 4 through Step 12 to build the Knowledge Graph and run the backend benchmarks.

Troubleshooting Verification (Step 10): If Step 10 fails due to API limits:

• Ensure you are using your own personal Google API Key.
• Try switching the model in Step 2 (e.g., to gemini-1.5-flash).

Crucial Step: The User Interface runs as a separate application script (app.py), so it cannot access the variables from Step 2. You must manually inject the keys again here.

1. Look inside the code block for Step 13 (under %%writefile app.py).
2. Find the section marked # PASTE YOUR KEYS HERE.
3. Paste the following keys from your Secret Keys.txt file:
   • GOOGLE_API_KEY
   • NEO4J_URI
   • NEO4J_PASSWORD
4. Run the cell to generate the app.py file.

Run the final step. It will start the Streamlit server using the NGROK_TOKEN you provided in Step 2.

• Wait for the output to display a public URL (e.g., https://....ngrok-free.app).
• Click this link to open the CitationSleuth UI.

Once the app is open in your browser, follow this workflow:

1. Ingest Data:

   • In the sidebar, upload the same PDF file (demo-file.pdf).
   • Click the "Ingest" button. Wait for the green success message.

2. View Global Graph:

   • Before asking any questions, switch to the "Graph" tab.
   • You will see the Global Knowledge Graph, visualizing the entire document's concepts and connections.

3. Ask & Verify:

   • Switch to the "Chat" tab.
   • Type a question (e.g., "What are the three modules?") and press Enter.
   • The AI will generate an answer with a Verification Badge (Green for Supported, Red for Flagged).

4. View Context Graph:

   • After getting an answer, switch back to the "Graph" tab.
   • The view will automatically update to show the Context Graph—visualizing only the entities and relationships relevant to your last question.

Based on the HaluEval benchmark (1,000 samples), CitationSleuth achieves:

• Accuracy: 80.7%
• Precision: 72.2%
• Recall: 99.7% (Extremely effective at catching hallucinations)
• F1-Score: 83.7%

Presented By:

• Koushik Vasa
• Kavya Shivakumar
• Sehaj Gill