Refer to the AIL Importers Documentation

AIL is an analysis tool, not a collector! However, if you want to collect some pastes and feed them to AIL, the procedure is described below. Nevertheless, moderate your queries!

1. AIL Importers
2. ZMQ: Be a collaborator of CIRCL and ask to access our feed. It will be sent to the static IP you are using for AIL.

To add a new processing or analysis module to AIL, follow these steps:

1. Add your module name in ./configs/modules.cfg and subscribe to at least one module at minimum (Usually, Item).
2. Use ./bin/modules/modules/TemplateModule.py as a sample module and create a new file in bin/modules with the module name used in the modules.cfg configuration.

Contributions are welcome! Fork the repository, experiment with the code, and submit your modules or patches through a pull request.

AIL supports crawling of websites and Tor hidden services. Ensure your Tor client's proxy configuration is correct, especially the SOCKS5 proxy settings.

Install Lacus

1. Lacus URL:
   In the web interface, go to Crawlers > Settings and click on the Edit button

2. Number of Crawlers: Choose the number of crawlers you want to launch

Chats message can be translated using libretranslate, an open-source self-hosted machine translation.

1. Install LibreTranslate by running the following command:

pip install libretranslate

2. Run libretranslate:

libretranslate

To enable LibreTranslate for chat translation, edit the LibreTranslate URL in the ./configs/core.cfg file under the [Translation] section.

[Translation]
libretranslate = http://127.0.0.1:5000

The Indexer module integrates with Meilisearch, a search engine used to index and search the content of chat messages, image and domain descriptions, and crawled content.

Install Meilisearch using the official installation script:

curl -L https://install.meilisearch.com | sh

./meilisearch --no-analytics

💡 Use the --no-analytics flag to Disable Meilisearch telemetry.

To enable the indexer in AIL, activate Meilisearch and configure its connection details in ./configs/core.cfg under the [Indexer] section:

# Indexer configuration
[Indexer]
meilisearch = True
meilisearch_url = http://<meilisearch_url>        # URL where Meilisearch is running (e.g., `http://127.0.0.1:7700`)  
meilisearch_key = <meilisearch_db_password>       # Meilisearch master key or API key

⚠️ Meilisearch indexes created with AIL versions before v6.7 are automatically removed during the update.
Search indexing was refactored, so the indexes must be rebuilt using the new schema.

Use tools/reindex_meilisearch.py to rebuild the indexes:

# Reindex everything
python3 tools/reindex_meilisearch.py --type all

# Reindex only one dataset
python3 tools/reindex_meilisearch.py --type messages

If you want to reset and recreate the indexes before reindexing:

python3 tools/reindex_meilisearch.py --reset --type all