Enhance the DataProcessor class located in src/ragbuilder/data_processor.py to improve error handling and optimize efficiency. The following changes will be made:

1. Add Error Handling:

   • Implement try-except blocks to catch errors in file reading, URL processing, and directory processing.
   • Log error messages using the logger for better traceability and debugging.

2. Optimize File and Directory Handling:

   • Simplify file and directory path operations.
   • Use built-in Python utilities for more robust file handling.

3. Improve Multiprocessing Usage:

   • Refine the use of multiprocessing.Pool to reduce overhead and enhance progress tracking.

4. Logging Enhancements:

   • Add detailed logging at various steps to provide insights into the data processing workflow.

Assign this issue to me to start working on these improvements.