Crawler Shield

Protects YOU from digital art theft!

Comment

Inspiration

The idea for Crawler Shield was born out of the growing need to protect digital artists from the unauthorized use and sale of their work. Platforms like DeviantArt and Amazon host vast amounts of content, and all too often, creators discover their art being stolen, reposted, or even sold without their permission. This issue became personal when a friend of mine had her artwork taken without consent, leaving her frustrated and powerless. Seeing the impact firsthand, we decided to take action—creating a tool that empowers artists to safeguard their creations. Crawler Shield helps artists monitor and protect their work by scanning major platforms for stolen images, giving them the ability to detect misuse and take appropriate action.

What It Does

Crawler Shield is a Golang application designed to scan popular websites like DeviantArt and Amazon for stolen artwork. It automatically detects whether images uploaded by the creator are being used without permission, providing real-time alerts. The app shows where the stolen artwork is being used and guides the creator on the next steps to take for enforcement and protection of their intellectual property.

How We Built It

We built Crawler Shield using Golang for its speed and efficiency in web scraping and concurrency handling. We utilized libraries such as colly and goquery for web scraping and image comparison. The app scans images on specific high-ranking pages of DeviantArt and Amazon, comparing them with images uploaded by the user. The algorithm checks for visual similarities using hash comparison techniques with orbs for image recognition.

Challenges We Ran Into

  • Image Comparison Accuracy: One of the biggest challenges was ensuring that our image comparison algorithms could handle variations such as resizing or filters applied to the stolen artwork. Rate Limiting and Scraping Restrictions: We had to handle rate limits imposed by platforms like DeviantArt and Amazon. This required us to implement appropriate delays and error handling to avoid getting blocked.
    • Scaling: As the application grows, the need for scaling and optimizing the process to scan a larger number of pages and images becomes critical. Luckly we built a nice system with redis to avoid this problem.
    • Legal and Privacy Concerns: We had to carefully navigate the legal implications of scanning and storing images to ensure user data privacy was maintained.

Built With

Share this project:

Updates