iHi! Interesting project.

I'm not really familiar with the robots.txt file. In your README you're listing some of them. How did you find them? I'm unsure if this "white-list" is enough so I'd like to look at others to expand that list but I don't know if there's a public list or something like that to go through. Any tips?