Photoshop and brushes

I'm a huge brush addict and have downloaded a ton of different brushes. What I find now, however, is that when I add a new set, the next time I open PS, it takes a full five minutes to load. So I end up going through the hundreds of brushes and picking out ones I don't particularly like one by one in order to delete them and get the number back down to one that allows PS to load in a minute or less. And now I'm not even adding brushes anymore for fear of making it hang again.

So I was wondering, is there really an upper limit on PS brushes beyond which the long-loading happens? Or is it just me who has this problem? And if it's an actual phenomenon and not my computer's fault, is there any way to fix that?

Thanks so much!