This is a big problem.

For some feeds the update/modify times are very important, but for some feeds the publish date is the deciding information. This clearly needs to be handled separately.

For example the youtube feeds update the modification time with each comment, so in this case the publish time is the one to go for.

On the other hand, news sites generally change the update/modify time when there was some significant change in the article, so in this case, the update/modify time is the important one.