With the explanation regarding pos_label provided by @amueller in #17704 (comment)

My understanding is that pos_label is not the semantics of the problem, but the semantics of the predict_proba/decision function that you passed. If you have some classes, 'a' and 'b' and a decision function, then pos_label='a' means that high values of the decision function are supposed to correspond to the class 'a' being likely.

I think we should either remove pos_label from PrecisionRecallDisplay and RocCurveDisplay or rename it to positive_class. Naming the parameter differently was originally suggested in #15573