Since we deal with a NumPy array, let's use numpy to solve this issue.

I think this causes the error. How about if cats.size != np.unique(cats).size:

Yep n_unique_categories = array([1, 1, 1, 1, 1]) is older NumPy version. I am not really sure why. The change that you propose is fine with me.

Unfortunately, this still causes error due to None, see the fails https://github.com/scikit-learn/scikit-learn/pull/27328/checks?check_run_id=18232777028.
For example, np.unique(np.array([None, 'a', 'z'], dtype=object)) will raise an error.

How about if cats.size != len(set(cats)):?

OK so let's revert to the set then. However, we need a test where we specify several time nan in the category as well to check this corner case.

I am going to use _unique function instead, see the code below.

import numpy as np
from sklearn.utils._encode import _unique

print(set(np.array(['a', None, None])))            # {'a', None}
print(set(np.array(['a', np.nan, np.nan])))        # {'a', 'nan'}
print(set(np.array([1., np.nan, np.nan])))         # {nan, 1.0, nan}

print(_unique(np.array(['a', None, None])))        # ['a' None]
print(_unique(np.array(['a', np.nan, np.nan])))    # ['a' 'nan']
print(_unique(np.array([1., np.nan, np.nan])))     # [ 1. nan]

For several nan in the category, I don't think it's necessary. Since now we assume nan must at the last, and PR #27309 will resolve this.

BTW, which way do you think is better? First check if nan is at the last or first check if category contain duplicated values?

Hi @thomasjpfan, would you like take a look at this PR and also #27309 ?

BTW, which way do you think is better? First check if nan is at the last or first check if category contain duplicated values?

Maybe the check for nan first since it is less expensive.

Sure, I will update code after #27309 is merged.