You can use the following methods to count duplicates in a pandas DataFrame:
Method 1: Count Duplicate Values in One Column
len(df['my_column'])-len(df['my_column'].drop_duplicates())
Method 2: Count Duplicate Rows
len(df)-len(df.drop_duplicates())
Method 3: Count Duplicates for Each Unique Row
df.groupby(df.columns.tolist(), as_index=False).size()
The following examples show how to use each method in practice with the following pandas DataFrame:
import pandas as pd #create DataFrame df = pd.DataFrame({'team': ['A', 'A', 'A', 'A', 'B', 'B', 'B', 'B'], 'position': ['G', 'G', 'G', 'F', 'G', 'G', 'F', 'F'], 'points': [5, 5, 8, 10, 5, 7, 10, 10]}) #view DataFrame print(df) team position points 0 A G 5 1 A G 5 2 A G 8 3 A F 10 4 B G 5 5 B G 7 6 B F 10 7 B F 10
Example 1: Count Duplicate Values in One Column
The following code shows how to count the number of duplicate values in the points column:
#count duplicate values in points column
len(df['points'])-len(df['points'].drop_duplicates())
4
We can see that there are 4 duplicate values in the points column.
Example 2: Count Duplicate Rows
The following code shows how to count the number of duplicate rows in the DataFrame:
#count number of duplicate rows
len(df)-len(df.drop_duplicates())
2
We can see that there are 2 duplicate rows in the DataFrame.
We can use the following syntax to view these 2 duplicate rows:
#display duplicated rows
df[df.duplicated()]
team position points
1 A G 5
7 B F 10
Example 3: Count Duplicates for Each Unique Row
The following code shows how to count the number of duplicates for each unique row in the DataFrame:
#display number of duplicates for each unique row
df.groupby(df.columns.tolist(), as_index=False).size()
team position points size
0 A F 10 1
1 A G 5 2
2 A G 8 1
3 B F 10 2
4 B G 5 1
5 B G 7 1
The size column displays the number of duplicates for each unique row.
Additional Resources
The following tutorials explain how to perform other common operations in pandas:
How to Drop Duplicate Rows in Pandas
How to Drop Duplicate Columns in Pandas
How to Select Columns by Index in Pandas
Hi Zach,
I am in an online class and we are covering Pandas and duplicate entries.
I appreciate your providing straight forward examples.
Your explanation of size may be better with number of occurances as the number of duplicates can be seen as the first occurance, plus 2 more.
With a size of 1, you say size is number of duplicates…
Thanks for your time.
Len
Hi Len…Thank you for your feedback! Keep us posted on your progress! We are always here to help if we can.