By default, the pandas Python library does not provide any direct method to remove duplicate columns. Pandas does have a method DataFrame.drop_duplicates() that can be used to remove duplicate rows. In this article, we will explain how we can use the same function to remove duplicate columns from a DataFrame.

import pandas

list1 = [[1, 2, 3, 2, 10], [4, 5, 6, 5, 12], [7, 8, 9, 8, 14]]
df = pandas.DataFrame(list1)
print("df before removing duplicate columns: \n", df)

df2 = df.T.drop_duplicates(keep="first", ignore_index=False).T
print("df2 after removing duplicate columns: \n", df2)

Now, df.T can be used to take a transpose of a DataFrame. Let’s say a DataFrame is like the following:

   0  1  2  3   4
0  1  2  3  2  10
1  4  5  6  5  12
2  7  8  9  8  14

So, the Python statement print(df.T) will print the following: