Ask a Question
Ask Question Login
Corporate Training
  1. Community
  2. Spark
  3. Question
Spark

Removing duplicates from rows based on specific columns in an RDD/Spark DataFrame

Asked by Smith Burgess May 6, 2021 2.5K views 1 answer
Share

About this question

Let's say I have a rather large dataset in the following form:
data = sc.parallelize([('Foo',41,'US',3),
('Foo',39,'UK',1),
('Bar',57,'CA',2),
('Bar',72,'CA',2),
('Baz',22,'US',6),
('Baz',36,'US',6)])

 What I would like to do is remove duplicate rows based on the values of the first,third and fourth columns only.

Removing entirely duplicate rows is straightforward:

data = data.distinct()

and either row 5 or row 6 will be removed

But how do I only remove duplicate rows based on columns 1, 3 and 4 only? i.e. remove either one one of these:

('Baz',22,'US',6)

('Baz',36,'US',6)

 In Python, this could be done by specifying columns with .drop_duplicates(). How can I achieve the same in Spark/Pyspark?

Your answer

1 Answer

More Spark discussions

Learn & Explore

Free tutorials and interview questions from industry experts — learn the skill, then get ready to prove it.

Latest Spark Blogs

Guides, tips and career advice on Spark from JanBask experts.