Removing duplicates from rows based on specific columns in an RDD/Spark DataFrame
Pyspark dropDuplicates() method. Follow the way given below and use the same approach in your problem: >>> from pyspark.sql import Row >>> df = sc.parallelize([ ... Row(name='Amit', id=5, marks=80), ... Row(name='Amit', id=5, marks=80), ... Row(name='Amit', id=10, marks=80)]).toDF() >>> df.dropDuplicates().show()…
Answered May 6, 2021 · 2.5K Views Read answer →