how to delete columns in pyspark dataframe

Question

>>> aDataFrame[id: bigint, julian_date: string, user_id: bigint]>>> bDataFrame[id: bigint, quan_created_money: decimal(10,0), quan_created_cnt: bigint]>>> a.join(b, a.id==b.id, 'outer')DataFrame[id: bigint, julian_date: string, user_id: bigint, id: bigint, quan_created_money: decimal(10,0), quan_created_cnt: bigint] There are two ids: bigint and I want to delete one. How can I do?

Chris Dyer · Accepted Answer

For Spark 1.4+ , Pyspark drop column function on a dataframe in order to remove a column.

You can use it in two ways:

df.drop('a_column').collect()
df.drop(df.a_column).collect()

Also, to drop multiple columns at a time you can use the following:

columns_to_drop = ['a column', 'b column']
df = df.drop(*columns_to_drop)

how to delete columns in pyspark dataframe

About this question

Your answer

1 Answer

More Spark discussions

Learn & Explore

Popular Tutorials

Spark Interview Q&A

Latest Spark Blogs

Want answers to more Spark questions?

Stay in the loop — it’s free

You’re on the list!