Ask a Question
Ask Question Login
Corporate Training
  1. Community
  2. Python
  3. Question
Python

PySpark: How to fill values in dataframe for specific columns?

Asked by Gayatri Jaiteley Jun 12, 2021 1.9K views 1 answer
Share

About this question

I have the following sample DataFrame:

a | b | c   |
1 | 2 | 4   |
0 | null | null|
null | 3 | 4   |

 

And I want to replace null values only in the first 2 columns - Column "a" and "b":

a | b | c   |
1 | 2 | 4   |
0 | 0 | null|
0 | 3 | 4   |
 

Here is the code to create sample dataframe:

rdd = sc.parallelize([(1,2,4), (0,None,None), (None,3,4)])
df2 = sqlContext.createDataFrame(rdd, ["a", "b", "c"])

 

I know how to replace all null values using:

df2 = df2.fillna(0)

 

And when I try this, I lose the third column:

df2 = df2.select(df2.columns[0:1]).fillna(0)

Your answer

1 Answer

More Python discussions

Learn & Explore

Free tutorials and interview questions from industry experts — learn the skill, then get ready to prove it.

Latest Python Blogs

Guides, tips and career advice on Python from JanBask experts.