Ask a Question
Ask Question Login
Corporate Training
  1. Community
  2. Tableau
  3. Question
Tableau

How to filter in NaN (pandas)?

Asked by Andrew Jenkins Jul 2, 2021 4.2K views 2 answers
Share

About this question

 I have a pandas dataframe (df), and I want to do something like:

newdf = df[(df.var1 == 'a') & (df.var2 == NaN)]

I've tried replacing NaN with np.NaN, or 'NaN' or 'nan' etc, but nothing evaluates to True. There's no pd.NaN.

I can use df.fillna(np.nan) before evaluating the above expression but that feels hackish and I wonder if it will interfere with other pandas operations that rely on being able to identify pandas-format NaN's later.

How pandas filter nan? Any advice is appreciated. Thank you.

Your answer

2 Answers

Ranjana Admin JanBask Expert Latest answer

Answered on May 16, 2024

In Pandas, you can filter out NaN (Not a Number) values from a DataFrame using the dropna() method or boolean indexing.


Here's how you can do it:

Using dropna() method:import pandas as pd
# Example DataFrame
df = pd.DataFrame({'A': [1, 2, None, 4],
                   'B': [5, None, 7, 8]})
# Drop rows containing NaN values
filtered_df = df.dropna()
print(filtered_df)

This will drop any row that contains at least one NaN value.

Using boolean indexing:

import pandas as pd
import numpy as np
# Example DataFrame
df = pd.DataFrame({'A': [1, 2, np.nan, 4],
                   'B': [5, np.nan, 7, 8]})
# Filter rows where NaN values are present in any column
filtered_df = df[~df.isnull().any(axis=1)]
print(filtered_df)

In this example, df.isnull().any(axis=1) returns a boolean Series indicating whether there are any NaN values in each row. Then, ~ is used to negate this Series, and it is used to filter out rows that contain any NaN values.

Both methods will result in a DataFrame with rows that do not contain any NaN values. Choose the method that best fits your workflow and preferences.









Was this helpful?

More Tableau discussions

Learn & Explore

Free tutorials and interview questions from industry experts — learn the skill, then get ready to prove it.

Latest Tableau Blogs

Guides, tips and career advice on Tableau from JanBask experts.