Pandas Tutorial · Cleaning Data

Cleaning Empty Cells

Learn all about Cleaning Empty Cells in this comprehensive tutorial.

5 min read intermediate

Cleaning Data

Cleaning Wrong Format

•Empty cells can potentially give you a wrong result when you analyze data.
•One way to deal with empty cells is to remove rows that contain empty cells.
•Another way of dealing with empty cells is to insert a new value instead.
•A common way to replace empty cells, is to calculate the mean, median or mode value of the column.

Empty Cells

Empty cells can potentially give you a wrong result when you analyze data.

Remove Rows

One way to deal with empty cells is to remove rows that contain empty cells.

This is usually OK, since data sets can be very big, and removing a few rows will not have a big impact on the result.

python

import pandas as pddf = pd.read_csv('data.csv')
new_df = df.dropna()print(new_df.to_string())

Note: Note: By default, the dropna() method returns a new DataFrame, and will not change the original.

If you want to change the original DataFrame, use the inplace = True argument:

python

import pandas as pddf = pd.read_csv('data.csv')df.dropna(inplace = True)
print(df.to_string())

Note: Note: Now, the dropna(inplace = True) will NOT return a new DataFrame, but it will remove all rows containing NULL values from the original DataFrame.

Replace Empty Values

Another way of dealing with empty cells is to insert a new value instead.

This way you do not have to delete entire rows just because of some empty cells.

The fillna() method allows us to replace empty cells with a value:

python

import pandas as pddf = pd.read_csv('data.csv')df.fillna(130, inplace = True)

The example above replaces all empty cells in the whole Data Frame.

To only replace empty values for one column, specify the column name for the DataFrame:

python

import pandas as pddf = pd.read_csv('data.csv')
  df.fillna({"Calories": 130}, inplace=True)

Replace Using Mean, Median, or Mode

A common way to replace empty cells, is to calculate the mean, median or mode value of the column.

Pandas uses the mean() median() and mode() methods to calculate the respective values for a specified column:

python

import pandas as pddf = pd.read_csv('data.csv')
val_1 = df["Calories"].mean()df.fillna({"Calories": val_1}, 
  inplace=True)

Note: Mean = the average value (the sum of all values divided by number of values).

python

import pandas as pddf = pd.read_csv('data.csv')
val_1 = df["Calories"].median()df.fillna({"Calories": val_1}, 
  inplace=True)

Note: Median = the value in the middle, after you have sorted all values ascending.

python

import pandas as pddf = pd.read_csv('data.csv')
val_1 = df["Calories"].mode()[0]df.fillna({"Calories": val_1}, 
  inplace=True)

Note: Mode = the value that appears most frequently.

Module quiz

2 questions

Which of the following is true about Cleaning Empty Cells?

It is strictly typedIt is evaluated at runtimeIt requires explicit memory managementIt is deprecated

What is the most common pitfall when working with Cleaning Empty Cells?

SyntaxErrorIndentationErrorNameErrorTypeError

Answer all questions to submit.

Cleaning Data

Cleaning Wrong Format

Cleaning Empty Cells

AI Summary

Empty Cells

Remove Rows

Replace Empty Values

Replace Using Mean, Median, or Mode

Module quiz