How to perform mean or median imputation using the sklearn Python library?

by | Nov 16, 2022 | Data Preprocessing, Machine Learning Using Python, Python Scikit-learn

the machine learning model’s accuracy. One way to handle these missing values is to use the dropna() method to drop rows that contain missing values.

This way of handling missing values may work well if the dataset is large enough. But a better approach is to fill the missing values with appropriate statistical measures. The method of filling missing values of a dataset with appropriate statistical measures is known as data imputation.

In sklearn, we can use the SimpleImputer class from the sklearn.impute module to perform data imputation. For example, let’s say we want to fill the missing values in the age column of the dataset with the mean age. We can use the following Python code for that purpose:

import numpy
import seaborn
from sklearn.impute import SimpleImputer

df = seaborn.load_dataset("titanic")

imputer = SimpleImputer(missing_values=numpy.nan, strategy="mean")
df[["age"]] = imputer.fit_transform(df[["age"]])
print(df.isnull().mean()*100)

Here, we are first reading the dataset using the seaborn Python library. After that, we are using the SimpleImputer class for data imputation. The missing_values parameter indicates that we are replacing all numpy.nan values from the mentioned column of the dataset. The strategy parameter indicates the imputation strategy. Here, we are filling all the missing values with the mean value of the column. So, the strategy parameter is “mean.” The other strategies can be “median”, “most_frequent” or “constant” that will fill the missing values with the median value, most frequent value of the column or a constant value, respectively.

After that, we are calling the fit_transform() method. Please note that we are filling in the missing values of the age column of the dataset.

After the data imputation, we are printing the percentage of missing values in each column of the dataset. The output now shows:

survived        0.000000
pclass          0.000000
sex             0.000000
age             0.000000
sibsp           0.000000
parch           0.000000
fare            0.000000
embarked        0.224467
class           0.000000
who             0.000000
adult_male      0.000000
deck           77.216611
embark_town     0.224467
alive           0.000000
alone           0.000000
dtype: float64

As we can see all the missing values of the age column of the dataset have been filled and now, the percentage of missing values in the age column is zero.

Facebooktwitterredditpinterestlinkedinmail

Calculate the pseudoinverse of a matrix using Python

What is the pseudoinverse of a matrix? We know that if A is a square matrix with full rank, then A-1 is said to be the inverse of A if the following condition holds: $latex AA^{-1}=A^{-1}A=I $ The pseudoinverse or the Moore-Penrose inverse of a matrix is a...

Cholesky decomposition using Python

What is Cholesky decomposition? A square matrix A is said to have Cholesky decomposition if it can be written as a product of a lower triangular matrix and its conjugate transpose. $latex A=LL^{*} $ If all the entries of A are real numbers, then the conjugate...

Tensor Hadamard Product using Python

In one of our previous articles, we already discussed what the Hadamard product in linear algebra is. We discussed that if A and B are two matrices of size mxn, then the Hadamard product of A and B is another mxn matrix C such that: $latex H_{i,j}=A_{i,j} \times...

Perform tensor addition and subtraction using Python

We can use numpy nd-array to create a tensor in Python. We can use the following Python code to perform tensor addition and subtraction. import numpy A = numpy.random.randint(low=1, high=10, size=(3, 3, 3)) B = numpy.random.randint(low=1, high=10, size=(3, 3, 3)) C =...

How to create a tensor using Python?

What is a tensor? A tensor is a generalization of vectors and matrices. It is easily understood as a multidimensional array. For example, in machine learning, we can organize data in an m-way array and refer it as a data tensor. Data related to images, sounds, movies,...

How to combine NumPy arrays using horizontal stack?

We can use the hstack() function from the numpy module to combine two or more NumPy arrays horizontally. For example, we can use the following Python code to combine three NumPy arrays horizontally. import numpy A = numpy.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]) B =...

How to combine NumPy arrays using vertical stack?

Let’s say we have two or more NumPy arrays. We can combine these NumPy arrays vertically using the vstack() function from the numpy module. For example, we can use the following Python code to combine three NumPy arrays vertically. import numpy A = numpy.array([[1, 2,...

Singular Value Decomposition (SVD) using Python

What is Singular Value Decomposition (SVD)? Let A be an mxn rectangular matrix. Using Singular Value Decomposition (SVD), we can decompose the matrix A in the following way: $latex A_{m \times n}=U_{m \times m}S_{m \times n}V_{n \times n}^T $ Here, U is an mxm matrix....

Eigen decomposition of a square matrix using Python

Let A be a square matrix. Let’s say A has k eigenvalues λ1, λ2, ... λk. And the corresponding eigenvectors are X1, X2, ... Xk. $latex X_1=\begin{bmatrix} x_{11} \\ x_{21} \\ x_{31} \\ ... \\ x_{k1} \end{bmatrix} \\ X_2=\begin{bmatrix} x_{12} \\ x_{22} \\ x_{32} \\ ......

How to calculate eigenvalues and eigenvectors using Python?

In our previous article, we discussed what eigen values and eigenvectors of a square matrix are and how we can calculate the eigenvalues and eigenvectors of a square matrix mathematically. We discussed that if A is a square matrix, then $latex (A- \lambda I) \vec{u}=0...

Amrita Mitra

Author

Ms. Amrita Mitra is an author, who has authored the books “Cryptography And Public Key Infrastructure“, “Web Application Vulnerabilities And Prevention“, “A Guide To Cyber Security” and “Phishing: Detection, Analysis And Prevention“. She is also the founder of Asigosec Technologies, the company that owns The Security Buddy.

0 Comments

Submit a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Not a premium member yet?

Please follow the link below to buy The Security Buddy Premium Membership.

Featured Posts

Translate »