Data Preprocessing 7 - Box - Whisker Plot

A Box and Whisker plot, commonly known as a box plot, is a standardized way of displaying the distribution of data based on a five-number summary: minimum, first quartile (Q1), median, third quartile (Q3), and maximum. It's an excellent tool for visualizing the spread and skewness of the data, as well as identifying outliers.

Box plot Structure

Here's how a box plot is typically structured:

The Box: Represents the interquartile range (IQR), which is the range between the first quartile (Q1) and the third quartile (Q3). The box shows the middle 50% of the data.

The Median Line: A line inside the box shows the median (second quartile) of the dataset.

The Whiskers: Extend from the box to the highest and lowest values within 1.5 times the IQR from the quartiles. These lines show the range of the data.

Outliers: Data points outside the whiskers are often plotted individually as dots and considered outliers.

Outliers

point > Q3 + 1.5 * IQR
point < Q1 - 1.5 * IQR

Python Code

Identify Outlier

import seaborn as sns
import matplotlib.pyplot as plt

# Example data
data = [4, 8, 15, 16, 23, 42, 108]

# Creating the box plot
sns.boxplot(x=data)
plt.title('Box Plot')
plt.xlabel('Values')
plt.show()

Interpreting Spread and Skewness:

import numpy as np
import matplotlib.pyplot as plt

# Symmetrically distributed data (e.g., normal distribution)
symmetric_data = np.random.normal(loc=0, scale=1, size=1000)

# Asymmetrically distributed data (e.g., skewed distribution)
skewed_data = np.random.exponential(scale=1, size=1000)

plt.figure(figsize=(12, 6))

# Box plot for symmetrically distributed data
plt.subplot(1, 2, 1)
plt.boxplot(symmetric_data)
plt.title("Symmetrically Distributed Data")
plt.ylabel("Values")

# Box plot for asymmetrically distributed data
plt.subplot(1, 2, 2)
plt.boxplot(skewed_data)
plt.title("Asymmetrically Distributed Data")
plt.ylabel("Values")

plt.show()

Symmetrically Distributed Data:

Spread: The spread of the data can be seen from the range of the box (IQR) and whiskers. A larger IQR or longer whiskers indicate greater variability in the data.

Skewness: In a symmetric distribution, the median line in the box is centered, and the whiskers are roughly equal in length on both sides. The distribution of data points within the box and outside the box (whiskers) is roughly symmetric.

Asymmetrically Distributed Data:

Spread: Similar to the symmetric case, the spread is indicated by the IQR and whisker lengths.

Skewness: Skewness is evident if the median line within the box is off-center, and one set of whiskers is significantly longer than the other. In a right-skewed distribution, the right whisker will be longer, and the median will be closer to the left quartile. The opposite is true for a left-skewed distribution.

These box plots provide a visual summary of the central tendency, variability, and skewness of the data, allowing for an intuitive understanding of the dataset's characteristics.

Python Code

Data-Science/box whisker.ipynb at main · lovelynrose/Data-Science (github.com)

Comments

Post a Comment

Popular posts from this blog

Data Preprocessing 1 - Key Steps

  Data preprocessing is a critical step in the data analysis and machine learning pipeline. It involves transforming raw data into an understandable and usable format . Raw data, as it's collected, is often incomplete, inconsistent, or lacking in certain behaviors or trends, and may contain many errors. Data preprocessing is used to clean and organize the data so that it can be analyzed and used effectively.   Key steps in data preprocessing include: 1. Data Cleaning:  This involves handling missing data, smoothing noisy data, identifying or removing outliers, and resolving inconsistencies. Missing data can be filled in using data imputation techniques, noise can be smoothed out using various methods like binning, clustering, or regression, and outliers can be detected and dealt with appropriately. 2. Data Integration:  Combining data from different sources and ensuring that the data is consistent. For example, if you are combining data from different databases...
Read more

Data Preprocessing 2 - Data Imputation

Data imputation is a process used in statistics and data science to handle missing values in datasets . When a dataset has missing or null values, it can lead to inaccurate results during analysis or model training. To address this, data imputation techniques are used to estimate and fill in these missing values.   When to do it Use it only when 5-15% of data has this.  This is surely not right data.  Ensure statistical properties of data are maintained even after imputation. Consider the following column in a dataset: [5, nan, nan, 8, 10] The key approaches and methods in data imputation include: 1. Mean Imputation:  Replacing the missing values with the mean value of the entire feature (column). This is best used when the data distribution is normal and the missing values are not too many. Eg. Finding mean we get (5+8+10)/3 = 7.67 Result : [5, 7.67, 7.67, 8, 10] 2. Median Imputation:  Similar to mean imputation, but using the median value. This is more r...
Read more

Data Preprocessing 8 - Data Normalization with Solved Example

 Normalization techniques are used in data preprocessing to adjust the scale of data without distorting differences in the ranges of values . Here are several commonly used normalization techniques: 1. Min-Max Normalization:    - Rescales the feature to a fixed range, typically [0, 1].    - Formula: (x – min)/(max – min) where min and max are the minimum and maximum values in the feature, respectively.    - Suitable for cases where you need to bound values but don't have outliers. 2. Z-score Normalization (Standardization):    - Transforms the feature to have a mean of 0 and a standard deviation of 1.    - Formula: (x – μ )/ σ μ   is the mean and σ is the standard deviation.    - Particularly useful when the data follows a Gaussian distribution. Solved problem is in the blog on z-score. 3. Decimal Scaling:    - Moves the decimal point of values of the feature.    - The d...
Read more