DATA
PREPARATION
WITH PYTHON
DATA PREPARATION
Data preparation (also called data preprocessing or
data cleaning) is the process of getting raw data ready for
analysis or machine learning. It is a crucial first step because
real-world data is often messy, incomplete, or inconsistent.
STEPS IN DATA PREPARATION
STEPS IN DATA
PREPARATION
1. Data Discovery
What it is: Understanding what data you have and where it
comes from.
Purpose: Identify available datasets, sources, and formats
(CSV, database, API).
Example: Finding sales data from multiple stores or
inventory logs.
2. DATA PROFILING
What it is: Examining data to understand its structure,
patterns, and quality.
Purpose: Detect anomalies, missing values, outliers, and
distributions.
Example: Checking the range of Quantity sold or average
Price.
3. DATA CLEANSING
What it is: Removing errors, inconsistencies, and irrelevant
data.
Purpose: Ensure data is accurate and usable.
Techniques:
Remove duplicates.
Fill or remove missing values.
Correct typos or inconsistent labels.
Example: Converting Region values like east → East.
4. DATA
TRANSFORMATION
What it is: Changing data into a usable format for analysis.
Purpose: Standardize and prepare data for modeling or
reporting.
Techniques:
Scaling numeric values.
Encoding categorical variables.
Aggregating or summarizing data.
Example: Converting Date strings into datetime objects,
creating Total = Quantity × Price.
5. DATA ENRICHMENT
What it is: Adding additional data to improve quality or
insight.
Purpose: Make data more useful for decision-making.
Example: Adding customer demographic info or regional
sales averages to your dataset.
6. DATA GOVERNANCE
What it is: Setting rules and policies for managing data.
Purpose: Ensure data integrity, security, and compliance.
Example: Defining who can access sales data, how long it is
retained, and data privacy rules.
7. DATA QUALITY
What it is: Ensuring the data is fit for purpose.
Dimensions of quality include:
• Accuracy
• Completeness
• Consistency
• Timeliness
• Validity
Purpose: Reliable data improves analysis and decision-making.
8. DATA VALIDATION
What it is: Checking that data meets certain rules or
constraints.
Purpose: Catch errors before analysis.
Example: Ensuring Quantity is ≥ 0, Price > 0, or Date is valid.
9. DATA
CONSISTENCY
What it is: Ensuring the same data is represented uniformly
across datasets.
Purpose: Avoid contradictions in reports.
Example: Region labeled as East in one table and EAST in
another → standardize to East.
10. DATA
COMPLETENESS
What it is: Ensuring all required data is present.
Purpose: Missing data can distort analysis.
Example: Every order should have OrderID, Date, Customer,
Quantity, and Price.
11. DATA ACCURACY
What it is: Ensuring data correctly represents reality.
Purpose: Reliable insights require accurate numbers.
Example: Total sales = Quantity × Price should match actual
sales records.
SALES DATA
PROBLEMS:
Missing Values
Quantity is missing for OrderID 1002 → cannot calculate sales.
Price is missing for OrderID 1003 → affects total calculation.
Date is missing for OrderID 1003 → difficult to track sales over time.
Duplicates
OrderID 1004 appears twice with the same details → could lead to double counting.
Data Type Issues
Quantity and Price may not be in numeric types (e.g., if read as strings from CSV).
Date may not be in proper datetime format → difficult for date operations.
PROBLEMS
Inconsistent or Inaccurate Data
Missing or zero values in numeric fields may not make sense for real sales.
Potential human errors in OrderID or Customer names.
No Calculated Fields
Total sales (Quantity × Price) is not present → needs to be derived.
Potential Formatting Issues
Extra spaces in Customer or Product names can cause mismatches.
Region names may have inconsistent capitalization (east vs East).
1. IMPORT LIBRARIES AND
LOAD DATA
import pandas as pd
# Load CSV
sales = pd.read_csv('[Link]')
# View first rows
print([Link]())
2. CHECK FOR MISSING VALUES
print([Link]().sum())
Output:
OrderID 0
Date 1
Customer 0
Product 0
Quantity 1
Price 1
Region 0
3. HANDLE MISSING
VALUES
# Fill numeric missing values with 0 or mean
sales['Quantity'].fillna(0, inplace=True)
sales['Price'].fillna(sales['Price'].mean(), inplace=True)
# Fill missing Date with a placeholder or drop
sales['Date'].fillna('2025-09-01', inplace=True)
4. REMOVE DUPLICATES
sales = sales.drop_duplicates()
[By default, it keeps the first occurrence and removes the
rest]
5. CORRECT DATA TYPES
sales['Date'] = pd.to_datetime(sales['Date'])
sales['Quantity'] = sales['Quantity'].astype(int)
sales['Price'] = sales['Price'].astype(float)
6. CREATE A NEW COLUMN FOR
TOTAL SALES
sales['Total'] = sales['Quantity'] * sales['Price']
7. CHECK CLEANED DATA
print(sales)
[Link]
Custom
OrderID Date Product Quantity Price Region Total
er
2025-
1001 Alice Pen 10 5.0 East 50.0
09-01
2025- Notebo
1002 Bob 0 15.0 West 0.0
09-02 ok
2025-
1003 Charlie Pencil 5 8.33 East 41.65
09-01
2025-
1004 Alice Pen 10 5.0 East 50.0
09-03
