0% found this document useful (0 votes)

5 views133 pages

Statistical Data Analysis Using Python

The document is a preface and introduction to the book 'Data Analysis with Python', which aims to equip readers with essential tools and techniques for data analysis using Python. It highlights Python's popularity due to its simplicity, versatility, and extensive libraries, making it suitable for various applications including data science, web development, and automation. The book is structured to guide both beginners and experienced practitioners through fundamental and advanced concepts of data analysis, emphasizing best practices and critical thinking skills.

Uploaded by

David Righetto

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

5 views133 pages

Statistical Data Analysis Using Python

Uploaded by

David Righetto

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

!"#$

% " &' ' ( ($

)+,$ # - ./#%0 1

( *23) ) !%4*+)* $ !
#

511 6 ! 7 %

5 *+*8 % " & ' ' ( ($

Preface

In an age where data drives decision-making across virtually every industry, the ability to
analyze and interpret this data has become an invaluable skill. From predicting market trends to
understanding customer behavior and even advancing scientific research, the applications of data
analysis are vast and varied. This book, Data Analysis with Python, is designed to equip you with
the essential tools and techniques needed to unlock the power of data using one of the most
versatile programming languages available today: Python.

Python has emerged as a leading language in the world of data analysis, thanks to its simplicity,
readability, and the extensive ecosystem of libraries and frameworks it supports. Its widespread
adoption across diverse fields—from finance and healthcare to technology and academia—attests
to its robustness and adaptability. However, mastering data analysis with Python is not just about
learning a language; it’s about understanding how to harness its capabilities to derive meaningful
insights from complex datasets.

This book is structured to guide both beginners and experienced practitioners through the
fundamental and advanced concepts of data analysis. We start with the basics, ensuring that
readers are comfortable with Python’s syntax and foundational data structures. We then
progressively delve into more complex topics, including data manipulation, data visualization,
and statistical analysis. Each chapter is designed to build on the previous one, with practical
examples and exercises that reinforce the concepts discussed.

In addition to the core content, we also address best practices in data analysis, including how to
handle missing data, manage data quality, and ensure reproducibility in your analysis. These
aspects are crucial for producing reliable and robust analyses that can stand up to scrutiny in
professional settings.

As you embark on this journey, remember that data analysis is as much an art as it is a science. It
requires curiosity, creativity, and a willingness to question and explore. Python is a powerful
tool, but it is your ability to think critically about data that will truly set you apart.

Thank you for choosing this book as your guide. We hope you find it to be a valuable resource in
your data analysis endeavors, and we look forward to seeing where your analytical skills will
take you.

Kindu Kebede Gebre(Assistant Professor)

Million Wesenu Demissie(Assistant Professor)

Ϯ
Contents
Preface ............................................................................................................................................ 2
CHAPTER ONE .............................................................................................................................. 4
1. Overview of Python .............................................................................................................. 4
1.1. Introduction to Python .................................................................................................. 4
1.2. Uses of Python .............................................................................................................. 6
1.3. Characteristics of Python ............................................................................................. 8
1.4. How to Install Python on Windows .............................................................................. 9
1.5. Python Development Environment ............................................................................ 12
1.6. Python Interfaces ........................................................................................................ 12
1.7. Getting Started with Python Programming ................................................................ 13
1.8. How to Create and Run Python Program .................................................................. 13
1.9. Python Basic Guide.................................................................................................... 14
1.10. Library in Python .................................................................................................... 14
1.11. How to install libraries in Python ........................................................................... 16
1.12. Help function in Python .......................................................................................... 17
1.13. How to Save Python Script ..................................................................................... 18
CHAPTER TWO............................................................................................................................ 19
2. Data Entry and Data management in Python .................................................................... 19
2.1. Data Entry .................................................................................................................. 19
2.2. How to Define Scalar Object in Python ..................................................................... 20
2.3. How to define Vectors in Python ................................................................................ 21
2.4. How to Create a Matrix in Python ............................................................................. 25
2.5. How to create a Data Frame in Python ..................................................................... 29
2.6. Importing Data ........................................................................................................... 32
2.7. Data Management ...................................................................................................... 33
CHAPTER THREE........................................................................................................................ 48
3. Basic Descriptive Statistics and Visualization................................................................... 48
3.1. Basic Descriptive Statistics ........................................................................................ 48
3.2. Data Visualization using Python ................................................................................ 55
3.3. Inferential Statistical Analysis in Python ................................................................... 71

ϯ
CHAPTER ONE

1. Overview of Python
[Link] to Python

Python has become one of the most popular programming languages in the world in recent years.
It's used in everything from machine learning to building websites and software testing. It can be
used by developers and non-developers alike. Python, one of the most popular programming
languages in the world, has created everything from Netflix’s recommendation algorithm to the
software that controls self-driving cars.

Python is an interpreter, object-oriented, high-level programming language with dynamic

semantics. Its high-level built in data structures, combined with dynamic typing and dynamic
binding; make it very attractive for rapid application development, as well as for use as a
scripting or glue language to connect existing components together. Python's simple, easy to
learn syntax emphasizes readability and therefore reduces the cost of program maintenance.
Python supports modules and packages, which encourages program modularity and code reuse.
The Python interpreter and the extensive standard library are available in source or binary form
without charge for all major platforms, and can be freely distributed.

Often, programmers fall in love with Python because of the increased productivity it provides.
Since there is no compilation step, the edit-test-debug cycle is incredibly fast. Debugging Python
programs is easy: a bug or bad input will never cause a segmentation fault. Instead, when the
interpreter discovers an error, it raises an exception. When the program doesn't catch the
exception, the interpreter prints a stack trace. A source level debugger allows inspection of local
and global variables, evaluation of arbitrary expressions, setting breakpoints, stepping through
the code a line at a time, and so on. The debugger is written in Python itself, testifying to
Python's introspective power. On the other hand, often the quickest way to debug a program is to
add a few print statements to the source: the fast edit-test-debug cycle makes this simple
approach very effective.

ϰ
As many government departments in the UK increase efforts to expand their digital economy,
professionals with skills in software development and computer programming are in high demand.
Because of its global popularity and ease of use, many novice programmers turn to Python to build
skills in various applications, including data science, software and web development, automation,
and improving the ease of everyday tasks.

Let’s take a closer look at what Python is, what it can do, and how you can start learning it. Python
is a computer programming language often used to build websites and software, automate tasks,
and analyze data. Python is a general-purpose language, used to create various programmers and
isn’t specialized for any specific problems. Thanks to its similarity with English syntax, Python is
widely considered to be one of the easiest programming languages to write, read, and learn. Many
professionals with little to no computer programming experience can pick up basic Python skills
and scale them to more advanced coding techniques.

Another reason Python is one of the most popular programming languages is that it is an open-
source language, meaning it is designed to be free, easily accessible, and modifiable by anyone.
There is an extensive library of publicly available Python codes, making it much more convenient
for coders to build and scale their own projects. This means that instead of starting from scratch on
each new project, you’re able to reference and modify existing code from previous public Python
projects.
Why Python so Popular?

Although there are several widely acknowledged flaws of this programming language, it is
considered one of the most popular and important languages worldwide. So what are the various
features of Python that have resulted in this huge importance of Python across the world? The list
mentioned below gives you the answer to this question.

¾ Easy to Learn and Use

¾ Mature and Supportive Python Community
¾ Support from Renowned Corporate Sponsors
¾ Hundreds of Python Libraries and Frameworks
¾ Versatility, Efficiency, Reliability, and Speed
¾ Big data, Machine Learning and Cloud Computing

ϱ
¾ First-choice Language
¾ The Flexibility of Python Language
¾ Use of python in academics
¾ Automation
¾ It is interpreted
¾ It is open-source

[Link] of Python

Python has become one of the most popular programming languages of the 21st century. It is
being used for multiple purposes in various sectors of business. Researchers use Python for
building applications and developing websites. Data Engineers use Python for performing data
analysis, statistical analysis, and building machine learning models.

Python is commonly used for developing websites and software, task automation, data analysis,
and data visualization. Since it’s relatively easy to learn, Python has been adopted by many non-
programmers such as accountants and scientists, for a variety of everyday tasks, like organizing
finances.

The common use of python in the real world:-

¾ Data analysis and machine learning

¾ Web development
¾ Automation or scripting
¾ Software testing and prototyping
¾ Everyday tasks

Data Analysis and Machine Learning

Python has become a staple in data science, allowing data analysts and other professionals to use
the language to conduct complex statistical calculations, create data visualizations, build machine
learning algorithms, manipulate and analyses data, and complete other data-related tasks. Python
can build various data visualizations, like line and bar graphs, pie charts, histograms, and 3D plots.

ϲ
Python also has many libraries that enable coders to write programmers for data analysis and
machine learning more quickly and efficiently, like TensorFlow and Keras.

The UK has recently launched a National Artificial Intelligence (AI) Strategy intending to expand
machine learning and artificial intelligence technology throughout UK operations. There are
several up-and-coming applications of machine learning, including health care innovations such as
genomics, automating business decisions, and improved public service efforts. As the UK pours
more resources into this field, learning Python and other machine learning skills has the potential
to open doors to expanding opportunities in the coming years.

Web development

Python is often used to develop the back end of a website or application—the parts that a user
doesn’t see. Python’s role in web development includes sending data to and from servers,
processing data and communicating with databases, routing URLs, and ensuring security. Python
offers several frameworks for web development. Commonly used ones include Django and Flask.
Some web development jobs that use Python include back-end engineers, full-stack engineers,
Python developers, software engineers, and DevOps engineers. According to the UK National
Careers Service, the ability to write computer programmes, build databases, and use Applications
Program Interfaces (APIs) are top skills in this field, all of which align closely with potential
applications of this popular language.

Automation or scripting

If you find yourself performing a task repeatedly, you could work more efficiently by automating it
with Python. Writing code used to build these automated processes is called scripting. In the
coding world, automation can be used to check for errors across multiple files, convert files,
execute simple maths, and remove duplicates in data. Automating simple tasks can save you an
abundance of time on repetitive to-dos, opening your time to build new knowledge and explore
more advanced applications of Python.

Software testing and prototyping

Python can aid in software development tasks like build control, bug tracking, and testing. With
Python, software developers can automate testing for new products or features.

Everyday tasks

Python isn't only for programmers and data scientists. Learning Python can open new possibilities
for those in less data-heavy professions, like journalists, small business owners, or social media

ϳ

marketers. Python can also enable non-programmers to simplify certain tasks in their lives. Here
are just a few of the tasks you could automate with Python:

Keep track of stock market or crypto prices

¾ Send yourself a text reminder to carry an umbrella anytime it’s raining

¾ Update your shopping list
¾ Rename large batches of files
¾ Convert text files to spreadsheets
¾ Randomly assign chores to family members
¾ Fill out online forms automatically

[Link] of Python

The ability of understand and use of coding skills continues to rise in importance in this world,
many people of all ages are looking for ways to increase their programming knowledge. Python is
a popular choice for several reasons when deciding which language to learn.

Here’s a summary of key features that make Python a top choice for coders of all levels.
x It has a simple syntax that mimics natural language, so it’s easier to read and understand. This
makes it quicker to build projects, and faster to improve on them.
x It’s versatile. Python can be used for many different tasks, from web development to machine
learning.
x It’s beginner friendly, making it popular for entry-level coders.
x It’s open source, which means it’s free to use and distribute, even for commercial purposes.
x Python’s archive of modules and libraries—bundles of code that third-party users have created to
expand Python’s capabilities—is vast and growing.
x Python has a large and active community that contributes to Python’s pool of modules and
libraries, and acts as a helpful resource for other programmers. The vast support community means
that if coders run into a stumbling block, finding a solution is relatively easy; somebody has likely
encountered the same problem before.
Simple and Easy to Learn: Python has a simple syntax, which makes it easy to learn and
read. It’s a great language for beginners who are new to programming.
Interpreted: Python is an interpreted language, which means that the Python code is executed
line by line. This makes it easy to test and debug code.

ϴ
High-Level: Python is a high-level language, which means that it abstracts away low-level
details like memory management and hardware interaction. This makes it easier to write and
understand code.
Dynamic Typing: Python is dynamically typed, which means that you don’t need to declare
the data type of a variable explicitly. Python will automatically infer the data type based on the
value assigned to the variable.
Strong Typing: Python is strongly typed, which means that the data type of a variable is
enforced at runtime. This helps prevent errors and makes the code more robust.
Extensive Standard Library: Python comes with a large standard library that provides tools
and modules for various tasks, such as file I/O, networking, and more. This makes it easy to
build complex applications without having to write everything from scratch.
Cross-Platform: Python is a cross-platform language, which means that Python code can run
on different operating systems without modification. This makes it easy to develop and deploy
Python applications on different platforms.
Community and Ecosystem: Python has a large and active community, which contributes to
its ecosystem. There are many third-party libraries and frameworks available for various
purposes, making Python a versatile language for many applications.
Versatile: Python is a versatile language that can be used for various purposes, including web
development, data science, artificial intelligence, game development, and more.

[Link] to Install Python on Windows

Python is usually not installed on Windows. However, you can check if it exists on the system by
running one line of command on the command prompt: python-version. To download Python,
you need to visit [Link], which is the official Python website.
Step 1: click on the official Python website ([Link])

ϵ
Step 2: Click on the Downloads tab and then select the Windows option.

This will take you to the page where the different Python releases for Windows can be found.

Step 3: select Windows x86-64 executable installer

Since I am using a 64bit system, I’ll select “Windows x86-64 executable installer”.

It does will start the installation process.

ϭϬ
If you want to save the installation file in a different location

Step 4: click on Customize installation; otherwise, continue with Install Now. Also, select the
checkbox at the bottom to Add Python 3.7 to PATH.

Once the installation is complete, the below pop-up box will appear: Setup was successful.

ϭϭ
Now that the installation is complete, you need to verify that everything is working fine.

1.5. Python Development Environment

An Integrated development environment makes coding easier using the popular choices
include PyCharm, Visual Studio Code, and Jupyter Notebook. Install one and set it up for
Python development or you can also use an online Python Integrated development
environment.

[Link] Interfaces

To create a better solution to this problem, we are going to use an important concept of object-
oriented programming called interfaces. In programming, an interface is a set of rules or
guidelines for a class or object to follow. It defines a contract for the class or object, outlining the
methods and properties that must be implemented. An interface does not provide an
implementation for the methods or properties, but instead, it defines a blueprint for classes or
objects to follow. It is commonly used to provide a common set of methods for different classes
or objects to implement, which can then be used interchangeably.

ϭϮ
In Python, interfaces are not a built-in feature like in some other programming languages such
as Java or C. However, you can use abstract base classes (ABCs) to achieve a similar effect. An
abstract base class is a class that cannot be instantiated and is used as a base class for one or
more derived classes. Classes that derive from an ABC are required to implement certain
methods or properties defined by the ABC.

Note: If you want to have a look at another example of using abstract classes in Python feel free
to check this text to see how they can be used to implement different sorting methods.

[Link] Started with Python Programming

Python is a high-level, interpreted, and general-purpose programming language. It was created

by Guido van Rossum and first released in 1991. Python emphasizes code readability and
simplicity, making it an excellent language for beginners and experienced developers. Getting
Started with Python is easy due to its straightforward syntax and extensive documentation,
which provides ample resources for learning and development.

[Link] to Create and Run Python Program

For the first program, we will try to print a very simple message “Hello World” in Python, the
code for which is given below:
Once you have Python installed, you can run the program by following these steps:
1. Open a text editor (e.g., Notepad on Windows, TextEdit on macOS, or any code editor like
VS Code, PyCharm, etc.).
2. Copy the code above and paste it into the text editor.
3. Save the file with a .py extension (e.g., hello_world.py).
4. Open a terminal or command prompt.
5. Navigate to the directory where you saved the file using the cd command (e.g., cd
path/to/your/directory).
6. Run the program by typing python hello_world.py and pressing Enter.
You should see the output “Hello, World!” printed in the terminal.

ϭϯ

1.9. Python Basic Guide

Python has a simple and readable syntax, making it an excellent language for beginners. Here
are some basics of Python syntax:
Comments

Comments in Python start with the # symbol and are used to explain code or make notes.
Comments are ignored by the Python interpreter.
#This is a comment
print("Hello, World!") # This is another comment

1.10. Library in Python

Many popular Python toolboxes/libraries:

• NumPy
• SciPy
• Pandas
• SciKit-Learn
Visualization libraries
• matplotlib
• Seaborn
NumPy:
Introduces objects for multidimensional arrays and matrices, as well as functions
that allow to easily performing advanced mathematical and statistical operations
on those objects.
provides vectorization of mathematical operations on arrays and matrices which
significantly improves the performance
many other python libraries are built on NumPy

NumPy is the most foundational package for numerical computing in Python. If you are
going to work on data analysis or machine learning projects, then having a solid
understanding of NumPy is nearly mandatory.

ϭϰ
Indeed, many other libraries, such as pandas and scikit-learn, use NumPy’s array objects as
the lingua franca for data exchange. One of the reasons as to why NumPy is so important for
numerical computations is because it is designed for efficiency with large arrays of data. The
reasons for this include:

9 It stores data internally in a continuous block of memory, independent of other in-built

Python objects.
9 It performs complex computations on entire arrays without the need for loops.

SciPy ¾ collection of algorithms for linear algebra, differential equations,

numerical integration, optimization, statistics and more
part of SciPy Stack
built on NumPy
Pandas ¾ it adds data structures and tools designed to work with table-like
data (similar to Series and Data Frames in R)
¾ It provides tools for data manipulation: reshaping, merging, sorting,
slicing, aggregation etc.
¾ allows handling missing data.

SciKit-Learn:

Provides machine learning algorithms: classification, regression, clustering,

model validation etc.
built on NumPy, SciPy and matplotlib

matplotlib:

python 2D plotting library which produces publication quality figures in a variety

of hardcopy formats
a set of functionalities similar to those of MATLAB
line plots, scatter plots, barcharts, histograms, pie charts etc.
relatively low-level; some effort needed to create advanced visualization

ϭϱ

Seaborn:

based on matplotlib
provides high level interface for drawing attractive statistical graphics
Similar (in style) to the popular ggplot2 library in R

1.11. How to install libraries in Python

Installing libraries in Python is a fundamental aspect of Python development, enabling

developers to extend the language's capabilities and streamline various tasks. In this
comprehensive blog, we explore the process of installing Python libraries across different
operating systems while emphasizing the significance of managing dependencies and sticking to
best practices.

Install Pandas on Windows

Python Pandas can be installed on Windows in two ways:

PIP is a package management system used to install and manage software

packages/libraries written in Python.
Example
Install Pandas library using pip
#pip install pandas

Loading library

A Python library contains things (objects, code) that is not part of the core Python language but
is nonetheless useful to some community of users. Libraries save us from re-inventing the wheel:
Once someone has created a library and made it available for download, we can use the contents
of the library in our own Python programs.
For example, to use the [Link] object in our programs, we must first import the
Pandas library into our environment.
Start by creating a new notebook in Jupyter and moving to the first cell:
#Import Python Libraries
import numpy as np
import scipy as sp

ϭϲ
import pandas as pd
import matplotlib as mpl
import seaborn as sns

It is now possible to directly reference [Link] in your scripts and Jupyter will know
what you talking about. Note, however, that programmers are lazy, and typing the word “pandas”
over and over again is seen as an unnecessary burden. As such, programmers typically alias the
library when they import it:
import pandas as pd
Here, “pd” is the alias for “pandas”. So we can write [Link] in my code rather
than [Link]. Not sure what I will do with all the time I save.

Finally, if we plan to use the data frame object frequently in your code, we can call it out by
name using the from keyword:

from pandas import DataFrame

Now we can type DataFrame rather than [Link].

1.12. Help function in Python

The Python help function is used to display the documentation of modules, functions, classes,
keywords, etc. It provides information about modules, classes, functions, and methods. It is a
useful tool for getting documentation and assistance on various aspects of Python. In Python,
the help () function is a built-in function that provides information about modules, classes, and
functions.
Examples

In this example, we are using help () without any object to access documentation in Python.
help()
help() function in Python Syntax

Syntax: help([object])
Parameters (Optional): Any object for which we want some help or the information. If the help
function is passed without an argument, then the interactive help utility starts up on the
console.

Let us check the documentation of the print function in the Python console.
help(print)

ϭϳ

This will display the docstring for the print() function, along with information about the
function's arguments and return values.
You can also use the help() function to get information about a module, for example math
module:
import math
help(math)

This will display the docstring for the math module, along with a list of all the functions and
constants available in the module.

1.13. How to Save Python Script

The contents of the Python window can be saved to a Python file or text file. Load or create a
Jupyter notebook. Go to File -> Save as and save the notebook in a different location / with a
different name.

ϭϴ
CHAPTER TWO

2. Data Entry and Data management in Python

[Link] Entry
Variables

Variables are used to store data. In Python, you don’t need to declare the data type of a
variable explicitly. Python will automatically infer the data type based on the value assigned to
the variable.

Data Types

Python supports various data types, including integers, floats, strings, lists, tuples, dictionaries,
and more.
9 Integers: Whole numbers without decimals.
9 Floats: Numbers with decimals.
9 Strings: Text enclosed in single or double quotes.
9 Lists: Ordered collections of items.
9 Tuples: Immutable collections of items.
9 Dictionaries: Key-value pairs.

Indentation

Python uses indentation to define blocks of code, such as loops and functions. Use four spaces
for indentation. Incorrect indentation can lead to syntax errors.

ϭϵ
Operators

Python supports various operators, including arithmetic, comparison, logical, and assignment
operators.
x Arithmetic operators: +, -, *, /, %, ** (exponentiation), // (floor division).
x Comparison operators: ==,!=, <, >, <=, >=.
x Logical operators: and, or, not.
x Assignment operators: =, +=, -=, *=, /=, %=, **=, //=.
x Bitwise operators: &, |, ^, ~, <<, >>.
x Strings: Strings can be enclosed in single or double quotes. You can use the + operator to
concatenate strings.
Control Flow

Python supports various control flow structures, such as if-else statements, loops, and more.

[Link] to Define Scalar Object in Python

A vector in a simple term can be considered as a single-dimensional array. With respect to

Python, a vector is a one-dimensional array of lists. It occupies the elements in a similar
manner as that of a Python list.
The >>> is called the prompt which is used to indicate where you are going to type commands
Suppose we define a Python object "X" and assign a value, say 10 to it.

ϮϬ
This is defined as:
>>> X=10
>>>Y=X+5
>>> Y
>>> z=(X+Y)/5
>>> z

[Link] to define Vectors in Python

A vector in a simple term can be considered as a single-dimensional array. With respect to

Python, a vector is a one-dimensional array of lists. It occupies the elements in a similar
manner as that of a Python list. Let us now understand the Creation of a vector in Python.
Python NumPy module is used to create a vector. We use [Link]() method to create a
one-dimensional array i.e. a vector.
NumPy is a Python library designed to work efficiently with arrays in Python. It is fast, simple
to learn, and efficient in storage. In NumPy, we may generate an n-dimensional array.

In python, vectors are built from components, which are ordinary numbers. A vector can be
considered as a list of numbers, and vector algebra as operations done on the numbers in the list.
In other words, a vector is the numpy 1-D array.

NumPy is a general-purpose array-processing package. It provides a high-performance

multidimensional array object, and tools for working with these arrays. It is the fundamental
package for scientific computing with Python. Numpy is basically used for creating array of n
dimensions.
Vector is built from components, which are ordinary numbers. We can think of a vector as a
list of numbers, and vector algebra as operations performed on the numbers in the list. In other
words vector is the numpy 1-D array. We use the [Link]() method to create a vector.
Syntax:
[Link](list)
Example 1: Horizontal Vector
import numpy as np
lst = [10,20,30,40,50]

vctr = [Link](lst)

Ϯϭ
vctr = [Link](lst)

print("Vector created from a list:")

print(vctr)

Example 2: Vertical Vector

import numpy as np

lst = [[2],
[4],
[6],
[10]]

vctr = [Link](lst)

Ϯϰ
[Link] to Create a Matrix in Python

A matrix is a table of numbers consisting of rows and columns. Each element has a unique index
that denotes the row and column in which it is located. For example, a 3x3 matrix will have 3
rows and 3 columns, and each element will have an index like (i, j), where i is the row number
and j is the column number.

With NumPy. NumPy is a library for mathematical calculations and data processing. It has an
Array class that can be used to create a matrix:

Example

import numpy as np

matrix = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

Note: If you need to perform standard mathematical operations with matrices, you should choose
NumPy. It is easier to use, and all operations are already implemented in the library.

Matrix Operations

Here is a list of basic manipulations:

Addition of Matrix

It is important to remember that when adding two matrices, their sizes must match. You can add
two matrices if their dimensions are the same. Each element of the resulting matrix will equal the
sum of the corresponding elements of the original matrices.

Here is an example of addition using nested lists and loops:

#Addition matrix using the add() method from the NumPy library:

Example

import numpy as np

matrix1 = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

matrix2 = [Link]([[9, 8, 7], [6, 5, 4], [3, 2, 1]])

result = [Link](matrix1, matrix2)

result

Ϯϱ

Subtraction of Matrix

In Python, you can perform matrix subtraction using the subtract () method from the NumPy
library. When subtracting, the dimensions must match. One matrix can be subtracted from
another if their dimensions are the same. Each element of the resulting matrix will equal the
difference of the corresponding elements of the original matrices.

Example: Subtraction using the subtract() method from the NumPy library

import numpy as np

matrix1 = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

matrix2 = [Link]([[9, 8, 7], [6, 5, 4], [3, 2, 1]])

result = [Link](matrix1, matrix2)

Multiplication by a number

In Python, multiplying a matrix by a number can be implemented using the dot() method from
the NumPy library. When multiplying a matrix by a number, each element is multiplied by that
number. Each element of the final matrix will be equal to the product of the corresponding
element of the original matrix by a number.

Example:

import numpy as np

matrix = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

scalar = 2

result = [Link](matrix, scalar)

print(result)

Note: You can also use the multiplication operation sign * instead of the dot() method:

import numpy as np

matrix = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

scalar = 2

result = matrix * scalar

Ϯϲ
Multiplication of two Matrices

This operation results in a new matrix with the number of rows of the first matrix and the number
of columns of the second matrix. In matrix multiplication, the number of columns of the first
matrix must match the number of rows of the second matrix. Each element of the new matrix is
the sum of the product of the row elements of the first matrix and the column elements of the
second matrix. Matrices can be multiplied if the number of columns of the first matrix equals the
number of rows of the second matrix. The result will be a new matrix with the size
corresponding to the number of rows of the first matrix and the number of columns of the second
matrix. We will describe this operation in more detail later.

NumPy uses the dot() method for matrix multiplication:

Example

import numpy as np

matrix1 = [Link]([[1, 2], [3, 4]])

matrix2 = [Link]([[5, 6], [7, 8]])

result = [Link](matrix1, matrix2)

print(result)

Note: The @ operation works similarly to the dot() method

import numpy as np

matrix1 = [Link]([[1, 2], [3, 4]])

matrix2 = [Link]([[5, 6], [7, 8]])

result = matrix1 @ matrix2

print(result)

Matrix Transpose

Transpose is an operation that turns the rows of the original matrix into the columns of the new
matrix, and the columns into the rows. In Python, you can perform a transpose using the T
property or the transpose() method from the NumPy library.

Ϯϳ

Transpose is an operation in which rows and columns are swapped. That is, the first column
becomes the first row, the second column becomes the second row, and so on. An example of
transpose using the T property:

import numpy as np

matrix = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

result = matrix.T

two-dimensional data container.

Pandas DataFrames empower users with an extensive array of functionalities. From the creation
of structured data using dictionaries or other data structures to employing robust indexing for
seamless data access, Pandas facilitates effortless data manipulation.

We can import the DataFrames from the external storage; these storages can be referred to as the
SQL Database, CSV file, and an Excel file. We can also use the lists, dictionary, and from a list
of dictionary, etc.

Example

# Here, we are importing the pandas library as pd

import pandas as pd

# Here, we are assigning the data of lists.

data = {'Name': ['Tom', 'Joseph', 'Krish', 'John'], 'Age': [20, 21, 19, 18]}

# Here, we are creating the DataFrame

Ϯϵ

df = [Link](data) # here, we are printing the dataframe

# Here, we are printing the output.

print(df) # here, we are printing the dataframe

Python is an object-orient programming language. You create an instance of an object by calling

its constructor, typically with some arguments.

Let’ us create a data frame called “data1” that contains data from a CSV file stored on my
computer in the “Data sets” folder. To do this, we call Panda’s read_csv method:

data1= pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\[Link]")

[Link]()# view data at the top

Note that the head(n) method can be called on the new Data Frame data1. Head simply prints out
the first n rows of the data frame so you can see if it imported okay.

Hint: Pointing to a file on your computer is always tricky when you don't have a pop-up file
system chooser like you do in most Windows applications. Some basic hints:

If practical, put the data file in the same folder as your Jupyter notebook. In the example above,
this is created a subfolder called "Data sets" and put the bank data there. This makes it easier for
you to remember (and type) the path to the data file.

If your data must reside somewhere else on your computer or network: Be very careful when
typing. Cut and paste the path from Windows Explorer if required.

Beware the slashes/backslashes. Unix-like operating systems seperate directory names with a
slash "/" whereas Microsoft Windows uses a backslash "\".

ϯϬ
In general, Python seperates folders with forward slashes. The backward slash is used in Python
as the 'escape' character. If you cut and paste from Windows Explorer, you must either change all
the backslashes to slashes or add the "r" prefix to your pathname to indicate a "raw" string of text
(Python will ingore the special meaning of backslashes). So the following should work for a
more complex file location:

pd.read_csv('C:/Users/Kindu/Desktop/Data sets/Data sets/[Link]") — all forward slashes

pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\[Link]")— mixture of slashes and

backslashes with the "r" prefix

The Pandas libraries supports importing from many different data sources, including CSV,
Excel, JSON, or even direct database access over a network.

Here we use read_excel instead of read_csv. It is the same data, but in a different file format on
my computer:

datxl = pd.read_excel("C:/Users/Kindu/Desktop/Data sets/Data sets/infant. xlsx")

[Link]()

Running head() after importing is a good idea just to make sure you imported what you think you
imported. If not, you will have to either:

edit your source file (Excel) to make sure it has the necessary tabular structure (with the first row
as headings, and so on)

add arguments to your read_... method to make sure you are starting at the correct row, reading
the correct spreadsheet, and so on.

There is a number of pandas commands to read other data formats:

pd.read_excel('[Link]',sheet_name='Sheet1', index_col=None, na_values=['NA'])

pd.read_stata('[Link]')

pd.read_sas('myfile.sas7bdat')

ϯϭ

As noted previously, objects in Python expose useful properties and methods. For example, we
can confirm the size of the data1 data frame with the shape property, which gives us the number
of rows (777) and columns (21).

[Link]

(777, 21)

We can use the describe() method to generate some summary statistics:

[Link]()

#List first 10 records

[Link](10)# view data at the top

[Link](10)# view the data at bottom level

#Check a particular column type

data1[‘CD4'].dtype

#Check types for all the columns

[Link]

[Link] Data

One of the most magical things about R is its ability to handle matrices of data using the data
frame object (or tibble in the Tidyverse). The pandas library gives Python much the same
functionality.

Recall that a data frame is an object consisting of rows and columns (just like a spreadsheet or
database table). In addition to storing the data, the data frame object also has useful properties
and methods that can be accessed through code. Thus, the first step in data analysis is load data
from a file into a data frame object.

import pandas

ϯϮ
It is now possible to directly reference [Link] in your scripts and Jupyter will know
what you talking about. Note, however, that programmers are lazy, and typing the word “pandas”
over and over again is seen as an unnecessary burden. As such, programmers typically alias the
library when they import it:

import pandas as pd

Here, “pd” is the alias for “pandas”. So we can write [Link] in my code rather
than [Link].

Reading data using pandas

#Read csv file

Let’ us create a data frame called “data1” that contains data from a CSV file stored on my
computer in the “Data sets” folder. To do this, we call Panda’s read_csv method:

data1= pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\[Link]")

There is a number of pandas commands to read other data formats:

pd.read_excel('[Link]',sheet_name='Sheet1', index_col=None, na_values=['NA'])

pd.read_stata('[Link]')

pd.read_sas('myfile.sas7bdat')

[Link] Management

Data management is a process associated with creating, changing, and deleting data, along with
organizing its storage. The main Pandas data structures are series and data-frames; here, we’ll
review how the Pandas library tools can help you in your data management and data analysis.

Data Extraction from data frame using iloc, range and len

data2 = [Link][range(0, len(data1), 1),[3,8,9,10]]# data extraction from data1 data frame

[Link]())#summary statistics for numeric variables

ϯϯ
First, notice the difference between a property like shape with no parentheses and
a method like describe() with parentheses for passing arguments.

[Link](include='object')# Character or categorical summary statistics

Note: using 'all' returns a summary of all columns with NaN where the statistic is inappropriate
for the datatype

[Link](include='all')

((data1["FNS"].value_counts(normalize=True))*100).round(2).astype(str) + ' %'# normalize is

return relative frequency

Hint: Press the Shift-Tab key while within the parantheses of a Python method for a list of the
possible arguments. Hit Shift-Tab twice to get more detailed help.

Second, by default, the describe() method only summarizes the numerical columns. Recall that
our data frame consists of nine columns. To see the issue, run the info() method:

Data Frame data types

data1. info()# it used to know the structure of data

ϯϰ
Here we see the 21 columns in the data frame listed by data type.

[Link]# number of dimensions

[Link]#list the row labels and column names

Selecting a column in a Data Frame

#Method 1: Subset the data frame using column name:

data1['sex']

#Method 2: Use the column name as an attribute:

[Link]

#Method 3: Subset the data frame using int location that range from 0 to n and value increase
by 1

data2 = [Link][range(0, len(data1), 1),[3,8,9,10]]# data extraction from data frame data1,0
and 1 means data interval,,,0,1,2,17,20 means col number

data2

Data Frames “groupby” method

#Group data1 using “MC”

data1_MC = [Link]([‘MC'])

#Calculate mean value for each numeric column per each group
ϯϱ
data1_MC.mean(‘CD4’)

#Calculate mean time for each MC group:

[Link](‘MC')[[‘time']].mean()

Note: If single brackets are used to specify the column (e.g. time), then the output is Pandas
Series object. When double brackets are used the output is a Data Frame

Data Frame: filtering

To subset the data we can apply Boolean indexing. This indexing is commonly known as a filter.
For example if we want to subset the rows in which the CD4 value is greater than 500:

# subset the rows in which the CD4 value is greater than 500:

df_sub = data1[ data1[‘CD4'] > 500 ]

Any Boolean operator can be used to subset the data:

> greater; >= greater or equal;

< less; <= less or equal;

== equal; != not equal;

ϯϲ
#Select only those rows that contain female patients:

df_f = data1[ data1['sex'] == 'F' ]

To illustrate, consider the following expression:

data1['sex'] == 'F'# it used to select variable sex from data 1

The expression tests whether each value of the sex column is equal to the string “Female”. As in
R and other languages, the logical comparison operator is double equals (“==”). The result of the
expression is a vector of trues and falses corresponding to whether each of the 777 values of sex
is equal to “Female”.

Now, once we have the vector of 777 values of true or false (the Boolean vector), we can apply
that to the original data frame. If the first value in the Boolean vector is true, the first row of the
data frame is returned; if the first value is false, the row is skipped.

This is what it looks like:

data1[data1['sex'] == "F"]

ϯϳ
If you find this notation too cumbersome, you can assign the results to a separate data frame that
contains only the 353 female patients:

The Python type() method is used to make sure the result is a Pandas data frame. It is, so we can
calculate the mean salary for female employees:

Python has some basic built-in functions that can be applied to the core data types, such as
integers, floating point numbers, and so on. For example, if I want to take the result
of mean() and round it to two decimals, I can wrap the whole expression inside
the round() function:

ϯϴ
Complex filtering criteria

Boolean vectors can be created by combining conditions with & (and) and | (or). The only trick
is that each condition must be in parentheses:

The vector can then be applied to the whole data set to filter the data frame to female patients
with measurement 6 months. Rather than listing the results, we call the shape property to confirm
that only 71 patients are included in the resulting vector.

Filtering by a list

It is not uncommon when you have categorical data to need to filter or recode on a specific list of
values. To reuse the example used previously, assume we want to create a list of managerial
employees. The easiest way to do this is to use a greater-than condition.

data1[data1['MC'] >= 12]

ϯϵ
Data Frames Slicing

There are a number of ways to subset the Data Frame:

one or more columns

one or more rows

a subset of rows and columns

Rows and columns can be selected by their position or label

When selecting one column, it is possible to use single set of brackets, but the resulting object
will be a Series (not a DataFrame):

# Select column CD4:

data1[‘CD4']

When we need to select more than one column and/or make the output to be a DataFrame, we
should use double brackets:

#Select column time and CD4:

data1[[‘time',‘CD4']]

Data Frames selecting rows

If we need to select a range of rows, we can specify the range using ":"

#Select rows by their position:

data1[0:10]

Notice that the first row has a position 0, and the last value in the range is omitted:So for 0:10
range the first 10 rows are returned with the positions starting with 0 and ending with 9

Data Frames: method “loc”

ϰϬ
If we need to select a range of rows, using their labels we can use method loc:

#Select rows by their labels:

[Link][10:20,['time','sex','CD4']]

Data Frames method “iloc”

If we need to select a range of rows and/or columns, using their positions we can use method
iloc:

#Select rows by their labels:

[Link][10:20,[0, 3, 4, 5]]

Data Frames method” iloc” (summary

[Link][0] # First row of a data frame

[Link][i] #(i+1)th row

[Link][-1] # Last row

[Link][:, 0] # First column

[Link][:, -1] # Last column

[Link][0:7] #First 7 rows

[Link][:,0:2] #First 2 columns

[Link][1:3,0:2] #Second through third rows and first 2 columns

[Link][[0,5], [1,3]] #1st and 6th rows and 2nd and 4th columns

Data Frames: Sorting

We can sort the data by a value in the column. By default the sorting will occur in ascending
order and a new data frame is return.

# Create a new data frame from the original sorted by the column CD4

ϰϭ

data1_sorted = data1.sort_values( by =‘time')

df_sorted.head()

We can sort the data using 2 or more columns:

data1_sorted = data1.sort_values( by =[‘MC', ‘CD4'], ascending = True)

data1_sorted.head(10)

Missing Values

Missing values are marked as NaN

# Read a dataset with missing values

data1= pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\[Link]")

# Select the rows that have at least one missing value

Example

data1[[Link]().any(axis=1)].head()

There are a number of methods to deal with missing values in the data frame:

[Link]() description

dropna()# Drop missing observations

dropna(how='all')# Drop observations where all cells is NA

ϰϮ
dropna(axis=1, how='all')# Drop column if all the values are missing

dropna(thresh = 5)# Drop rows that contain less than 5 non-missing values

fillna(0)# Replace missing values with zeros

isnull()# returns True if the value is missing

notnull()# Returns True for non-missing values

Aggregation Functions

Aggregation - computing a summary statistic about each group, i.e.

compute group sums or means

compute group sizes/counts

Common aggregation functions:

min, max

count, sum, prod

mean, median, mode, mad

std, var

Note: agg() method are useful when multiple statistics are computed per column:

Example

data1[[‘time',‘CD4']].agg(['min','mean','max'])

ϰϯ
Basic Descriptive Statistics

[Link]() description

describe: Basic statistics (count, mean, std, min, quantiles, max)

min, max: Minimum and maximum values

mean, median, mode: Arithmetic average, median and mode

var, std :Variance and standard deviation

sem: Standard error of mean

skew: Sample skewness

kurt: kurtosis

Converting strings to categories

Recall in R that we made a distinction between strings (character data) like a person’s name
and factors, which look like strings but have a limited set of repeated values. Color is a factor,
for example, because it takes on a few known values like “red”, “green”, and “blue”. Similarly, a
column called car_brand would be a factor because it would contain a finite number of string
values, such as “Ford”, “Tesla”, “Ferarri”.

Pandas permit the same distinction, but instead use the term category instead of factor. This is
telling: R uses the old statistical technical term “factor” whereas Pandas/Python uses the more
straightforward term “category”. This is the difference between the two languages in a nutshell.

The process for replacing the two (string) “Object” columns with categories is similar to the one
we used in R. The key understands how to reference columns in Python. Two possibilities:

square bracket notation: data1['sex'] # list data values with each variables

dot notation: [Link]

ϰϰ
Of these two, square bracket notation is slightly more flexible because it permits column names
with spaces, e.g., dataframe['column name']. The dot notation of this would fail because Python
has no way of knowing what the space after “column” means: [Link] name.

Once we know how to reference a column (or a “Series” in Pandas-speak), we can run the type
conversion method and specify “category” as the output data type.

data1['sex'].astype('category') # it used to change character or string data to factor

Note at the bottom of the summary that the Sex category consists of two values: Female and
[Link] course, running the command above just lists the converted values; it does not do
anything to the “data1” data frame. To replace the existing column in the data frame, we use the
assignment operator.

data1['sex'] = data1['sex'].astype('category')

data1['FNS'] = bank['FNS'].astype('category')

We can confirm the result by running the describe() method again, but this time using an
argument to ensure we get only the categorical columns.

[Link](include='category')

ϰϱ
This output tells us that we have 777 rows of data. Both the “sex” and “FNS” columns have two
unique categorical values and the most frequent are “M” and “A” respectively. 424 of the 777
employees are female while 370/777 do not have FNS.

Type conversion using programming constructs

As an aside, it is possible to do the same type conversions en masse in Python using core
programming constructs such as lists, iterators, and looping. Specifically, for a larger data set
with many columns, it might make sense to do the following:

make a list of the columns to convert

iterate through each column making the conversion assignment, as above

loop through the values of the variable col until all the columns are converted

Use of an iterated list is not really worth it in this case (two lines of code using either approach):

for col in ['sex', 'FNS']:

datal[col] = datal[col].astype('category')

[Link]()

RangeIndex: 208 entries, 0 to 207

Data columns (total 9 columns):

# Column Non-Null Count Dtype

ϰϲ
--- ------ -------------- -----

0 Employee 208 non-null int64

1 EducLev 208 non-null int64

2 JobGrade 208 non-null int64

3 YrHired 208 non-null int64

4 YrBorn 208 non-null int64

5 Gender 208 non-null category

6 YrsPrior 208 non-null int64

7 PCJob 208 non-null category

8 Salary 208 non-null float64

dtypes: category(2), float64(1), int64(6)

memory usage: 12.1 KB

ϰϳ

CHAPTER THREE

3. Basic Descriptive Statistics and Visualization

Descriptive statistics refers to a set of methods used to summarize and describe the main features
of a dataset, such as its central tendency, variability, and distribution. These methods provide an
overview of the data and help identify patterns and relationships. Examples include measures of
central tendency, such as mean, median, and mode, which provide information about the typical
value in the dataset. Measures of variability, such as range, variance, and standard deviation,
describe the spread or dispersion of the data. Descriptive statistics can also include graphical
methods, including histograms, box plots, and scatter plots, to visually represent the data.

In this session we show how Python and its graphics libraries can be used to create the two most
common types of distributional plots: histograms and boxplots. This is including the data import
and library import commands at the start of each lesson so that the lessons are self-contained.
Exploring your data is a crucial step in data analysis. It involves:

Organising the data set

Maybe producing some numerical summaries; central tendency and spread, etc.

Plotting aspects of the data set

import pandas as pd

data1=pd.read_csv('C:/Users/Kindu/Desktop/Data sets/Data sets/[Link]')

[Link] Descriptive Statistics

This section will be covering Descriptive Statistics with the help of the Python Pandas Library.
Pandas provide basic descriptive statistic functions as methods of the Series object. Recall that
each DataFrame object consists of multiple Series (columns). Thus, the average age for HIV
patients can be found as:

Descriptive Statistics for Numeric Data with Python

# Importing Pandas Library

import pandas as pd

ϰϴ
Read csv file

Let’ us create a data frame called “data1” that contains data from a CSV file stored on my
computer in the “Data sets” folder. To do this, we call Panda’s read_csv method:

data1= pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\[Link]")

# Mean the Dataset

mean_var = [Link]()

print(mean_var)

# Median the Dataset

median_var = [Link]()

print(median_var)

# Mode the Dataset

mode_var = [Link]()

print(mode_var)

#Count the Number of Values in variables

var_count = [Link]()

print(var_count)

#Range of Variables

Var_range = [Link]() -[Link]()

print(Var_range)

# Quartile of Dataset

Q1 = [Link](0.25)

ϰϵ

Q2 = [Link](0.50)

Q3 = [Link](0.75)

print(Q1)

print(Q2)

print(Q3)

# Interquartile Range

Q1 = [Link](0.25)

Q3 = [Link](0.75)

iqr = Q3 - Q1

print(iqr)

#Variance of variables

CD4_var = [Link]()

print(CD4_var)

#Standard Deviation of variables

std_CD4 = [Link]()

print(std_CD4)

Other functions

abs(): The absolute value of the values.

prod(): Product of values.

sum(): The sum of the values.

cumsum(): Cumulative sum of values.

ϱϬ
cumprod(): Cumulative product of values.

[Link](n): selection of random n sample of respondents from population

[Link](4)

Numerical Summary Statistics

All of the aforementioned functions find one descriptive statistic at a time. But if we want a
simple way to see all this information at once, there's also a function for that: .describe()

We can use the describe() method to generate some summary statistics:

data1[‘’CD4"].describe()#To get the descriptive statistics for a specific column in your

DataFrame

[Link]() # Summery Statistics for numeric variables only

[Link](include="all")#To get the descriptive statistics for an entire DataFrame

Second, by default, the describe () method only summarizes the numerical columns. Recall that
our data frame consists of 21columns. To see the issue, run the info() method: data1. info()# it
used to know the structure of data

If you want to find all descriptive statistics for a single column at once, .describe() can do that,
too. With only one column, the results are returned as a list.

data1[[“CD4", “time"]].describe()# use double bracket when the variables are more than two

data1[["CD4", "time","rt"]].describe()# use double bracket when the variables are more than 2

What is the average CD4 for male versus female HIV Patients?

data1[[“sex", “CD4"]].groupby(“sex").mean()# the average CD4 for male versus female HIV
Patients

What is the mean CD4 level of HIV patients for each of the “sex” and “FNS” class
combinations?

ϱϭ

[Link]([“sex", “FNS"])[“CD4"].mean()# the mean CD4 level of HIV patients for each of
the “sex” and “FNS” class combinations

data2 = [Link][range(0, len(data1), 1),[3,8,9,10]]# data extraction from data1 data frame

[Link]()

Or, recall, we can get statistical summary of all numerical columns using the describe() method:

[Link]()

Printing the lists of column in data frame

Summary from subset of data frame

ϱϮ
Similarly, using a variable to save some typing:

Descriptive Statistics for Categorical Data with Python

We can use the describe() method to generate some Categorical summary statistics:

data1[“sex"].describe()#To get the descriptive statistics for the “sex” column, which contains
categorical data

[Link](include='object')# To get the descriptive statistics of categorical summary

statistics

((data1["FNS"].value_counts(normalize=True))*100).round(2).astype(str) + ' %'# normalize is

return relative frequency

((data1[["FNS","sex"]].value_counts(normalize=True))*100).round(2).astype(str) + ' %'#

normalize is return relative frequency of FNS with Sex

Creating a contingency table

Pandas have a very simple contingency table feature. Below, we specify the two variables of
interest (Sex and FNS) and set margins=True so we get marginal totals (“All”).

data1=pd.read_csv(r"C:\Users\Kindu\Desktop\SPSS Traning Materials\Data sets\Data

sets\[Link]")

contab_freq = [Link](

ϱϯ
data1['sex'],

data1['FNS'],

margins = True

contab_freq

Showing row percentages

Typically, showing frequencies is less useful than relative frequencies. Here, we have interested
in the row percentages: what is the probability that a female is a functional status of HIV patients
versus the probability a male is a functional status of HIV patients.

We can get relative frequencies using the normalize argument. If normalize = True, then we get
the relative frequency in each cell relative to the total number of patients. This is not very useful.
What we want instead is to normalize by row. The parameter for this is: normalize = 'index'.
Why “index” instead of “row”? Because each row has a row number (or index).

conttab_relfreq = [Link](

data1['sex'],

data1['FNS'],

ϱϰ
margins = True

normalize='index'

conttab_relfreq

Here, each row sums to 100%. Thus, for the total set of female patients, 48.7% are Ambulatory,
3.7% are Bedridden and 47.6% are working functional status. For males, 46.7% are Ambulatory,
7.3% are Bedridden and 46% are working functional status.

[Link] Visualization using Python

Data Visualization is the art of presenting complex and unorganized data into a form that is
easier to comprehend, visually appealing, and actionable. It is used for presentation of data
makes it easy to understand the patterns, trends, and outliers in the data.

The process of finding trends and correlations in our data by representing it pictorially is
called Data Visualization. Python data visualization has become a powerhouse in the world of
data analytics. It offers efficiency and flexibility in a world continuously driven by data.

In our modern digital world, we have huge amounts of data all around us. Data scientists and ML
engineers get most of the data they deal with data in a structured or unstructured data format,
however, it’s difficult for humans to understand and analyze this. Data visualizations (or
graphical representations of data) are vital for understanding the data. They help users explore
data through visual elements like charts, graphs, plots, maps, and other visualizations.

ϱϱ
There are a lot of libraries that can be used in Python to visualize data. Some of the most
common are matplotlib, seaborn, plotly, GGPlot, Bokeh, Geoplotlib, and PyGal. Here, we will
mostly be focussing on the matplotlib, seaborn, and plotly for our visualizations.

First we import the Python packages we are going to use.

Then we use Pandas to load in the dataset as a data frame.

import seaborn as sns# loading seaborn library

import pandas as pd#Create an alias with the as keyword while importing: import pandas as pd

import numpy as np# it used to create multiple arrays and perform operations

import [Link] as plt# it used to creates a figure, creates a plotting area in a figure,
plots some lines in a plotting area, decorates the plot with labels.

# Read a dataset with missing values

data1= pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\[Link]")

[Link]()

Let’s now get into the details of different types of visualizations and when and how should we
use them:

Line Chart

It is one of the most common charts that is used to observe a single or multiple variables with the
change of another variable. Basically, it is used in trend analysis and time series analysis. It is
always a best practice to use line charts with continuous data.

Importing the libraries

The libraries we would need for the graphs we create in this section.

import numpy as np

import pandas as pd

ϱϲ
import [Link] as plt

import seaborn as sns

import plotly as py

import [Link] as px

import plotly.graph_objects as go

Getting the Data using matplotlib

data1=pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\[Link]")

ϱϳ

import [Link] as plt

days = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]

steps_walked = [8934, 14902, 3409, 25672, 12300, 2023, 6890]

steps_last_week = [9788, 8710, 5308, 17630, 21309, 4002, 5223]

fig, ax = [Link]()

[Link](days, steps_walked, "o-g")

[Link](days, steps_last_week, "v--m")

ax.set_title("Step count | This week and last week")

ax.set_xlabel("Days of the week")

ax.set_ylabel("Steps walked")

ϱϴ
[Link](True)

[Link](["This week", "Last week"])

[Link]()

2. Plotting a Histogram in Python

It is often used to indicate the major features of the distribution of the data in a convenient form.
g=[Link](data1.CD4,histtype=‘bar’,ec=‘black’,)
g=[Link](‘CD4Level’)
g=[Link](‘frequency’)
[Link]()

ϱϵ

Here, I specified where the edges of the bars of the histogram are; the bin edges.

bin_edges=[0,10,20,30,40,50,60,70,80,90,100,110,120,130,140,150]

g=[Link](data.CD4,histtype=‘bar’,ec=‘black’,bins=bin_edges)
g=[Link](‘CD4Level’)
g=[Link](‘frequency’)
[Link]()

You could also specify the number of bins, and Matplotlib will automatically generate a number
of evenly spaced bins.

g=[Link](data.CD4,histtype=‘bar’,ec=‘black’, bins=15)
g=[Link](‘CD4 Level’)

g=[Link](‘frequency’)
[Link]()

ϲϬ
Histograms in Seaborn

Matplotlib is a powerful, but sometimes unwieldy, Python library.

Seaborn provides a high-level interface to Matplotlib and makes it easier to produce graphs like
the one on the right.

One of Seaborn's greatest strengths is its diversity of plotting functions.

Most plots can be created with one line of code.

Two graphics libraries are in common use in Python: Matplotlib and Seaborn. Seaborn is an
extension of Matplotlib that addresses a few specific graphics challenges, including histograms
and boxplots. As such, we will restrict our attention here to Seaborn.

Allow you to plot the distributions of numeric variables.

sns.set_style()

[Link](data1.CD4)

ϲϭ
Histograms are created in Seaborn using the histplot() (histogram plot) method. The syntax of
Seaborn is closer to R than Python. For example, the plot is called on a Seaborn library object
(sns) and passed a data frame as an argument.

[Link](x=data1['age '])# it used to plot histogram for variable “Age”

A few things to notice about this output

The histplot() method returns an Axes Subplot value. Since we don’t need this (or even know
what it is), we can clean-up our output in ending each Seaborn (or Matplotlib) call with a
semicolon.

Seaborn guesses at a good number of bins. It appears to be more than the default in R. But recall
that the point of a histogram is to get a rough sense of the shape of the distribution of the
variable. We can certainly change the number of bins (to say 10 or 12), but it is not critical.

We can pass some arguments to the method to get a more elaborate histogram. Turning on the
kernel density estimate (kde=True) gives us a smoothed “kernel density” line, like in SAS EG.

[Link](x=data1['age '], bins=10, kde=True);

ϲϮ
Of course, it is possible to change colors, and so on. We have split the more detailed method call
below over multiple lines, which is more readable and more with keeping with R-style coding.

[Link](x=data1['age '],bins=10, kde=False, stat="probability",color='green');

Creating a Boxplot

Creating a boxplot in Seaborn is very simple:

[Link](x=data1['age ']);

ϲϯ
If you prefer a vertical orientation, you can plot your data as the y variable instead of
the x variable, as done above. Also, notice that Seaborn does not provide an indicator of the
mean by default. Obviously, skewed data such as this pulls the mean away from the median. We
like to eyeball the difference between the two measures.

[Link](y=data1['age '], color='lightgreen', showmeans=True);

Seaborn has a number of themes you can use to alter the appearance of plots.

For example, we can use “whitegrid” to add grid lines to our boxplot.

[Link](data1.CD4)

Sns.set_style(‘whitegrid’)
ϲϰ
Creating a Scatter Plot

A scatter plot identifies a possible relationship between changes observed in two different sets of
variables. It provides a visual and statistical means to test the strength of a relationship between
two variables.

Seaborn doesn't have a dedicated scatter plot function.

We used Seaborn's function for fitting and plotting a regression line; hence lmplot()

However, Seaborn makes it easy to alter plots.

To remove the regression line, we use the fit_reg=False command

ϲϱ
[Link](x='CD4',y='time',fit_reg=False,data=data1)# scatter plot

Another useful function in Seaborn is the hue function, which enables us to use a variable to
colour code our data points. The “hue” function:-

[Link](x='CD4',y='time',data=data1,fit_reg=False,hue='sex')

ϲϲ
Factor plots
Factor Plot is used to draw different types of categorical plot. The default plot that is shown is a
point plot, but we can plot other seaborn categorical plots by using of kind parameter, like box
plots, violin plots, bar plots, or strip plots.

Make it easy to separate plots by categorical classes.

g=[Link](x="time",

y="CD4",

data=data1,

hue="sex",

col="sex",

kind="strip")

g.set_xticklabels(rotation=-45)

Note: You must specify the value of the kind argument since in catplot its default value is
“point" instead of " strip“, “violin”, “bar”,”box” and “count”

Plotting all data: Empirical cumulative distribution functions (ECDFs)

ϲϳ
An alternative way of visualising a distribution of a variable in a large dataset is to use an ECDF.
Here we have an ECDF that shows the percentages of different CD4 levels of patients. An x-
value of an ECDF is the quantity you are measuring; i.e. CD4 level. The y-value is the fraction of
data points that have a value smaller than the corresponding x-value.

x=[Link](data1[‘CD4’])

y=[Link](1,len(x)+1)/len(x)

g=[Link](x,y,marker=‘.’,linestyle=‘none’)

g=[Link](‘CD4’)

g=[Link](‘ECDF’)

[Link](0.02)

[Link]()

ϲϴ
Bar plot

It is used to visualise the distributions of categorical variables.

[Link](x=FNS,data=data1,palette=type_colors)

[Link](rotation=-45)

Example2

[Link]([Link],data1.CD4, color ='maroon', width = 0.4)

[Link]('functional Status')

[Link]('frequency')

[Link]('Functional status of HIV patients')

[Link]()

[Link](rotation=-45)

ϲϵ

Pie Chart

Pie Chart is a circular chart that uses pie slices to show the relative size of data. The arc length of
each pie slice is proportional to the quantity it represents. It works beautifully on categorical
values.

data1['FNS'].value_counts().[Link](autopct='%1.1f%%',shadow=True,figsize=(8,8))

[Link]("Pie chart of Functional Status")

[Link]()

ϳϬ
Joint Distribution Plot
Joint distribution plots combine information from scatter plots and histograms to give you
detailed information for bi-variate distributions.

[Link](x='weight ',y= ‘CD4’, data=data1)

[Link] Statistical Analysis in Python

In data analysis, Statistics is important in understanding data, discovering trends and analyzing
data efficiently which coincides with the purpose of data analysis. Statistics is divided into 2
broad areas based on purpose: Descriptive Statistics and Inferential Statistics. This section is
focus the second type of Statistics for Data Analysis and it only covers Inferential Statistics
using Python.

Inferential statistics generally involves generating deductions and/or predictions about a

population. In several cases, inferences are made about a population using a sample. Unlike
descriptive statistics where a known sample/population data is described, inferential statistics
uses sample data to make conclusions about the population. For example, we want to understand
how Ethiopia people’s opinions about AI. However, the study would take too long if we
surveyed everyone in the Ethiopia population. Hence, we use the sample data representing the
population and make inferences about the Ethiopia population's opinion about AI.

ϳϭ
Gathering information about the total population can be very difficult and in some cases,
impossible. Due to this limitation, a smaller fraction of the population, known as the sample, is
analyzed and inferences are made concerning the population using the sample data collected. It
should be noted that the sample collected from a population has to be a representation of the
population for correct deductions. Usually, this is dependent on factors such as the sample size
and sampling techniques used.

Sampling Techniques

Generally, there are two sampling categories: Random/Probabilistic sampling and Non-
Probability sampling. For the former, sampling is done at random and is not biased. However,
for non-probability sampling, sampling is by deliberate choice.

The [Link]() function is typically used to select samples from a population in python,
where the number of samples to be collected is passed as an argument.

Sampling Techniques

import random

fruits = ['mango','orange','apple','pawpaw','guava','strawberry','blackberry','cucumber','lemon']

fruit_sample = [Link](fruits, k=3) # Three fruits are selected from the list of 9 fruits

print(fruit_sample)

ϳϮ
Statistical Inference
In the earlier slide, we have seen how to select random sample from fruit population. Here, we
will be taking forward the understanding the sample study to draw appropriate conclusions on
the larger population problem. Before moving any further, we need to understand two important
terminologies that are parameter and statistic.

Parameter: It is a measure that could be mean, median, variance, and many more for population
data.

Statistic: It is a measure that could be mean, median, variance, and many more for sample data.

In real-time, the population data could have millions of observations, which would make the
calculations on the entirety of the data complex and slow. Hence, we will be using the statistic
measure from the sample data to estimate or test a hypothesis (assumption) about the population
parameter. There are two types of inference:

Estimation

Hypotheses testing

Estimation in Python

The objective of estimation is to determine the value of a population parameter on the basis of a
sample statistic. There are two types of estimators:

Point Estimator

Interval estimator

ϳϯ
A point estimator draws inference about a population by estimating the value of an unknown
parameter using a single value or point while an interval estimator draws inferences about a
population by estimating the value of an unknown parameter using an interval.

For our example, we would use the following python datasets to do point and interval
Estimation.

import pandas as pd

import numpy as np

import seaborn as sns

tips = sns.load_dataset('tips')

[Link]()

We can use the following code to get the CI with Python.

import [Link] as st

from [Link] import sem# it used to find standared errors

data_mean = round(tips['tip'].mean(),3)# to find mean

data_sem = round(sem(tips['tip']),3)# to find standard error

[Link](confidence=0.95, loc=data_mean, scale=data_sem)

95 CI Output:

(2.8246682963727068, 3.171889080676473)

The above result could be interpreted that our data true population mean falls between the range
2.82 to 3.17 with 95% confidence level.

Hypothesis test in Python

Hypothesis testing is a statistical inference technique used to confirm statements made about a
population using the sample data provided. Hypothesis testing is one of the most significant

ϳϰ
aspects of inferential statistics. There are several tests applied in hypothesis testing and the
specific test to use depends on the data and purpose of the test. There are several hypothesis tests
you would need to be familiar with in your journey as a data analyst.

The purpose of hypothesis testing is determined whether two samples of data are different. In our
running example, we want to determine whether Sample 1 (CD4 cell of female patients in the
data1) is different from Sample 2 (CD4 cell of male patients at the data1). This section covers the
following tests:

Z Test & T-Test

Correlation Test

Chi Square Tests

Hypothesis Testing: Steps

State the null and the alternative hypothesis

Compute Test Statistic (or compute P value)

Search for Critical Value

Make Statistical Decision rule

Express Decision

Z-Test

The Z-Test is a hypothesis test typically used to determine if the means of two populations are
significantly different or if the mean of a population is greater than, less than or equivalent to a
specific value. This test is used when the variance(s) of the population(s) is/are known. It is also
applied when the data follows a normal distribution. When the sample size is large, it is also
assumed that the data follows a normal distribution.

For-example, a case study of the performance of students in 2 classes, the Z Test can be used to
ascertain if there is a significant difference in score. In this scenario, the null hypothesis is that
the mean scores from the two classes are equal. The hypothesis test would enable us to support

ϳϱ

or refute this claim. Usually, for hypothesis tests, a 5% level of significance is applied and the
claim is rejected if the p-value produced is less than the level of significance.

from [Link] import ztest

classA = [70, 80, 65, 42, 39, 92, 81, 54, 34, 56, 78, 92, 10, 95, 63, 55, 52, 58, 50, 61, 72, 73, 81,
82, 90, 42, 34, 42, 95, 82, 70, 70]

classB = [70, 68, 69, 40, 80, 29, 50, 54, 34, 52, 67, 45, 50, 8, 50, 52, 52, 85, 40, 60, 72, 37, 17,
28, 70, 56, 34, 24, 80, 34, 80, 70]

ztest(classA, classB, value=0)

# P Value: 0.01

#We have enough evidence to conclude that the scores of students in classA are significantly
different from classB scores.

Note: The T-Test has a similar purpose as the Z-Test. However, it is applied when the population
standard deviation is not known, or for samples with small sample sizes (n < 30).

T-test

It is used for comparing mean value when our observation comes from normal distribution. We
use the t-test at this point to formally test the hypothesis that two distributions have the same
sample mean (and thus are “the same”—or at least close enough). As in Excel and R, the two
main preconditions to running the test in Python are:

Getting the data in the right format

Determining which version of the t-test to run: equal variance or unequal variance

There are three different types of t-tests

One-sample t-test: Compares the mean of one quantitative variable data with a known or
hypothesized value

Hypothesis to be tested is:

ϳϲ
HO : = O

HA : not HO or ƒ= O , > O , < O

Formatting the data

The easiest way to format data for t-tests in Python is to use the filtering and selection techniques
already covered to create two arrays of data: CD4 cell of female patients and CD4 cell of male
patients.

The steps to do this are straightforward, although the syntax may look a bit odd initially:

Start with the data frame data1

Create a Boolean vector ( true or false values) based on the value of the “sex” column (true if
female, false otherwise)

Filter the data frame based on the Boolean vector. This creates a subset of the original data
frame.

Extract the “CD4” column from the subsetted data frames into two new
vectors: female_CD4 and ‘male_CD4

female_CD4 = data1[data1['sex'] == "F"]['CD4']

male_CD4 = data1[data1['sex'] == "M"]['CD4']

If we list the contents of female_CD4, we see it contains the CD4 of the 353 female patients.

female_CD4

ϳϳ

Example2

Let us paint another scenario of a coach who trains junior athletes to run a 100meters race. The
coach believes that the average speed of her student is 10 seconds. To confirm this, she selects
10 athletes. The command used to do t-test is:

import [Link] as stats

speed = [15, 20, 10, 8, 7, 10, 8, 11, 12, 11]

stats.ttest_1samp(speed, popmean=10)

#p value = 0.35

#since the pvalue > 0.05, we fail to reject the null hypothesis and conclude that the average speed
is 10 seconds.

Independent-samples T-test

We can perform an independent-samples t-test (also called a between-subjects t-test) when we

want to determine if the mean value on a given target variable for one group differs from the
mean value on the target variable for a different group. It is used to compare differences between
separate groups. This test is only valid if the two groups have entirely different members.

It needs

One categorical variable / independent variable

ϳϴ
One continuous variable / dependent variable

What the test will do

It will tell you whether there is a statistically significant difference in the mean scores for the 2
groups.

For example, Python would perform a t-test on two different populations to see if this data is
significantly different than the other.

st.ttest_ind(tips[tips['sex'] == 'Male']['tip'], tips[tips['sex'] == 'Female']['tip'])

pvalue=0.17

The t-test result shows that the tip between the Male and Female is not significantly different
because the P-value is above 0.05. It means we failed to reject the null hypothesis and conclude
that there are no differences between the two groups' means.

Optionally: In Machine Learning, we can perform this test using

-Using scipy library

Using Statsmodels

Using scipy library

[Link].ttest_ind(a, b, axis=0, equal_var=True)

axis : int or None, optional Axis along which to compute test. If None, compute over the whole
arrays, a, and b.

equal_var : bool, optional If True (default), perform a standard independent 2 sample test that
assumes equal population variances If False, perform Welch’s t-test, which does not assume
equal population variance.

import [Link] as stats

a = [Link]([5,6,7,8,2,3,4,5])

ϳϵ

b = [Link]([12,13,14,15,16,2,3,4])

stats.ttest_ind(a, b, equal_var=True) # Assuming that the 2 groups have equal variance

stats.ttest_ind(a, b, equal_var=False) # Assuming that the 2 groups doesn't have equal variance

Using statsmodels

from [Link] import ttest_ind

a = [Link]([12,14,16,4,5,11,12,11])

b = [Link]([12,13,14,15,16,2,3,4])

ttest_ind(a,b)

Paired-Samples T Test

The Paired-Samples T-Test procedure compares the means of two variables for a single group.

It computes the differences between values of the two variables for each case and tests whether
the average differs from 0.

One group but 2 different occasion / conditions

E.g. pre/post test

Data: For each paired test, specify two quantitative variables (interval- or ratio-level of
measurement).

For a matched-pairs or case-control study, the response for each test subject and its matched
control subject must be in the same case in the data file.

Assumptions: Observations for each pair should be made under the same conditions. The mean
differences should be normally distributed. Variances of each variable can be equal or unequal.

In Machine Learning, this Paired T-test can be calculated by using ttest_rel() method defined in
[Link] library

Example

ϴϬ
import [Link] as stats

a = [Link]([12, 14, 16, 4, 5, 11, 12, 11])

b = [Link]([12, 13, 14, 15, 16, 2, 3, 4])

stats.ttest_rel(a,b)

p-value: 0.8

From the above result, the assumed alpha value which is 0.05 is less than the obtained p-value.
Hence we can accept the Null hypothesis H0, saying that the difference between two means of
two groups is zero.

ANOVA means Analysis of Variance

Analysis of variance (ANOVA) is a statistical procedure concerned with comparing means of

several samples.

ANOVA: compare means of two or more levels of the independent variable

It can be thought of as an extension of the t-test for two independent samples to more than two
groups that the basic test uses F distribution.

Comparing means is a special case of a regression analysis

Data considerations

Independent variable (factor variable) is categorical.

Dependent variable should be quantitative (interval level of measurement).

Assumptions

Independent: each group is an independent random sample from a normal population.

Normality: analysis of variance is robust to departures from normality, although the data should
be symmetric.

Homogeneity: the groups should come from populations with equal variances.

ϴϭ

H0: m1 = m2 = m3 = . . .
= mk

Ha: Not all population means are equal

If H0 is rejected, we cannot conclude that all population means are different.

Rejecting H0 means that at least two population means have different values.

Example

Suppose a researcher is interested in determining whether there is a difference in reading

achievement using three different reading approaches. The researcher randomly assigns 5
children to each of the three reading programs.

Is there significant mean differences among the three groups at = 0.05?

# F-Test

group1=[48,50,53,52,50]

group2=[51,50,52,50,51]

group3=[61,62,60,63,60]

# Now calculate the one-way anova test for the obtained individual groups

stats.f_oneway(group1,group2,group3)

[Link]=0.000

From the above obtained result, we can decide on to the output that the pvalue which is 0.00 is
less than alpha value (0.05). Hence we can say that there is significant difference among the
variances of different reading approaches.

Note: This kind of data organization is not possible to find multiple comparisons

ϴϮ
Performing Analysis of Variance (ANOVA)

If you are comparing three or more groups of data and want to know whether the data in these
groups are different from one another the statistical method called Analysis of Variance
(ANOVA) can be used. Python has a library called statsmodels that enables one to perform
ANOVA and if needed to do then multiple comparisons test.

# import numpy and pandas libraries

import numpy as np

import pandas as pd

# import statsmodels sm,ols and pairwise_tukeyhsd

import [Link] as sm

from [Link] import ols

from [Link] import pairwise_tukeyhsd

df = pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\house_prices.csv")

[Link]()

# create a model with statsmodels ols and fit data into the model

model = ols(formula='price ~ neighborhood',data=df).fit()

# call statsmodels stats.anova_lm and pass model as an argument to perform ANOVA

anova_result = [Link].anova_lm(model, type=2)#use Type I only when there is a serious

theoretical reason for it, use Type II when there is no interaction, use Type III when there is
interaction.

print(anova_result)

ϴϯ

Performing of Multiple Comparisons Test with Python

# call statsmodels' pairwise_tukeyhsd to perform multiple comparisons test# assign ‘df price’
and 'neighborhood’set alpha to 0.05

multiple_comp_result = pairwise_tukeyhsd(endog=df['price'],

groups=df['neighborhood'],alpha=0.05) # endog means outcome var while exog means [Link]

print(multiple_comp_result.summary())

Two-way ANOVA in Python

A two-way ANOVA is used to estimate how the mean of a quantitative variable changes
according to the levels of two categorical variables. Use a two-way ANOVA when you want to
know how two independent variables, in combination, affect a dependent variable

If you are comparing three or more groups of data and want to know whether the data in
these groups are different from one another the statistical method called Analysis of
Variance (ANOVA) can be used. Python has a library called statsmodels that enables one to
perform ANOVA and if needed to do then multiple comparisons test.

ϴϰ
# import numpy and pandas libraries

import numpy as np

import pandas as pd

# import statsmodels sm,ols and pairwise_tukeyhsd

import [Link] as sm

from [Link] import ols

from [Link] import pairwise_tukeyhsd

df = pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\house_prices.csv")

[Link]()

model2 = ols(formula='price ~ neighborhood +style',data=df).fit()

# call statsmodels stats.anova_lm and pass model as an argument to perform ANOVA

anova_result2 = [Link].anova_lm(model2, type=2)

print(anova_result2)

Performing of Multiple Comparisons of Two-way ANOVA

# call statsmodels' pairwise_tukeyhsd to perform multiple comparisons test# assign price

and neighborhood columns and set alpha to 0.05

multiple_comp_result2 = pairwise_tukeyhsd(endog=df['price'],
groups=df['style'],alpha=0.05)#

multiple_comp_result3 = pairwise_tukeyhsd(endog=df['price'],
groups=df['neighborhood'],alpha=0.05)

ϴϱ
print(multiple_comp_result2.summary())

print(multiple_comp_result3.summary())

Testing for Equality of Variance

The authors of Scipy, a popular Python library for statistics, object to this choice for various
technical reasons and, as a consequence, did not implement a method to perform F-tests. Instead,
they offer two refinements: Levene’s test and Bartlett’s test (Levene’s and Bartlett’s tests are
also available in R). We will use Levene’s test:

# ensure the scipy stats module is loaded

from scipy import stats

group1=[48,50,53,52,50]

group2=[51,50,52,50,51]

[Link](group1,group2)

OUTPUT

LeveneResult(statistic=np.float64(1.523809523809523),
pvalue=np.float64(0.25207199999999985))

The output is impressive, because it contains the probability (p-value) that indicate the two
variances are equal.

Linear Regression Analysis

What is regression analysis?

Regression analysis is statistical technique for investigating and modeling the relationship
between variables.

ϴϲ
Statistical technique for assessing the relationship between dependent and one or more
independent variable

Used to calculate the extent to which one variable changes (DV) when other variable(s) change
(IV(s)).

Used to help understand possible causal effects of one variable on another.

Given pairs of X and Y variables, regression analysis measures the direction (positive and
negative) and the rate of change in Y as X changes (Slope)

Allow to predict the values of the dependent variable

Simple Linear Regression

Simple linear regression is used to predict the value of a single continuous dependent variable
(which we will call Y) from a single continuous independent variable (which we will call X).

Regression assumes that the relationship between independent variable and the dependent
variable can be represented by the equation.

Yi = 0 + 1Xi + i,

Where Yi is the value of the DV for case i,

Xi is the value of the IV for case i,

0 and 1 are constants, and

i is the error in prediction for case i.

When you perform a regression, what you are basically doing is determining estimates of 0 and
1 that let you best predict values of Y from values of X.

You may remember from geometry that the above equation is equivalent to a straight line.

0 is the intercept of the line, indicating the expected value of Y when X = 0.

ϴϳ

1 is the slope of the line, indicating how much we expect Y will change when we increase X by
a single unit.

Simple Linear Regression in Python

# Load Library

import numpy as np

import pandas as pd

import [Link] as sm;

import [Link] as plt

%matplotlib inline#This means that when you execute code that generates Matplotlib plots, the
plots are displayed inline within the notebook, rather than in a separate window or output cell.

# load datasets from Python

import seaborn as sns

df = sns.load_dataset('iris')# load python data sets from seaborn library

[Link]()

# Simple linear regression model using sepal_length and sepal_width

#add intercept

x = sm.add_constant(df['sepal_width'])

# fit model

model = [Link](df.sepal_length, x)

res = [Link]()

[Link]()

ϴϴ
Regression Diagnostics

Like R, Statsmodels exposes the residuals. That is, keeps an array containing the difference
between the observed values Y and the values predicted by the linear model. A fundamental
assumption is that the residuals (or “errors”) are random: some big, some some small, some
positive, some negative, but overall, the errors are normally distributed around a mean of zero.
Anything other than normally distributed residuals indicates a serious problem with the linear
model.

Histogram of Residuals

ϴϵ

Plotting residuals in Seaborn is straightforward: we simply pass the histplot() function the array
of residuals from the regression model.

import seaborn as sns

[Link](model_result.resid);

A slightly more useful approach for assessing normality is to compare the kernel density estimate
with the curve for the corresponding normal curve. To do this, we generate the normal curve that
has the same mean and standard deviation as our observed residual and plot it on top of our
residual.

We use a Python trick to assign two values at once: the fit() function returns both the mean and
the standard deviation of the best-fit normal distribution.

from scipy import stats

mu, std = [Link](model_result.resid)

mu, std

ϵϬ
We can now re-plot the residuals as a kernel density plot and overlay the normal curve with the
same mean and standard deviation:

import [Link] as plt

import numpy as np

fig, ax = [Link]()

# plot the residuals

[Link](x=model_result.resid, ax=ax, stat="density", linewidth=0, kde=True)

[Link](title="Distribution of residuals", xlabel="residual")

# plot corresponding normal curve

xmin, xmax = [Link]() # the maximum x values from the histogram above

x = [Link](xmin, xmax, 100) # generate some x values

p = [Link](x, mu, std) # calculate the y values for the normal curve

[Link](x=x, y=p, color="orange", ax=ax)

[Link]()

ϵϭ

Boxplot of Residuals

A boxplot is often better when the residuals are highly non-normal. Here we see a reasonable
distribution with the mean close to the median (indicating symmetry).

[Link](x=model_result.resid, showmeans=True);

Q-Q Plot

ϵϮ
A Q-Q plot is a bit more specialized than a histogram or boxplot, so the easiest thing is to use the
regression diagnostic plots provided by Statsmodels. How did we know Statsmodels has
regression diagnostic plots? We Googled it. These plots are not as attractive as the Seaborn plots,
but they are intended primarily for the data analyst. We think it is safe to assume that high-level
decision makers will not be asking for Q-Q plots.

[Link](model_result.resid, line='s');

Fit Plot

A fit plot shows predicted values of the response variable versus actual values of Y. If the linear
regression model is perfect, the predicted values will exactly equal the observed values and most
data points in a predicted versus actual scatterplot will fall on the 45° diagonal.

The fit plot provided by Statsmodels is okay in the sense that it gives a rough sense of the quality
of the model. Since the ܴ2 of this model is only 0.165, it should come as no surprise that the fit
model is not particularly good.

[Link].plot_fit(model_result,1, vlines=False);

ϵϯ
Fit plot in seaborn

As in R, creating a better fit plot is a bit more work. The central issue is that the observed and
predicted axis must be identical for the reference line to be 45°. To achieve this, we do the
following below:

Determine the min and max values for the observed values of Y

Predict values of Y

Create a plot showing the observed versus predicted values of Y. Save this to an object (in my
case ax)

Modify the chart object so that the two axes share the same minimum and maximum values

Generate data on a 45° line and add the reference line to the plot

model_result.fittedvalues

ϵϰ
Y_max = [Link]()

Y_min = [Link]()

ax = [Link](x=model_result.fittedvalues, y=Y)

[Link](ylim=(Y_min, Y_max))

[Link](xlim=(Y_min, Y_max))

ax.set_xlabel("Predicted value of Strength")

ax.set_ylabel("Observed value of Strength")

X_ref = Y_ref = [Link](Y_min, Y_max, 100)

[Link](X_ref, Y_ref, color='red', linewidth=1)

[Link]()

ϵϱ

Multiple Regressions

It is used to explain variability in a continuous dependent variable using several different

continuous independent variables.

Multiple regressions allow us to build an equation predicting the value of the dependent variable
from the values of two or more independent variables.

Linear Regression estimates the coefficients of the linear equation, involving one or more
independent variables that best predict the value of the dependent variable.

For example, you can try to predict a salesperson’s total yearly sales (the dependent variable)
from independent variables such as age, education, and years of experience.

The parameters of this equation can be used to relate the variability in our dependent variable to
the variability in specific independent variables.

The term “multiple" to describe analyses that examine the effect of two or more independent
variables on a single dependent variable.

ϵϲ
The general form of the multiple regression models is

Yi = 0 + 1Xi1 + 2Xi2 + … + kXik + i

Notation

0 is the Y intercept, where the regression line crosses the Y axis

1 is the partial slope for X1 on Y

1 indicates the change in Y for one unit change in X1, controlling for X2

2 is the partial slope for X2 on Y

2 indicates the change in Y for one unit change in X2, controlling for X1

Multiple Linear Regressions in Python

# Load Library

import numpy as np

import pandas as pd

import [Link] as sm;

import [Link] as plt

%matplotlib inline#This means that when you execute code that generates Matplotlib plots, the
plots are displayed inline within the notebook, rather than in a separate window or output cell.

# load datasets from Python

import seaborn as sns

df = sns.load_dataset('iris')# load python data sets from seaborn library

[Link]()

# Multiple linear regression model using petal_length , sepal_width and petal_width

ϵϳ

#add intercept

x = sm.add_constant(df[['petal_length', 'sepal_width', 'petal_width']])

# fit model

mod = [Link](df.sepal_length, x)

res = [Link]()

[Link]()

Dummy Variables for Categorical Variables

# import dataset

df = pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\house_prices.csv")

[Link]()

Create regression model with Baseline

ϵϴ
Fit a linear model using neighborhood A as baseline to predict the price.

# get dummy variables

import pandas as pd

#convert "neighborhood" to dummy variable

df1= pd.get_dummies(df,columns=['neighborhood'])

# view new df with dummy variables

[Link]()

Dummy Variables for Categorical Variables

# Let X = predictor variable and y = target variable

X2 = [Link](df1[['neighborhood_B', 'neighborhood_C','area']])

y2 = [Link](df1['price'])

# Add intercept and Fit linear model for neighborhood levels

df1['intercept'] = 1

lm = [Link](y2,[Link](float))

results = [Link]()

[Link]()

Since the coefficients are a comparison of the category to the baseline which is
neighbourhood A, a positive coefficient implies that category is more expensive than baseline.
Thus, neighborhood B is the most expensive and neighborhood C is the least expensive.

Visual Representation of Results

The histogram below shows the difference in average home prices based on neighbourhood. As
expected, differences are observed for Neighbourhood B against A and C.

ϵϵ

[Link]([Link]("neighborhood_C == 1")['price'], alpha = 0.3, label = 'neighborhood_C');

[Link]([Link]("neighborhood_A == 1")['price'], alpha = 0.3, label = 'neighborhood_A');

[Link]([Link]("neighborhood_B == 1")['price'], alpha = 0.3, label = 'neighborhood_B');

[Link]();

Multicollinearity and VIF (Variance Inflation Factor)

# Load Library

import pandas as pd

import numpy as np

import seaborn as sns

from patsy import dmatrices

import [Link] as sm;

from [Link].outliers_influence import variance_inflation_factor

%matplotlib inline

# import dataset

df = pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\house_prices.csv")

[Link]()

Use seaborn to look at pairwise relationships for all explanatory variables in the dataset

[Link](df[['bedrooms', 'bathrooms', 'area']]);

ϭϬϬ
Linear model to predict prices

# Add intercept and Fit linear model for neighborhood levels

df['intercept'] = 1

lm = [Link](df['price'], df1[['intercept', 'area', 'bedrooms', 'bathrooms']])

results = [Link]()

[Link]()

ϭϬϭ
VIFs Calculation

# get y and X dataframes based on regression:

y, X = dmatrices('price ~ area + bedrooms + bathrooms', df, return_type='dataframe')

# For each X, calculate VIF and save in dataframe

vif = [Link]()vif["VIF Factor"] = [variance_inflation_factor([Link], i) for i in

range([Link][1])]

vif["features"] = [Link]

[Link](1)

ϭϬϮ
The general rule is that multicollinearity exists if the VIF factor is greater than 10. Hence, both
bedrooms and bathrooms are highly correlated factors. To fix this issue, one of these variables
can be removed from the regression model.

Correlation Test in Python

In this section we use the “concrete strength” data set to explore relationships between two
continuous variables. Correlation is used to test for a relationship between two quantitative
[Link] correlation coefficient is a statistical measure that shows the degree to which,
changes to a value of one variable predict change to the value of another. The letter r is used to
represent the correlation coefficient and the r is a unit-free value between -1 and 1.

Scatterplots are a fundamental graph type that much less complicated than histograms and
boxplots.

Positive Correlation: If the quantity of milk increases, the price also increases.

Negative Correlation: If the price of a stock goes down, then the buying of that stock increases.

Zero Correlation : There is no relationship between score in video games and grades of an
examination.

The strength of the correlation is considered as - weak positive correlation as 0<r<0.3,

weak negative correlation as -0.3<r<0

strong positive correlation as 0.5<r<1,

ϭϬϯ

strong negative correlation as -1<r<-0.5

no correlation as r=0

As such, we might use the Mathplotlib library instead of the Seaborn library. But since we have
already used Seaborn, we will stick with it here. Just know that there are many ways to create
scatterplots and other basic graphs in Python.

To create a bare-bones scatterplot, we must do four things:

Load the seaborn library

Specify the source data frame

Set the x axis, which is generally the name of a predictor/independent variable

Set the y axis, which is generally the name of a response/dependent variable

Let’s start by listing the column names. A Pandas DataFrame object exposes a list of columns
through the columns property. Here we use the list() type conversion method to convert the
results to a simple list (which prints nicer):

import seaborn as sns

[Link](x='weight ', y='CD4', data=data1);

Adding labels

To this point, we have not said much about decorating Seaborn charts with labels and other
details. This is because Seaborn does a pretty good job by default. But if we do need to clean up
our charts a bit, here is the thing to know: the Seaborn chart methods return an object (of type
AxesSubplot, whatever that is) for which properties can be set.

Here we assign the results of the scatterplot() call to a variable called ax and then set various
properties of ax. we end the last line of the code block with a semicolon to suppress return
values:

import seaborn as sns

ϭϬϰ
ax=[Link](x='weight ', y='CD4', data=data1);

ax.set_title("weight vs. CD4")

ax.set_xlabel("weight of patient's");

Adding a best fit line

As we saw with SAS Enterprise Guide and R, it is sometimes useful to add a best fit line (with
confidence intervals around the slope) to a scatterplot. But let’s be clear: this is not one of these
situations. It is obvious from the scatterplot above that the relationship between concrete strength
and fly ash is only weakly linear. The easiest way to “add” a best-fit line to a scatterplot is to use
a different plotting method. Seaborn’s lmplot() method (where “lm” stands for “linear model”) is
one possibility:

[Link](x="weight ", y="CD4", data=data1);

ϭϬϱ
Adding color as a third dimension

A graphics “party trick” made fashionable by tools like Tableau is to use color, size, or some
other visual cue to add a third dimension to a two-dimensional scatterplot. In the case of color (or
“hue” in Seaborn terminology), this third dimension need to be a non-continuous variable. This
is because the palette of colors available has a finite number of options.

[Link](x="weight ", y="CD4", hue="FNS", data=data1);

ϭϬϲ
# correlation Test from numpy and scipy library

import numpy as np

import [Link]

x = [Link](10, 20)

y = [Link]([2, 1, 4, 5, 8, 12, 18, 25, 96, 48])

[Link](x, y) # Pearson's r

[Link](x, y) # Spearman's rho

import pandas as pd# corrrlation test from pandas library

x = [Link](range(10, 20))

ϭϬϳ
y = [Link]([2, 1, 4, 5, 8, 12, 18, 25, 96, 48])

[Link](y)

[Link](y, method='spearman') # Spearman's rho

[Link](y, method='kendall') # Kendall's tau

import pandas as pd

import numpy as np

data2 = [Link][range(0, len(data1), 1),[3,8,9,10]]# data extraction from data1 data frame

[Link](method='pearson') # or [Link]() gives the same result as method pearson is defaulted

[Link](method='spearman') # correlaion coefficient using spearman method

[Link](method='kendall') # correlation coefficient using kendall method

Coefficient of Correlation

A correlation coefficient (typically denoted r) is a single number that describes the extent of
the linear relationship between two variables. A value of +1 indicates perfect linearity (the two
variables move together, like “height in inches” and “height in centimeters”).

A value of r = 0 indicates no correlation (the variables are independent) and r = -1 indicates the
variables are inversely correlated (an increase in one variable is associated with a decrease in the
other).

Like many other statistics (measures derived from raw data), there are slightly different ways to
calculate the correlation coefficient that are more or less sensitive to outliers and other
characteristics of the data. The most common measure is the Pearson correlation coefficient. The
Scipy library provides a method called pearsonr() (Pearson’s r).

from scipy import stats

[Link](data1['weight '], data1['CD4'])

ϭϬϴ
Again, it is not the prettiest result. But, if we were so inclined, we could write the results to a
data frame and apply whatever formatting in Python we wanted to. In this form, however, we get
two numbers:

Pearson’s r (0,04063—same as we got in Excel, R, etc.)

A p-value. This is the probability that the true value of r is zero (no correlation).

We conclude based on this that there is weak linear relationship between weight and CD4 but not
so weak that we should conclude the variables are uncorrelated. In other words, it seems that
CD4 does have some influence on weight.

Of course, correlation does not imply causality. It is equally correct, based on the value of r, to
say that concrete weight has some influence on the amount of CD4 in the mix.

Corrleation matrix

A correlation matrix is a handy way to calculate the pairwise correlation coefficients between
two or more (numeric) variables. The Pandas data frame has this functionality built-in to
its corr() method, which we have wrapped inside the round() method to keep things tidy. Notice
that every correlation matrix is symmetrical: the correlation of “weight” with “age” is the same
as the correlation of “age” with “weight ” (-0.24). Thus, the top (or bottom, depending on your
preferences) of every correlation matrix is redundant. The correlation between each variable and
itself is 1.0, hence the diagonal.

data2 = [Link][range(0, len(data1), 1),[3,8,9,10]]# data extraction from data frame data1

cormatrix = [Link]()

round(cormatrix,2)

ϭϬϵ

Correlation matrix to heat map

Python, and its libraries, make lots of things easy. For example, once the correlation matrix is
defined (we assigned to the variable cormat above), it can be passed to
Seaborn’s heatmap() method to create a heatmap (or headgrid). The basic idea of heatmaps is
that they replace numbers with colors of varying shades, as indicated by the scale on the right.
Cells that are lighter have higher values of r. This type of visualization can make it much easier
to spot linear relationships between variables than a table of numbers. For example, if we focus
on the “weight” column, we immediately see that “age” and “time” have the largest positive
correlations whereas “CD4” has the large negative correlation.
[Link](cormatrix);

Chi-squared Test of Independence in Python

Measures the association between two categorical variables; compares the observed frequencies
with the expected frequencies. Expected frequencies are those you would expect according to the
null hypothesis. Determines whether two variables are independent; Allows us to find out
whether various sub-groups are homogeneous.

ϭϭϬ
The row percentages leave us with the impression that functional status depends on gender. We
can test this more formally using the ߯2 (Chi-square) test of independence.

In Machine learning, to perform chi-squared test we use a method named chisquare which is
imported from [Link].

This section will cover how to conduct a 2 test of independence using [Link] and
researchpy. First, let's import pandas, [Link], [Link], and researchpy

# load library

import pandas as pd

import researchpy as rp

import [Link] as stats

import [Link] as sm

Scipy has a method called chi2_contingency() that takes a contingency table of observed
frequencies as input. Note that this table cannot include marginal totals or marginal frequencies.
Instead, it must consist of m x n observations:

# ensure the scipy stats module is loaded

from scipy import stats

contab_obs = [Link](

data1['sex'],

data1['FNS'],

margins = False)

chi = stats.chi2_contingency(contab_obs)

chi

ϭϭϭ

The output of the chi2_contingency() method is not particularly attractive but it contains what we
need:

The first line is the ߯2 statistic, which we can safely ignore

The second line is the probability of getting a ߯2 statistic that large if the two variables are
independent. This p-value is very large (0.093) so we conclude that gender and functional status
are independent.

The third line is the degrees of freedom, which we can safely ignore.

The remainder of the output is a matrix showing the expected frequencies under the assumption
in independence. These expected values are quite different from the observed values above.

You may notice that the ߯2 statistic and p-value are different from those provided by R. This is
because scipy defaults to the “Pearson’s Chi-squared test with Yates’ continuity correction”
version of the test.

Example 2:

# import data

df = pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data sets\house_prices.csv")

[Link]()

Before testing this relationship, let's see some basic univariate statistics.

rp.summary_cat(df[["neighborhood", "style"]])

ϭϭϮ
The method that needs to be used is [Link].chi2_contingency and it requires one to pass a
crosstabulation table, this can be accomplished using [Link].

#crosstabulation table

crosstab = [Link](df["neighborhood"], df["style"])

crosstab

Now to pass this contingency table to the [Link] method. The output isn't the best formatted,
but all the information is there. The information is returned within a tuple where the first value is
the test static, the second value is the p-value, and the third number is the degrees of freedom.
An array is also returned which contains the expected cell counts.

stats.chi2_contingency(crosstab)

ϭϭϯ

Logistic Regression

Logistic regression is useful for situations in which you want to be able to predict the presence or
absence of a characteristic or outcome based on values of a set of predictor variables. It is similar
to a linear regression model but is suited to models where the dependent variable is dichotomous.
Logistic regression coefficients can be used to estimate odds ratios for each of the independent
variables in the model.

Example

1. Suppose that we are interested in the factors that influence whether a political candidate wins
an election. The outcome (response) variable is binary (0/1); win or lose. The predictor
variables of interest are the amount of money spent on the campaign, the amount of time spent
campaigning negatively and whether or not the candidate is an incumbent.

2. A researcher is interested in how variables, such as GRE (Graduate Record Exam scores),
GPA (grade point average) and prestige of the undergraduate institution, effect admission into
graduate school. The response variable, admit/don't admit, is a binary variable.

Data: The dependent variable should be dichotomous. Independent variables can be interval level
or categorical; if categorical, they should be dummy or indicator coded (there is an option in the
procedure to recode categorical variables automatically). Examples:-

Attitude (positive/negative)

HIV test(yes/no)

The "logit" model solves these problems:

ln[p/(1-p)] = D + EX + e

p is the probability that the event Y occurs, p(Y=1)

p/(1-p) is the "odds ratio" =exp(E)

ln[p/(1-p)] is the log odds ratio, or "logit"

ϭϭϰ
The logistic distribution constrains the estimated probabilities to lie between 0 and 1.

The estimated probability is:

p = 1/[1 + exp(-D - E X)]

if you let D + E X =0, then p = .50

as D + E X gets really big, p approaches 1

as D + E X gets really small, p approaches 0

Binary Logistic Regression

It is used to determine the impact of multiple independent variables presented simultaneously to

predict membership of one or other of the two dependent variable categories.

Since the dependent variable is dichotomous we cannot predict a numerical value for it using
logistic regression so the usual regression least squares deviations criteria for best fit approach of
minimizing error around the line of best fit is inappropriate (It’s impossible to calculate
deviations using binary variables!).

Logistic regression forms a best fitting equation or function using the maximum likelihood (ML)
method, which maximizes the probability of classifying the observed data into the appropriate
category given the regression coefficients.

The Purpose of binary logistic regression is used to calculates the probability of success over the
probability of failure, the results of the analysis are in the form of an odds ratio. It also provides
the relationships and strengths among the variables.

Fitting a logistic regression model to a dataset to predict if a transaction is fraud or not using
Python.

Logistic Regression in Python With StatsModels

# All you need to import is NumPy and [Link]:

import numpy as np

ϭϭϱ

import [Link] as sm

StatsModels doesn’t take the intercept ܾ into account, and you need to include the additional
column of ones in x. You do that with add_constant():

x = [Link](10).reshape(-1, 1)

y = [Link]([0, 1, 0, 0, 1, 1, 1, 1, 1, 1])

x = sm.add_constant(x)

NB:add_constant() takes the array x as the argument and returns a new array with the
additional column of ones.

model = [Link](y, x)# logistic regression model is going to be an instance of the

class [Link].discrete_model.Logit

result = [Link](method='newton')# The model is now ready, and the variable result holds
useful data. For example, you can obtain the values of ܾ and ܾ with .params:

[Link]# The first element of the obtained array is the intercept ܾ, while the second is the
slope ܾ. For more information, you can look at the official documentation on Logit, as well
as .fit() and .fit_regularized().

[Link](x)# These probabilities are calculated with .predict(). You can use their values to
get the actual predicted outputs

[Link]()# .summary() and .summary2() get output data that you might find useful in
some circumstances:

result.summary2()

Multinomial Logistic Regression

A multinomial logistic regression (or multinomial regression for short) is used when the outcome
variable being predicted is nominal and has more than two categories.

Examples of multinomial logistic regression

ϭϭϲ
Example 1. People’s occupational choices might be influenced by their parents’ occupations and
their own education level. We can study the relationship of one’s occupation choice with
education level and father’s occupation. The occupational choices will be the outcome variable
which consists of categories of occupations.

Example 2. A biologist may be interested in food choices that alligators make. Adult alligators
might have different preferences from young ones. The outcome variable here will be the types
of food, and the predictor variables might be size of the alligators and other environmental
variables.

Example 3. Entering high school students make program choices among general program,
vocational program and academic program. Their choice might be modeled using their writing
score and their social economic status.

If you have categorical dependent variable with more than two possible values you can use an
extension of binary logistic regression to examine the relationship between the dependent
variable and independent variable, called multinomial logistic regression.

The models are called multinomial since for each combination of values of independent variable
the counts of the dependant variable are assumed to have a multinomial distribution.

The counts at the different combination are also assumed to be independent with a fixed total.

Now, we can use the statsmodels api to run the multinomial logistic regression, the data that we
will be using in this section would be from the first we are going to import necessary packages
and available data.

import numpy as np

import pandas as pd

from sklearn import datasets df = pd.read_csv(r"C:\Users\Kindu\Desktop\Data sets\Data

sets\[Link]")

import [Link] as smf

ϭϭϳ

# you can check the structure of data

print(df['style'].unique())

print(df['neigh1orhood'].unique())

model= [Link]('neigh1orhood~C(style)', df).fit()

[Link]

[Link]([Link])

[Link]()

model.summary2()

Ordinal Logistic Regression

Ordinal logistic regression is a statistical model that can be used to model the relationship
between an ordinal response variable and one or more explanatory variables. In this model, there
is a coefficient for each category. The data set we used has three categories, which included
“very likely,” “somewhat likely”, and “unlikely.” We have two coefficients (intercepts) for our
categories in the ordinal logistic model.

Based on the distribution of error term ordinal regression can be classify as three. These are
Logit , probit and cumulative log-log models are basically the same, the difference is in the
distribution: Logit model – Cumulative standard logistic distribution (F) while probit model –
Cumulative standard normal distribution () Both models provide similar results. combined
effect, of all the variables in the model, is different from zero.

Probit regression, also called a probit model, is used to model dichotomous or binary outcome
variables. In the probit model, the inverse standard normal distribution of the probability is
modeled as a linear combination of the predictors.

ϭϭϴ
Probit Ordinal Logistic Regression
# load Library

import numpy as np

import pandas as pd

import [Link] as stats

from [Link].ordinal_model import OrderedModel

# load datasets

url = [Link]

data_student = pd.read_stata(url)

data_student.head(5)

# identify data type

data_student.dtypes

data_student['apply'].dtype

mod_prob = OrderedModel(data_student['apply'], data_student[['pared', 'public', 'gpa']],

distr='probit')

res_prob = mod_prob.fit(method='bfgs')

res_prob.summary()

ϭϭϵ
Since there are 3 categories in the target variable (unlikely, somewhat likely, very likely), we
have two thresholds to estimate. OrderedModel.transform_threshold_params, the first estimated
threshold is the actual value and all the other thresholds are in terms of cumulative exponentiated
increments.

Actual thresholds values can be computed as follows:

Thresholds is a value that sets a limit or boundary, above or below which a different state or
condition is observed.

num_of_thresholds = 2

mod_prob.transform_threshold_params(res_prob.params[-num_of_thresholds:])

Statistically, the logit is the inverse of the standard logistic function of error.

mod_log = OrderedModel(data_student['apply'], data_student[['pared', 'public', 'gpa']],

distr='logit')

ϭϮϬ
res_log = mod_log.fit(method='bfgs', disp=False)#disp=False indicate convergence
information is not printed the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm is an
iterative method for solving unconstrained nonlinear optimization problems

res_log.summary()

Logit Ordinal Regression

# Optionally logit model

modf_logit = OrderedModel.from_formula("apply ~ 0 + pared + public + gpa",

data_student,distr='logit')

resf_logit = modf_logit.fit(method='bfgs')

resf_logit.summary()

ϭϮϭ

predicted = res_log.[Link](res_log.params, exog=data_student[['pared', 'public', 'gpa']])

predicted

Cloglog Regression in Python

Cloglog regression is an extension of the logistic regression model and is particularly useful
when the probability of an event is very small or very large. Most of the time cloglog regression
is used while dealing with rare events or situations where the outcome is extremely skewed.

# minimal definition of a custom scipy [Link] CLogLog(stats.rv_continuous):

def _ppf(self, q): return [Link](-[Link](1 - q))

ϭϮϮ
def _cdf(self, x): return 1 - [Link](-[Link](x))

cloglog = CLogLog()

res_cloglog = OrderedModel(data_student['apply'], data_student[['pared', 'public', 'gpa']],

distr=cloglog).fit(method='bfgs', disp=False)

res_cloglog.summary()

ϭϮϯ
Reference

Chun, W., 2001. Core python programming (Vol. 1). Prentice Hall Professional.

Fandango, A., 2017. Python Data Analysis. Packt Publishing Ltd.

Haslwanter, T., 2016. An introduction to statistics with python. With applications in the life
sciences. Switzerland: Springer International Publishing.

Lutz, M., 2001. Programming python. " O'Reilly Media, Inc.".

McKinney, W., 2022. Python for data analysis. " O'Reilly Media, Inc.".

Sahoo, K., Samal, A.K., Pramanik, J. and Pani, S.K., 2019. Exploratory data analysis using
Python. International Journal of Innovative Technology and Exploring Engineering, 8(12),
pp.4727-4735.

Seabold, S. and Perktold, J., 2010. Statsmodels: econometric and statistical modeling with
python. SciPy, 7(1).

Waskom, M.L., 2021. Seaborn: statistical data visualization. Journal of Open Source
Software, 6(60), p.3021.

ϭϮϰ
Buy your books fast and straightforward online - at one of world’s
fastest growing online book stores! Environmentally sound due to
Print-on-Demand technologies.
Buy your books online at
[Link]
Kaufen Sie Ihre Bücher schnell und unkompliziert online – auf einer
der am schnellsten wachsenden Buchhandelsplattformen weltweit!
Dank Print-On-Demand umwelt- und ressourcenschonend produzi
ert.
Bücher schneller online kaufen
[Link]

info@[Link]
[Link]

Python For Data Analysis Guide
No ratings yet
Python For Data Analysis Guide
8 pages
Python for Data Analysis Overview
No ratings yet
Python for Data Analysis Overview
18 pages
Data Analysis
No ratings yet
Data Analysis
18 pages
Data Analysis with Python Essentials
No ratings yet
Data Analysis with Python Essentials
18 pages
Python for Data Analysis Overview
No ratings yet
Python for Data Analysis Overview
18 pages
Lecture#01
No ratings yet
Lecture#01
16 pages
COS 305 WK 1-Introduction
No ratings yet
COS 305 WK 1-Introduction
6 pages
Python Data Analysis Essentials
0% (1)
Python Data Analysis Essentials
17 pages
Python for Data Analysis Overview
No ratings yet
Python for Data Analysis Overview
16 pages
Python for Data Analysis_ the Ultimate and Definitive Manual to Learn Data Science and Coding With Python. Master the Basics of Machine Learning, To Clean Code and Improve Artificial Intelligence [BooksRack.net]
No ratings yet
Python for Data Analysis_ the Ultimate and Definitive Manual to Learn Data Science and Coding With Python. Master the Basics of Machine Learning, To Clean Code and Improve Artificial Intelligence [BooksRack.net]
73 pages
The Python Bible For Beginners
100% (2)
The Python Bible For Beginners
185 pages
Python Data Science Wilkinson CH
100% (1)
Python Data Science Wilkinson CH
153 pages
Data Analytics Using Python
100% (11)
Data Analytics Using Python
982 pages
Python For Beginners ?
No ratings yet
Python For Beginners ?
155 pages
Christopher Wilkinson - Python Data Science - An Ultimate Guide For Beginners To Learn Fundamentals of Data Science Using Python (2020)
100% (2)
Christopher Wilkinson - Python Data Science - An Ultimate Guide For Beginners To Learn Fundamentals of Data Science Using Python (2020)
141 pages
Data Analytics Training Report: Python
No ratings yet
Data Analytics Training Report: Python
31 pages
Introduction to Python Programming
No ratings yet
Introduction to Python Programming
9 pages
Python 6 Books in 1 The Ultimate Bible To Lear... Z Library
No ratings yet
Python 6 Books in 1 The Ultimate Bible To Lear... Z Library
755 pages
Quick Guide to Python Programming
No ratings yet
Quick Guide to Python Programming
74 pages
Python for Auditing: A Quick Guide
100% (2)
Python for Auditing: A Quick Guide
4 pages
Data Science Internship Report in Python
No ratings yet
Data Science Internship Report in Python
20 pages
Introduction to Python Programming
No ratings yet
Introduction to Python Programming
5 pages
Session 1 - Summary-5999
No ratings yet
Session 1 - Summary-5999
5 pages
Introduction to Python Programming
No ratings yet
Introduction to Python Programming
10 pages
Introduction to Python Programming
No ratings yet
Introduction to Python Programming
7 pages
Exploring Python's Versatile Applications
No ratings yet
Exploring Python's Versatile Applications
12 pages
Learn Data Science with Python Basics
No ratings yet
Learn Data Science with Python Basics
28 pages
Unit3 Complete WIT
No ratings yet
Unit3 Complete WIT
98 pages
Python For Beginners An Essential Guide To Easy Learning With Basic Exercises Python Programming Crash Course For Data (Walsh, Conley) (Z-Library)
No ratings yet
Python For Beginners An Essential Guide To Easy Learning With Basic Exercises Python Programming Crash Course For Data (Walsh, Conley) (Z-Library)
114 pages
1710496889134
100% (1)
1710496889134
155 pages
Unit-1 Introduction Python
No ratings yet
Unit-1 Introduction Python
11 pages
Python For Beginners An Essential Guide To Easy Learning With Basic Exercises Python Programming Crash Course Compress
No ratings yet
Python For Beginners An Essential Guide To Easy Learning With Basic Exercises Python Programming Crash Course Compress
110 pages
Data Science Internship with Python
No ratings yet
Data Science Internship with Python
35 pages
PDS Unit1 DrNimritaKoul
No ratings yet
PDS Unit1 DrNimritaKoul
96 pages
Python Basics for Beginners Guide
No ratings yet
Python Basics for Beginners Guide
7 pages
Master Python for Data Science Basics
No ratings yet
Master Python for Data Science Basics
10 pages
A Complete Tutorial To Learn Data Science With Python From Scratch
No ratings yet
A Complete Tutorial To Learn Data Science With Python From Scratch
43 pages
Python Fundamentals Book
No ratings yet
Python Fundamentals Book
91 pages
Python Programming For Beginner - Jackson, Kit
100% (2)
Python Programming For Beginner - Jackson, Kit
311 pages
Basics of Python Programming and Statistics
No ratings yet
Basics of Python Programming and Statistics
56 pages
Python Programming for Beginners_ From Basics to AI Integrations. 5-Minute Illustrated Tutorials, Coding Hacks, Hands-On Exercises & Case Studies to Master Python in 7 Days and Get Paid More by Prince
100% (18)
Python Programming for Beginners_ From Basics to AI Integrations. 5-Minute Illustrated Tutorials, Coding Hacks, Hands-On Exercises & Case Studies to Master Python in 7 Days and Get Paid More by Prince
244 pages
Report R
No ratings yet
Report R
28 pages
Quick Python Programming Guide
100% (1)
Quick Python Programming Guide
207 pages
Introduction to Python Interpreter
No ratings yet
Introduction to Python Interpreter
18 pages
Python For Data Analysis
No ratings yet
Python For Data Analysis
2 pages
Introduction to Python Programming
No ratings yet
Introduction to Python Programming
2 pages
1710988761593
100% (2)
1710988761593
169 pages
Deep Learning Programming Basics
No ratings yet
Deep Learning Programming Basics
12 pages
Data Analysis of Visualization: CHAPTER - 1 Preliminaries
No ratings yet
Data Analysis of Visualization: CHAPTER - 1 Preliminaries
93 pages
History and Basics of Python Programming
No ratings yet
History and Basics of Python Programming
76 pages
Data Analysis From Scratch With Python - Beginner Guide Using Python, Pandas, NumPy, Scikit-Learn, IPython, TensorFlow and
100% (10)
Data Analysis From Scratch With Python - Beginner Guide Using Python, Pandas, NumPy, Scikit-Learn, IPython, TensorFlow and
104 pages
Python Programming Language Overview
No ratings yet
Python Programming Language Overview
11 pages
Python for Machine Learning Basics
No ratings yet
Python for Machine Learning Basics
115 pages
Python Overview: History, Features, Installation
No ratings yet
Python Overview: History, Features, Installation
42 pages
Introduction to Python Programming
No ratings yet
Introduction to Python Programming
9 pages
Python Programming: A Comprehensive Guide
No ratings yet
Python Programming: A Comprehensive Guide
25 pages
Python Highway 2 Books in 1 The Fastest Way For Beginners To Learn Python Programming, Data Science and Machine Learning in 3 Days (Or Less) + Practical Exercises Included by Cox, Aaron
100% (1)
Python Highway 2 Books in 1 The Fastest Way For Beginners To Learn Python Programming, Data Science and Machine Learning in 3 Days (Or Less) + Practical Exercises Included by Cox, Aaron
174 pages
Beginner's Guide to Python Programming
No ratings yet
Beginner's Guide to Python Programming
14 pages
Python Data Analysis Basics Guide
No ratings yet
Python Data Analysis Basics Guide
6 pages
How To Read HTML Tables With Pandas
No ratings yet
How To Read HTML Tables With Pandas
18 pages
How To Set Up Python Virtual Environments (Venv) On Ubuntu
No ratings yet
How To Set Up Python Virtual Environments (Venv) On Ubuntu
30 pages
How To Read HTML Tables With Pandas - From - MD
No ratings yet
How To Read HTML Tables With Pandas - From - MD
7 pages
Statistics Made Easy 6 8wmqgs
No ratings yet
Statistics Made Easy 6 8wmqgs
454 pages
How To Read HTML Tables With Pandas (Including Example)
No ratings yet
How To Read HTML Tables With Pandas (Including Example)
7 pages
National Basketball Association - Wikipedia
No ratings yet
National Basketball Association - Wikipedia
27 pages
How Not To Match A Character After Repetition in Python Regex
No ratings yet
How Not To Match A Character After Repetition in Python Regex
4 pages
Ok - Python Regex Metacharacters
100% (1)
Ok - Python Regex Metacharacters
13 pages
Normalize JSON Data Using Pandas - Json - Normalize - by Pongsakorn Akkaramanee - Medium
No ratings yet
Normalize JSON Data Using Pandas - Json - Normalize - by Pongsakorn Akkaramanee - Medium
11 pages
0k STD Deviation in Stats
No ratings yet
0k STD Deviation in Stats
8 pages
Pandas - How To Read Data From HTML Tables - Proclus Academy
No ratings yet
Pandas - How To Read Data From HTML Tables - Proclus Academy
13 pages
4 Ways of Multiplying Matrice
No ratings yet
4 Ways of Multiplying Matrice
1 page
Unit Iv - Python Notes
No ratings yet
Unit Iv - Python Notes
30 pages
Bca New Syl
No ratings yet
Bca New Syl
20 pages
Your C How To Program 9e 9th Edition Paul Deitel PDF
100% (10)
Your C How To Program 9e 9th Edition Paul Deitel PDF
188 pages
Job Portal System OOSE 20 Pages
No ratings yet
Job Portal System OOSE 20 Pages
22 pages
Embedded System (Unit-3)
No ratings yet
Embedded System (Unit-3)
32 pages
Java OOPS Notes Basic To Advanced
No ratings yet
Java OOPS Notes Basic To Advanced
10 pages
Class and Object in Python
No ratings yet
Class and Object in Python
4 pages
Unified Fine-Grained Access Control For Personal Health Records in Cloud Computing
No ratings yet
Unified Fine-Grained Access Control For Personal Health Records in Cloud Computing
62 pages
Calorie Calculator Project Report
No ratings yet
Calorie Calculator Project Report
2 pages
DSTP3.0-Batch-03 AIP301 1
No ratings yet
DSTP3.0-Batch-03 AIP301 1
3 pages
Java: A Beginner's Guide 7th Edition Herbert Schildt - Completed Chapters
100% (3)
Java: A Beginner's Guide 7th Edition Herbert Schildt - Completed Chapters
203 pages
Software Engineering Lab Record
No ratings yet
Software Engineering Lab Record
70 pages
Python Viva Notes
No ratings yet
Python Viva Notes
4 pages
Object Oriented Programming CheatSheet - by Love Babbar
No ratings yet
Object Oriented Programming CheatSheet - by Love Babbar
1 page
Csharp Interview Questions With Answers
No ratings yet
Csharp Interview Questions With Answers
52 pages
Java How To Program Early Objects Eleventh Edition by Deitel Ebook & Testbank
No ratings yet
Java How To Program Early Objects Eleventh Edition by Deitel Ebook & Testbank
214 pages
Java JSP and MySQL Project On Online Food Ordering System
No ratings yet
Java JSP and MySQL Project On Online Food Ordering System
181 pages
Oop - Course Outline
No ratings yet
Oop - Course Outline
4 pages
Programming: Game Component Development
No ratings yet
Programming: Game Component Development
21 pages
Local Need Doc-2
No ratings yet
Local Need Doc-2
32 pages
Core Java Microservices Interview Manual Alacriti Focus
No ratings yet
Core Java Microservices Interview Manual Alacriti Focus
28 pages
BridgeLabz Python Maker Plan 2
No ratings yet
BridgeLabz Python Maker Plan 2
2 pages
Unified Modeling Language (UML)
No ratings yet
Unified Modeling Language (UML)
4 pages
Ltimindtree Technical Interview Questions
No ratings yet
Ltimindtree Technical Interview Questions
3 pages
3 Ch-01 C++ Programming - Class, Object, Functions
No ratings yet
3 Ch-01 C++ Programming - Class, Object, Functions
61 pages
PPS Question Bank Solution-1
No ratings yet
PPS Question Bank Solution-1
23 pages
Python Internship Report
No ratings yet
Python Internship Report
56 pages
Oodbms Lecture 21
No ratings yet
Oodbms Lecture 21
31 pages
Question Bank For PHP Programming With MySQL The Web Technologies Series 2nd Edition by Don Gosselin Diana Kokoska Robert Easterbrooks
No ratings yet
Question Bank For PHP Programming With MySQL The Web Technologies Series 2nd Edition by Don Gosselin Diana Kokoska Robert Easterbrooks
11 pages
300 Core Java Interview Questions - Javatpoint
No ratings yet
300 Core Java Interview Questions - Javatpoint
38 pages

Statistical Data Analysis Using Python

Uploaded by

Statistical Data Analysis Using Python

Uploaded by

            

  

)*+,$ # -   .*/#%0 1

 511   6  ! 7 %  

Kindu Kebede Gebre(Assistant Professor)

Million Wesenu Demissie(Assistant Professor)

Python is an interpreter, object-oriented, high-level programming language with dynamic

¾ Easy to Learn and Use

The common use of python in the real world:-

¾ Data analysis and machine learning

Data Analysis and Machine Learning

Software testing and prototyping

Keep track of stock market or crypto prices

¾ Send yourself a text reminder to carry an umbrella anytime it’s raining

[Link] to Install Python on Windows

Step 3: select Windows x86-64 executable installer

It does will start the installation process.

1.5. Python Development Environment

[Link] Started with Python Programming

Python is a high-level, interpreted, and general-purpose programming language. It was created

[Link] to Create and Run Python Program

1.10. Library in Python

Many popular Python toolboxes/libraries:

9 It stores data internally in a continuous block of memory, independent of other in-built

SciPy ¾ collection of algorithms for linear algebra, differential equations,

 Provides machine learning algorithms: classification, regression, clustering,

 python 2D plotting library which produces publication quality figures in a variety

1.11. How to install libraries in Python

Installing libraries in Python is a fundamental aspect of Python development, enabling

Install Pandas on Windows

Python Pandas can be installed on Windows in two ways:

PIP is a package management system used to install and manage software

from pandas import DataFrame

Now we can type DataFrame rather than [Link].

1.12. Help function in Python

1.13. How to Save Python Script

2. Data Entry and Data management in Python

[Link] to Define Scalar Object in Python

A vector in a simple term can be considered as a single-dimensional array. With respect to

[Link] to define Vectors in Python

A vector in a simple term can be considered as a single-dimensional array. With respect to

NumPy is a general-purpose array-processing package. It provides a high-performance

print("Vector created from a list:")

Example 2: Vertical Vector

print("Vector created from a list:")

Performing addition operation on a Python Vector

print("Addition of two vectors: ",vctr_add)

Performing Subtraction of two vectors

print("Subtraction of two vectors: ",vctr_sub)

Performing multiplication of two vectors

print("multiplication of two vectors: ",vctr_mul)

Performing Vector division operation

print("division of two vectors: ",vctr_div)

Vector Dot Product

print("Dot product of two vectors: ",vctr_dot)

matrix = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

Here is a list of basic manipulations:

Here is an example of addition using nested lists and loops:

matrix1 = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

matrix2 = [Link]([[9, 8, 7], [6, 5, 4], [3, 2, 1]])

result = [Link](matrix1, matrix2)

matrix1 = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

matrix2 = [Link]([[9, 8, 7], [6, 5, 4], [3, 2, 1]])

result = [Link](matrix1, matrix2)

matrix = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

result = [Link](matrix, scalar)

matrix = [Link]([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

result = matrix * scalar

NumPy uses the dot() method for matrix multiplication:

matrix1 = [Link]([[1, 2], [3, 4]])

matrix2 = [Link]([[5, 6], [7, 8]])

result = [Link](matrix1, matrix2)

Note: The @ operation works similarly to the dot() method

matrix1 = [Link]([[1, 2], [3, 4]])

matrix2 = [Link]([[5, 6], [7, 8]])

result = matrix1 @ matrix2

)+,$ # - ./#%0 1

511 6 ! 7 %

Provides machine learning algorithms: classification, regression, clustering,

python 2D plotting library which produces publication quality figures in a variety