Note of data science training EP 2: Pandas & Matplotlib – from a thousand mile above
Here we go see the data structure that we need to work with Pandas, Dataframe.
Hi all guys~
From what I put at the end of EP 1, we are going next to browsing the data.
At the first we obtained data, most of them are raw and cannot be used until we cleanse it. Therefore, this is a good time to introduce the library that I can call it is the root of data science work. It is ~
Pandas
Pandas is almost everything for our works. Here we go see the data structure that we need to work with Pandas, Dataframe.
Dataframe is like a simple table. It contains Columns as our data and we call a single column as Series.
More than that, there is Index to identify a cell location in a table and Index can be one of these:
Row indexis a list of unique values of a single row.
Index column must be assigned first.Column indexis a list of unique values of a single column
Let’s say from the sample table above, we assigned the index column as column A. We find the value of row index “A3” and column index “C” is “C3”. That’s it.
Deal with the dataset
Go playing with the titanic dataset. I got this from Kaggle.com/hesh97.
Read
We write this to open the CSV file from the link.
1
2
3
import pandas as pd
titanic = pd.read_csv("./titanicdataset-traincsv.csv")
titanic
Display more
The result is sparse so we can write this below to show them all1.
1
2
3
4
# show 15 rows
pd.set_option('display.max_rows', 15)
# show ALL rows
pd.set_option('display.max_rows', None)
Using brackets
Now we go ahead to see the main functionalities of Dataframe.
1
2
print(type(titanic[['Sex']]))
titanic[['Sex']]
Using 2 pairs of square brackets means a Dataframe of selected columns.
2 pairs of brackets mean DataFrame
while…
1
2
print(type(titanic['Sex']))
titanic['Sex']
A pair of square brackets is a Series of selected column.
A pair of brackets means Series
Head and tail
1
2
titanic.head()
titanic.tail()
head() and tail() show first 5 rows and last 5 rows of the Dataframe correspondingly. Add a number to change the number of rows to display.
Columns
1
dataframe.columns
Show all of column names of the Dataframe. We can rename columns by assigning new name with = symbol.
Indexes
1
dataframe.index
Show index. It’s a row number by default.
Shape
1
dataframe.shape
Display number of rows and columns of the Dataframe.
Info
1
dataframe.info()
Show the detail of the Dataframe.
Statistics
1
dataframe.describe()
Display basic statistics of the measurable columns of the Dataframe.
Location
1
dataframe.loc[a,b]
Find the values at specific value of Row Index and Column Index.
1
dataframe.iloc[a,b]
Find the values at specific order of Row Index and Column Index.
Sorting
1
dataframe.sort_values()
Sorting rows in the Dataframe.
As the example below, sorting the Dataframe by 'Age' and put it at last when 'Age' is NaN (na_position='last').
Handling Non-values
1
dataframe.dropna()
Remove NaN from the Dataframe. The example removes NaN for a whole row (axis=0) when found any NaN in that row (how='any').
1
dataframe.fillna()
Fill a value if it is NaN. The example fills 0.
Grouping
1
dataframe.groupby()
Grouping Dataframe by given columns. As the example, the result is grouped by values of 'Pclass'.
Managing indexes
1
dataframe.set_index()
Set Row Index by values of a specific column, 'PassengerId' as the example.
1
dataframe.reset_index()
Cancel Row Index and get a new column named “index”.
Example 1
Fill a value into a specific column when NaN.
For example, fill a text into 'Cabin' at the Dataframe itself (inplace=True) when NaN.
1
dataframe.single_column.fillna()
Example 2
Grouping a Dataframe and find the average of a column.
As the example, grouping by 'Pclass' and 'Sex' then find the average 'Age'.
1
dataframe.groupby().single_column.mean()
Full document
Here is the full documentation of Dataframe. We can find much more functionalities there.
matplotlib
matplotlib is the most powerful library to plot a graph quickly. It can be used itself but not since this time that we knew pandas.
1
2
import matplotlib.pyplot as plt
%matplotlib inline
%matplotlib inline is to display the graphs on the Jupyter or we have to call function show().
We will create a graph from the Dataframe Titanic.
1. How about the age of the passengers?
1
titanic[['Age']].plot.hist()
Histogram is a frequency graph. We can conclude most passengers are 20-30 years old.
2. Between male and female, which is more?
1
titanic['Sex'].value_counts().plot(kind='bar')
value_counts() is to count a number of each unique values in a Series. We use it to get a number of each sex then plot the bar graph.
3. Ratio of sex and survivability
1
titanic[['Sex', 'Survived']].groupby(['Sex', 'Survived']).Survived.count().plot.pie()
We now use pie chart as it suits ratio visualization. Our data is a group of 'Sex' and 'Survived' then count a number of each group before plotting.
4. Relationship of age and fare
1
titanic[['Age', 'Fare']].plot.scatter(x='Age', y='Fare')
We build a scatter graph by assign X-axis as 'Age' while Y-axis is 'Fare'.
See you next time.
Bye~