Pandas Tutorial


exds_00: Getting Started

Introduction:

Now more than ever, data has a unique ability to describe the many facets of our reality. Thanks to relatively recent (and accelerating) advances in computing,those with proper motivation can use data to affect change in any realm they so choose, from ones more directly related to computer science such as machine learning to others with not so obvious connections such as biology and economics!

You have likely heard a lot of talk recently about data science, but what exactly is data science? A quick search gives this definition: “an inter-disciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from structured and unstructured data, and apply knowledge from data across a broad range of application domains”. Now, this is a mouthful, but essentially, data science pulls from the best of computer science, statistics, and mathematics to make data understandable and actionable. This is the key, because what worth does data have if we can not interpret it and make decisions based on it?

pandas and Data Science in Python:

Learning Python in this class has already given you a headstart in data science, since Python is incredibly well-equipped for handling large datasets with many specialized third-party libraries. The industry standard for this type of work is a library called pandas. There are other libraries that work well with data science in Python that we may explore later, but pandas is all you need for now! :)

Below, you will see the usual convention for importing pandas:

Importing pandas

Now, that was not so bad! :) but we have a ways to go…

The End-all Be-all of pandas: the DataFrame

In pandas, the primary data type that you will work is known as a DataFrame. Those who have worked in R may know this by the same name, and those who have in the SAS programming lanaguage may know this as a data set! Here, you will see a declaration of a DataFrame in pandas:

   col1  col2
0     1     3
1     2     4

You might have noticed something very interesting, a marked similarity between this declaration and something you have worked with before, perhaps in EX04! Full disclosure, this is entirely purposeful. EX04 was meant to help you understand the underlying functionality of a DataFrame in pandas! Many things work similarly in pandas to how you implemented them in EX04. But, more on that later!

Importing Data with pandas

It stands to reason that, to work with data, we need data! In pandas, we can read in many kinds of files, but the most common for datasets is the csv or “comma-separated value” file. The dataset imported below contains information about some Pokemon! To view your data, you should use a command of the form print(dataframe.head(n)), where n is the number of top rows you wish to display. If left blank, this will default to 5.

Additionally, if we choose to modify the DataFrame, we can actually write to a new csv that will include the modifications using a similar command to the one you will see below. Do NOT worry if the code for modification seems foreign, we will get to that in time, but the gist of it is that we are ordering or sorting the pokemon DataFrame by ascending values of HP! It is worth noting that we could write back to the original csv, but in some cases it may be worth preserving the original dataset.

Summarizing the data in your DataFrame

After we load the data, the structure of our DataFrame may not be immediately obvious, and usually using a command such as df.head() works totally fine. But, sometimes we need a bit more information to understand our data before we attempt to analyze it. See the code below for some commands that may be useful in interpreting the structure of your DataFrame.

800
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 800 entries, 0 to 799
Data columns (total 14 columns):
 #   Column      Non-Null Count  Dtype 
---  ------      --------------  ----- 
 0   Unnamed: 0  800 non-null    int64 
 1   #           800 non-null    int64 
 2   Name        800 non-null    object
 3   Type 1      800 non-null    object
 4   Type 2      414 non-null    object
 5   Total       800 non-null    int64 
 6   HP          800 non-null    int64 
 7   Attack      800 non-null    int64 
 8   Defense     800 non-null    int64 
 9   Sp. Atk     800 non-null    int64 
 10  Sp. Def     800 non-null    int64 
 11  Speed       800 non-null    int64 
 12  Generation  800 non-null    int64 
 13  Legendary   800 non-null    bool  
dtypes: bool(1), int64(10), object(3)
memory usage: 82.2+ KB

Subsetting by variable (column) in a DataFrame

Of course, having the csv loaded is great, but there is not much use in a DataFrame if we can not actually get to the data and perform meaningful analyses. So, we will start by selecting columns! In pandas, there several ways to do this. See the code below and read the comments for insight into the action of each command and note the addition of another library called numpy, which will help us perform some basic analysis on the data we collect.

Something of note is that the very last command uses something called a regular expression. We do not cover this topic in COMP 110, but they are quite useful and something you will learn more about in later COMP courses if you so choose to take them! :)

69.25875
31.164004777146342
                    Name  HP  Attack  Defense
0              Bulbasaur  45      49       49
1                Ivysaur  60      62       63
2               Venusaur  80      82       83
3  VenusaurMega Venusaur  80     100      123
4             Charmander  39      52       43
                    Name Type 1  Type 2
0              Bulbasaur  Grass  Poison
1                Ivysaur  Grass  Poison
2               Venusaur  Grass  Poison
3  VenusaurMega Venusaur  Grass  Poison
4             Charmander   Fire     NaN
                    Name  HP  Attack  Defense
0              Bulbasaur  45      49       49
1                Ivysaur  60      62       63
2               Venusaur  80      82       83
3  VenusaurMega Venusaur  80     100      123
4             Charmander  39      52       43
  Type 1  Type 2  Total
0  Grass  Poison    318
1  Grass  Poison    405
2  Grass  Poison    525
3  Grass  Poison    625
4   Fire     NaN    309
   Unnamed: 0  #  Total  HP  Attack  Defense  Sp. Atk  Sp. Def  Speed  \
0           0  1    318  45      49       49       65       65     45   
1           1  2    405  60      62       63       80       80     60   
2           2  3    525  80      82       83      100      100     80   
3           3  3    625  80     100      123      122      120     80   
4           4  4    309  39      52       43       60       50     65   

   Generation  
0           1  
1           1  
2           1  
3           1  
4           1  

Subsetting by observation (row) in a DataFrame

Next, we will focus on how to subset data by row. This is where Boolean expressions come into play since we typically want the rows we filter to meet a specific requirement such as one of its values falling within a range of numbers or being equal to some particular string. In this section, you will see how to generate a sample from larger dataset. However, pay particular attention to the use of Boolean conditions to filter rows of the DataFrame, as this is the most common and readable method in pandas.

   Unnamed: 0  #                   Name Type 1  Type 2  Total  HP  Attack  \
0           0  1              Bulbasaur  Grass  Poison    318  45      49   
1           1  2                Ivysaur  Grass  Poison    405  60      62   
2           2  3               Venusaur  Grass  Poison    525  80      82   
3           3  3  VenusaurMega Venusaur  Grass  Poison    625  80     100   
4           4  4             Charmander   Fire     NaN    309  39      52   

   Defense  Sp. Atk  Sp. Def  Speed  Generation  Legendary  
0       49       65       65     45           1      False  
1       63       80       80     60           1      False  
2       83      100      100     80           1      False  
3      123      122      120     80           1      False  
4       43       60       50     65           1      False  
     Unnamed: 0    #                 Name   Type 1 Type 2  Total  HP  Attack  \
795         795  719              Diancie     Rock  Fairy    600  50     100   
796         796  719  DiancieMega Diancie     Rock  Fairy    700  50     160   
797         797  720  HoopaHoopa Confined  Psychic  Ghost    600  80     110   
798         798  720   HoopaHoopa Unbound  Psychic   Dark    680  80     160   
799         799  721            Volcanion     Fire  Water    600  80     110   

     Defense  Sp. Atk  Sp. Def  Speed  Generation  Legendary  
795      150      100      150     50           6       True  
796      110      160      110    110           6       True  
797       60      150      130     70           6       True  
798       60      170      130     80           6       True  
799      120      130       90     70           6       True  
     Unnamed: 0    #        Name  Type 1 Type 2  Total   HP  Attack  Defense  \
44           44   39  Jigglypuff  Normal  Fairy    270  115      45       20   
45           45   40  Wigglytuff  Normal  Fairy    435  140      70       45   
96           96   89         Muk  Poison    NaN    500  105     105       75   
120         120  112      Rhydon  Ground   Rock    485  105     130      120   
121         121  113     Chansey  Normal    NaN    450  250       5        5   

     Sp. Atk  Sp. Def  Speed  Generation  Legendary  
44        45       25     20           1      False  
45        85       50     45           1      False  
96        65      100     50           1      False  
120       45       45     40           1      False  
121       35      105     50           1      False  
   Unnamed: 0  #                   Name Type 1  Type 2  Total  HP  Attack  \
0           0  1              Bulbasaur  Grass  Poison    318  45      49   
1           1  2                Ivysaur  Grass  Poison    405  60      62   
2           2  3               Venusaur  Grass  Poison    525  80      82   
3           3  3  VenusaurMega Venusaur  Grass  Poison    625  80     100   
4           4  4             Charmander   Fire     NaN    309  39      52   

   Defense  Sp. Atk  Sp. Def  Speed  Generation  Legendary  
0       49       65       65     45           1      False  
1       63       80       80     60           1      False  
2       83      100      100     80           1      False  
3      123      122      120     80           1      False  
4       43       60       50     65           1      False  
     Unnamed: 0    #        Name  Type 1    Type 2  Total  HP  Attack  \
720         720  652  Chesnaught   Grass  Fighting    530  88     107   
249         249  230     Kingdra   Water    Dragon    540  75      95   
160         160  148   Dragonair  Dragon       NaN    420  61      84   
659         659  598  Ferrothorn   Grass     Steel    489  74      94   
461         461  414      Mothim     Bug    Flying    424  70      94   

     Defense  Sp. Atk  Sp. Def  Speed  Generation  Legendary  
720      122       74       75     64           6      False  
249       95       95       95     85           2      False  
160       65       70       70     70           1      False  
659      131       54      116     20           5      False  
461       50       94       50     66           4      False  
     Unnamed: 0    #                 Name  Type 1    Type 2  Total   HP  \
529         529  477             Dusknoir   Ghost       NaN    525   45   
585         585  526             Gigalith    Rock       NaN    515   85   
787         787  711  GourgeistSuper Size   Ghost     Grass    494   85   
777         777  707               Klefki   Steel     Fairy    470   57   
363         363  332             Cacturne   Grass      Dark    475   70   
398         398  363               Spheal     Ice     Water    290   70   
313         313  289              Slaking  Normal       NaN    670  150   
401         401  366             Clamperl   Water       NaN    345   35   
277         277  256            Combusken    Fire  Fighting    405   60   
254         254  235             Smeargle  Normal       NaN    250   55   

     Attack  Defense  Sp. Atk  Sp. Def  Speed  Generation  Legendary  
529     100      135       65      135     45           4      False  
585     135      130       60       80     25           5      False  
787     100      122       58       75     54           6      False  
777      80       91       80       87     75           6      False  
363     115       60      115       60     55           3      False  
398      40       50       55       50     25           3      False  
313     160      100       95       65    100           3      False  
401      64       85       74       55     32           3      False  
277      85       60       85       60     55           3      False  
254      20       35       20       45     75           2      False  

Conclusion of Tutorial:

To wrap things up, let’s recall what we have learned so far:

  • importing pandas
  • creating a DataFrame, underlying structure
  • reading from and writing to .csv files
  • observing the structure of a specific DataFrame
  • selecting data by column and by row
  • some basic analysis using numpy

Congratulations on taking the first step in your data science journey. If you so choose, you can try out the Data Science Challenge in this folder which will push you to apply the skills you learned in this tutorial and do some of your own analysis on a dataset! Great work and good luck to all of you future data scientists! :)