Pandas Basics

Python

Pandas

NumPy Arrays

The basic NumPy array looks like this:

import numpy as np

# Create a NumPy array
arr = np.array([1, 2, 3, 4, 5])  # 1D array
arr_2d = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])  # 2d array
for array in arr_2d:
    print(array)
[1 2 3]
[4 5 6]
[7 8 9]

Pandas DataFrames

  • The DataFrame function takes four arguments:
    • data, index, columns, and dtype.
import pandas as pd  # Notice the alias pd for pandas

# Create a Pandas DataFrame
df = pd.DataFrame(data=[[1, 2, 3], [4, 5, 6], [7, 8, 9]])
df
0 1 2
0 1 2 3
1 4 5 6
2 7 8 9

Pandas DataFrames

  • Naming Columns and indices
import pandas as pd
df = pd.DataFrame(data=[[1, 2, 3], [4, 5, 6], [7, 8, 9]], index=['a', 'b', 'c'], columns=['x', 'y', 'z'])

df
x y z
a 1 2 3
b 4 5 6
c 7 8 9

Pandas DataFrames

  • DataFrame from a dictionary
import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)
df
x y z
0 1 4 7
1 2 5 8
2 3 6 9

Renaming columns and indices

import pandas as pd
df.rename(index={0: 'a', 1: 'b', 2: 'c'}, inplace=True)
df = df.rename(columns={'x': 'X', 'y': 'Y', 'z': 'Z'})
df
X Y Z
a 1 4 7
b 2 5 8
c 3 6 9

Accessing Data in Pandas DataFrames

Reading the whole DataFrame

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)
df.values
array([[1, 4, 7],
       [2, 5, 8],
       [3, 6, 9]])

Accessing a column

  • df['column_name'] or df.column_name will return the column as a Pandas Series object.
import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

print("x: ", df['x'])
print("x: ", df.x)
x:  0    1
1    2
2    3
Name: x, dtype: int64
x:  0    1
1    2
2    3
Name: x, dtype: int64

Index notation for Values.

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)


print("x: ", df['x'][0])
x:  1
  • Not the best way to access values in a DataFrame.

Accessing with .loc

  • Use loc
  • df.loc[row, column].
  • We call the name of the index and the name of the column.
import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)
print("0:", df.loc[0, 'x'])
print("\nrow:\n", df.loc[0, :])
print("\ncolumn:\n", df.loc[:, 'x'])
0: 1

row:
 x    1
y    4
z    7
Name: 0, dtype: int64

column:
 0    1
1    2
2    3
Name: x, dtype: int64

Using iloc

import pandas as pd

data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

print("0:", df.iloc[0, 0])
print("\nrow:\n", df.iloc[0, :])
print("\ncolumn:\n", df.iloc[:, 0])
0: 1

row:
 x    1
y    4
z    7
Name: 0, dtype: int64

column:
 0    1
1    2
2    3
Name: x, dtype: int64

loc and iloc are better patterns for accessing data

  • loc or iloc attributes over the index notation because it is more explicit and less prone to errors.
  • Especially better for modifying data.

Using the at and iat attributes

  • Work like loc and iloc, but faster
import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

print("0:", df.at[0, 'x'])
print("0", df.iat[0, 0])
0: 1
0 1

Reassigning Rows, Columns, Values

Individual Values

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

df.at[0, "x"] = 10
df.iat[0, 1] = 11
df.loc[0, "z"] = 12
df.iloc[1, 0] = 13
df["y"][1] = 14
df
x y z
0 10 11 12
1 13 5 8
2 3 6 9

Changing Rows

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

df.loc[2] = [10, 11, 12]
df.iloc[0] = [13, 14, 15]
df
x y z
0 13 14 15
1 2 5 8
2 10 11 12

Adding Rows

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

df.loc["x", :] = [10, 11, 12]
df.iloc[3] = [13, 14, 15] 
df
x y z
0 1.0 4.0 7.0
1 2.0 5.0 8.0
2 3.0 6.0 9.0
x 13.0 14.0 15.0

Filtering DataFrames

  • df[df['column_name'] > value]
  • returns rows where the condition is true.
import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)
filtered = df[df['y']>4]
print(filtered)
   x  y  z
1  2  5  8
2  3  6  9

Filtering based on Columns

We can also combine our operators.

  • & for and
import pandas as pd

data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

filtered = df[(df['y']>4) & (df['z']>8)]
filtered
x y z
2 3 6 9

Combining Filters

  • | for or
import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

filtered = df[(df['y']>4) | (df['z']>8)]
filtered
x y z
1 2 5 8
2 3 6 9

Filtering based on Columns

  • filter columns based on a condition
  • rows that include ‘x’
import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

df.filter(like='x', axis="columns")
x
0 1
1 2
2 3

Sorting DataFrames

  • sort_index()
  • columns
import pandas as pd
data = [[1, 2, 3], [4, 5, 6],[7, 8, 9]]
df = pd.DataFrame(data, index=['b', 'a', 'c'], columns=['y', 'x', 'z'])

df.sort_index(ascending=True, inplace=True, axis="columns")
df
x y z
b 2 1 3
a 5 4 6
c 8 7 9

Sorting DataFrames, ii

  • sort_index()
  • rows
import pandas as pd
data = [[1, 2, 3], [4, 5, 6],[7, 8, 9]]
df = pd.DataFrame(data, index=['b', 'a', 'c'], columns=['y', 'x', 'z'])

df.sort_index(ascending=True, inplace=True, axis="rows")
df
y x z
a 4 5 6
b 1 2 3
c 7 8 9

Sorting DataFrames, iii

  • sort_values()
import pandas as pd
data = [[1, 2, 3], [4, 5, 6],[7, 8, 9]]
df = pd.DataFrame(data, index=['b', 'a', 'c'], columns=['y', 'x', 'z'])

df.sort_values(by=['x'], ascending=False, inplace=True)
df
y x z
c 7 8 9
a 4 5 6
b 1 2 3

Sort Values with Multiple Columns

import pandas as pd

data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
df = pd.DataFrame(data, index=["b", "a", "c"], columns=["y", "x", "z"])

df.sort_values(by=["x", "z"], ascending=False, inplace=True)
df
y x z
c 7 8 9
a 4 5 6
b 1 2 3

Pandas Series Objects

import pandas as pd

# Create a Pandas Series
s = pd.Series([1, 2, 3, 4, 5], index=['a', 'b', 'c', 'd', 'e'], name='numbers')
s
a    1
b    2
c    3
d    4
e    5
Name: numbers, dtype: int64

We can do much of the above on a Pandas Series.

Iterating

  • iterrows()
  • itertuples()
  • items()

Iterating: iterrows

import pandas as pd

data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

for i, row in df.iterrows():
    print(i, row)
0 x    1
y    4
z    7
Name: 0, dtype: int64
1 x    2
y    5
z    8
Name: 1, dtype: int64
2 x    3
y    6
z    9
Name: 2, dtype: int64

Iterating: itertuples

import pandas as pd

data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

for row in df.itertuples():
    print(row.x, row.y, row.z)
1 4 7
2 5 8
3 6 9

Iterating: items

import pandas as pd

data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

for row in df.items():
    print(row[1])
0    1
1    2
2    3
Name: x, dtype: int64
0    4
1    5
2    6
Name: y, dtype: int64
0    7
1    8
2    9
Name: z, dtype: int64