Pandas Basics

Pandas

NumPy Arrays

The basic NumPy array looks like this:

import numpy as np

# Create a NumPy array
arr = np.array([1, 2, 3, 4, 5])  # 1D array
arr_2d = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])  # 2d array
for array in arr_2d:
    print(array)

[1 2 3]
[4 5 6]
[7 8 9]

Pandas DataFrames

• The DataFrame function takes four arguments:
  • data, index, columns, and dtype.

import pandas as pd  # Notice the alias pd for pandas

# Create a Pandas DataFrame
df = pd.DataFrame(data=[[1, 2, 3], [4, 5, 6], [7, 8, 9]])
df

	0	1	2
0	1	2	3
1	4	5	6
2	7	8	9

Pandas DataFrames

• Naming Columns and indices

import pandas as pd
df = pd.DataFrame(data=[[1, 2, 3], [4, 5, 6], [7, 8, 9]], index=['a', 'b', 'c'], columns=['x', 'y', 'z'])

df

	x	y	z
a	1	2	3
b	4	5	6
c	7	8	9

Pandas DataFrames

• DataFrame from a dictionary

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)
df

	x	y	z
0	1	4	7
1	2	5	8
2	3	6	9

Renaming columns and indices

import pandas as pd
df.rename(index={0: 'a', 1: 'b', 2: 'c'}, inplace=True)
df = df.rename(columns={'x': 'X', 'y': 'Y', 'z': 'Z'})
df

	X	Y	Z
a	1	4	7
b	2	5	8
c	3	6	9

Accessing Data in Pandas DataFrames

Reading the whole DataFrame

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)
df.values

array([[1, 4, 7],
       [2, 5, 8],
       [3, 6, 9]])

Accessing a column

• df['column_name'] or df.column_name will return the column as a Pandas Series object.

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

print("x: ", df['x'])
print("x: ", df.x)

x:  0    1
1    2
2    3
Name: x, dtype: int64
x:  0    1
1    2
2    3
Name: x, dtype: int64

Index notation for Values.

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)


print("x: ", df['x'][0])

x:  1

• Not the best way to access values in a DataFrame.

Accessing with .loc

• Use loc
• df.loc[row, column].
• We call the name of the index and the name of the column.

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)
print("0:", df.loc[0, 'x'])
print("\nrow:\n", df.loc[0, :])
print("\ncolumn:\n", df.loc[:, 'x'])

0: 1

row:
 x    1
y    4
z    7
Name: 0, dtype: int64

column:
 0    1
1    2
2    3
Name: x, dtype: int64

Using iloc

import pandas as pd

data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

print("0:", df.iloc[0, 0])
print("\nrow:\n", df.iloc[0, :])
print("\ncolumn:\n", df.iloc[:, 0])

0: 1

row:
 x    1
y    4
z    7
Name: 0, dtype: int64

column:
 0    1
1    2
2    3
Name: x, dtype: int64

loc and iloc are better patterns for accessing data

• loc or iloc attributes over the index notation because it is more explicit and less prone to errors.
• Especially better for modifying data.

Using the at and iat attributes

• Work like loc and iloc, but faster

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

print("0:", df.at[0, 'x'])
print("0", df.iat[0, 0])

0: 1
0 1

Reassigning Rows, Columns, Values

Individual Values

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

df.at[0, "x"] = 10
df.iat[0, 1] = 11
df.loc[0, "z"] = 12
df.iloc[1, 0] = 13
df["y"][1] = 14
df

/tmp/ipykernel_504978/4257647607.py:9: ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment.
Such chained assignment never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

Try using '.loc[row_indexer, col_indexer] = value' instead, to perform the assignment in a single step.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html#chained-assignment
  df["y"][1] = 14

	x	y	z
0	10	11	12
1	13	5	8
2	3	6	9

Changing Rows

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

df.loc[2] = [10, 11, 12]
df.iloc[0] = [13, 14, 15]
df

	x	y	z
0	13	14	15
1	2	5	8
2	10	11	12

Adding Rows

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

df.loc["x", :] = [10, 11, 12]
df.iloc[3] = [13, 14, 15] 
df

	x	y	z
0	1.0	4.0	7.0
1	2.0	5.0	8.0
2	3.0	6.0	9.0
x	13.0	14.0	15.0

Filtering DataFrames

• df[df['column_name'] > value]
• returns rows where the condition is true.

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)
filtered = df[df['y']>4]
print(filtered)

   x  y  z
1  2  5  8
2  3  6  9

Filtering based on Columns

We can also combine our operators.

• & for and

import pandas as pd

data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

filtered = df[(df['y']>4) & (df['z']>8)]
filtered

	x	y	z
2	3	6	9

Combining Filters

• | for or

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

filtered = df[(df['y']>4) | (df['z']>8)]
filtered

	x	y	z
1	2	5	8
2	3	6	9

Filtering based on Columns

• filter columns based on a condition
• rows that include ‘x’

import pandas as pd
data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

df.filter(like='x', axis="columns")

	x
0	1
1	2
2	3

Sorting DataFrames

• sort_index()
• columns

import pandas as pd
data = [[1, 2, 3], [4, 5, 6],[7, 8, 9]]
df = pd.DataFrame(data, index=['b', 'a', 'c'], columns=['y', 'x', 'z'])

df.sort_index(ascending=True, inplace=True, axis="columns")
df

	x	y	z
b	2	1	3
a	5	4	6
c	8	7	9

Sorting DataFrames, ii

• sort_index()
• rows

import pandas as pd
data = [[1, 2, 3], [4, 5, 6],[7, 8, 9]]
df = pd.DataFrame(data, index=['b', 'a', 'c'], columns=['y', 'x', 'z'])

df.sort_index(ascending=True, inplace=True, axis="rows")
df

	y	x	z
a	4	5	6
b	1	2	3
c	7	8	9

Sorting DataFrames, iii

• sort_values()

import pandas as pd
data = [[1, 2, 3], [4, 5, 6],[7, 8, 9]]
df = pd.DataFrame(data, index=['b', 'a', 'c'], columns=['y', 'x', 'z'])

df.sort_values(by=['x'], ascending=False, inplace=True)
df

	y	x	z
c	7	8	9
a	4	5	6
b	1	2	3

Sort Values with Multiple Columns

import pandas as pd

data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
df = pd.DataFrame(data, index=["b", "a", "c"], columns=["y", "x", "z"])

df.sort_values(by=["x", "z"], ascending=False, inplace=True)
df

	y	x	z
c	7	8	9
a	4	5	6
b	1	2	3

Pandas Series Objects

import pandas as pd

# Create a Pandas Series
s = pd.Series([1, 2, 3, 4, 5], index=['a', 'b', 'c', 'd', 'e'], name='numbers')
s

a    1
b    2
c    3
d    4
e    5
Name: numbers, dtype: int64

We can do much of the above on a Pandas Series.

Iterating

• iterrows()
• itertuples()
• items()

Iterating: iterrows

import pandas as pd

data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

for i, row in df.iterrows():
    print(i, row)

0 x    1
y    4
z    7
Name: 0, dtype: int64
1 x    2
y    5
z    8
Name: 1, dtype: int64
2 x    3
y    6
z    9
Name: 2, dtype: int64

Iterating: itertuples

import pandas as pd

data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

for row in df.itertuples():
    print(row.x, row.y, row.z)

1 4 7
2 5 8
3 6 9

Iterating: items

import pandas as pd

data = {"x": [1, 2, 3], "y": [4, 5, 6], "z": [7, 8, 9]}
df = pd.DataFrame(data)

for row in df.items():
    print(row[1])

0    1
1    2
2    3
Name: x, dtype: int64
0    4
1    5
2    6
Name: y, dtype: int64
0    7
1    8
2    9
Name: z, dtype: int64