Num2Num Stats

Bivariate Statistics: Num 2 Num

Correlation

Wisdom

Age

Independent and Dependent Variables

• Dependent variable
  • What you want to predict or explain
• Independent variable
  • the variable that explains or predicts the outcome of the dependent variable.
• Age = Independent variable
• Wisdom = Dependent variable

Correlation Measurement

• How much wisdom can be explained by age?

Pearson Correlation Coefficient: r

• Measures the strength and direction of the linear relationship between two variables.
• Ranges from -1 to 1
• 0 means no correlation
• 1 means perfect positive correlation
• -1 means perfect negative correlation

Pearson coefficient formula

\[ r_{x,y} = \frac{\sum_{i=1}^{n} (X_i - \bar{X})(Y_i - \bar{Y})}{\sqrt{\sum_{i=1}^{n} (X_i - \bar{X})^2 \sum_{i=1}^{n} (Y_i - \bar{Y})^2}} \]

• numerator: sum of the product of the differences between each pair of observations and the mean of each variable
• denominator: square root of the product of the variance of each variable

Pearson Coefficient Output

• Positive number means positive correlation
  • As the independent variable increases, the dependent variable also increases.
• Negative number means negative correlation
  • As the independent variable increases, the dependent variable decreases.

Qualifications

• Must be continuous data.
  • not ranked data, ordinal data, or categorical data
• Must have Linear relationship
  • Will not detect variability that expands or contracts at different points.
• Homoscedastic (equal variance)
  • The variability of the dependent variable is the same at all levels of the independent variable.

Skewness, Kurtosis and these assumptions

• Skewness can indicate non-linearity
• Kurtosis can indicate presence of outliers.

Example: Positively correlated

Example: Negatively correlated

Example: No correlation

Probability value

• p-value
• Measures probability that future data will return a 0.0 correlation.
• p-value < 0.05 is typically considered significant.
• p-value > 0.05 is typically considered not significant.
• TL;DR: measures statistical significance of the correlation & lower == better.

Two ways to calculate

• scipy.stats.pearsonr(x, y)
  • includes p-value
• df.corr(numeric_only=True)
  • does not include p-value

Example: df.corr()

import pandas as pd

url = "https://raw.githubusercontent.com/dataprofessor/data/refs/heads/master/penguins_cleaned.csv"

df = pd.read_csv(url)
df.corr(numeric_only=True)

	bill_length_mm	bill_depth_mm	flipper_length_mm	body_mass_g
bill_length_mm	1.000000	-0.228626	0.653096	0.589451
bill_depth_mm	-0.228626	1.000000	-0.577792	-0.472016
flipper_length_mm	0.653096	-0.577792	1.000000	0.872979
body_mass_g	0.589451	-0.472016	0.872979	1.000000

Example: scipy.stats.pearsonr()

import pandas as pd
import scipy.stats as stats

url = "https://raw.githubusercontent.com/dataprofessor/data/refs/heads/master/penguins_cleaned.csv"

df = pd.read_csv(url)
stats.pearsonr(df["bill_length_mm"], df["body_mass_g"])

PearsonRResult(statistic=np.float64(0.5894511101769491), pvalue=np.float64(1.538613514485931e-32))

r, p = stats.pearsonr(df["bill_length_mm"], df["body_mass_g"])
print(f"r: {round(r, 4)}, p: {round(p, 6)}")

r: 0.5895, p: 0.0