Lecture 2 - Choice Models and its Applications
Evaluating Performance: Statistical Tests (continued)
- A constrained case, as a baseline, can be the case where the \(Y\) has no predictors (i.e, all \(\beta\)’s are equal to zero)
Confusion Matrix
- Green cells indicate the True Positive and True Negative cases
- Red cells indicate the False Positive and False Negative cases
- Type I Errors (153) are the False Positive cases: we wrongly classified customers that did not churn as churned!
- Type II Errors (1788) are the False Negative cases: we wrongly classified customers that did churn as not churned!
Putting all together
- As we increase our threshold, \(p^\star\), we minimize the Type II error (i.e, we identify the churned customers) as we’re identifying the true positives
- On the other hand, at the same time, we are increasing the Type I error, since there is going to be a higher number of false positives!
Finding the optimal \(p^{\star}\)
- If, for example, we want to find \(p^{\star}\) such that it maximizes the sum of specificity + sensitivity, we can:
- Redo our confusion matrix for each \(p^{\star}=\{0,0.01,0.02,...,0.99,1\}\)
- Calculate the specificity and the sensitivity
- Pick the threshold that maximizes the sum of both metrics
- Note that, depending upon the problem, we might want to use a different criterion to find the optimal \(p^\star\)
Show the code
Data=read.csv('Exercises/bank-dataset.csv')
Reg=glm(churn ~ credit_score + gender + age + tenure + balance + products_number + credit_card + active_member + estimated_salary,
family = binomial(link = "logit"),
data = Data)
#Creating a comparison between actual and predicted churn
Comparison=data.frame(Actual=Data$churn,
Predicted=ifelse(predict(Reg,type='response')>=mean(Data$churn),1,0))
Coords=coords(ROC,print.thres=TRUE)%>%mutate(metric=sensitivity+specificity)
Best=pull(coords(ROC,print.thres=TRUE,"best")[1])
Coords%>%
ggplot(aes(x=threshold,y=metric))+
geom_point()+
theme_minimal()+
labs(y='Specificity + Sensitivity',x='Threshold for churn assignment',
title=paste0('Best value achieved using threshold = ',round(Best,2)))+
geom_vline(xintercept = pull(coords(ROC,print.thres=TRUE,"best")[1]),linetype='dashed')+
theme(title=element_text(size=20,face='bold'),
axis.text = element_text(size=15),
axis.title = element_text(size=15))Show the code
# Read the CSV file into a pandas DataFrame
Data = pd.read_csv('Exercises/bank-dataset.csv')
# Convert 'gender' into dummy variables
Data = pd.get_dummies(Data, columns=['gender'], drop_first=True)
# Define independent and dependent variables
X = Data[['credit_score', 'gender_Male', 'age', 'tenure', 'balance', 'products_number', 'credit_card', 'active_member', 'estimated_salary']]
y = Data['churn']
# Add a constant to the independent variables matrix for the intercept
X = sm.add_constant(X)
# Fit the linear regression model
model = sm.Logit(y, X.astype(float)).fit()
# Predict churn probabilities
probs = model.predict()
# Calculate specificity and sensitivity
fpr, tpr, thresholds = roc_curve(y, probs)
specificity = 1 - fpr
sensitivity = tpr
# Find the best threshold value
best_threshold_index = np.argmax(specificity + sensitivity)
best_threshold = thresholds[best_threshold_index]
# Visualize the results
plt.figure(figsize=(10, 6))
plt.plot(thresholds, sensitivity + specificity, marker='o')
plt.axvline(x=best_threshold, color='r', linestyle='--', label=f'Best Threshold: {best_threshold:.2f}')
plt.xlabel('Threshold for churn assignment')
plt.ylabel('Specificity + Sensitivity')
plt.title(f'Best value achieved using threshold = {best_threshold:.2f}')
plt.legend()
plt.grid(True)
plt.show()Comparing among different classifiers
- Say that, for some reason, you have ommitted
agefrom the analysis. How can you assess how the predictive power of your model is going to deteriorate? One way to do this is to analyze what we call Area under the Curve (or simply AUC):
AUC under different models
- An ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. This curve plots two parameters:
- True Positive Rate
- False Positive Rate
AUC provides an aggregate measure of performance across all possible classification thresholds
AUC isn’t a very useful metric whenever the costs of Type I and Type II errors are very different
Show the code
Data=read.csv('Exercises/bank-dataset.csv')
Reg_1=glm(churn ~ credit_score + gender + tenure + balance + products_number + credit_card + active_member + estimated_salary,
family = binomial(link = "logit"),
data = Data)
Reg_2=glm(churn ~ credit_score + gender + age + tenure + balance + products_number + credit_card + active_member + estimated_salary,
family = binomial(link = "logit"),
data = Data)
P_1=Data%>%cbind(data.frame('Predicted'=predict(Reg_1,type='response')))
P_2=Data%>%cbind(data.frame('Predicted'=predict(Reg_2,type='response')))
ROC_1 = roc(P_1$churn ~ P_1$Predicted)
ROC_2 = roc(P_2$churn ~ P_2$Predicted)
ROC_1=coords(ROC_1)%>%mutate(TPR=sensitivity,FPR=1-specificity,type='Without Age')
ROC_2=coords(ROC_2)%>%mutate(TPR=sensitivity,FPR=1-specificity,type='With Age')
rbind(ROC_1,ROC_2)%>%
ggplot(aes(x=FPR,y=TPR,col=type))+
geom_point()+
theme_minimal()+
labs(x='FPR (1 - Specificity)',
y='TPR (Sensitivity)',
title='Model with Age outperforms the other for any threshold!')+
theme(plot.title = element_text(size=40,face='bold'),
axis.text = element_text(size=30),
axis.title = element_text(size=30),
legend.position ='bottom')Show the code
import pandas as pd
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_curve, roc_auc_score
import matplotlib.pyplot as plt
# Read the CSV file into a pandas DataFrame
Data = pd.read_csv('Exercises/bank-dataset.csv')
# Convert 'gender' into dummy variables
Data['gender'] = pd.get_dummies(Data['gender'], drop_first=True)
# Define independent and dependent variables for Reg_1 (without 'age')
X1 = Data[['credit_score', 'gender', 'tenure', 'balance', 'products_number', 'credit_card', 'active_member', 'estimated_salary']]
# Define independent and dependent variables for Reg_2 (with 'age')
X2 = Data[['credit_score', 'gender', 'age', 'tenure', 'balance', 'products_number', 'credit_card', 'active_member', 'estimated_salary']]
# Dependent variable
y = Data['churn']
# Fit logistic regression models
reg_1 = model = sm.Logit(y, X1.astype(float)).fit()
reg_2 = model = sm.Logit(y, X2.astype(float)).fit()
# Predict probabilities for both models
probs_1 = reg_1.predict()
probs_2 = reg_2.predict()
# Compute ROC curves
fpr_1, tpr_1, _ = roc_curve(y, probs_1)
fpr_2, tpr_2, _ = roc_curve(y, probs_2)
# Compute AUC scores
auc_1 = roc_auc_score(y, probs_1)
auc_2 = roc_auc_score(y, probs_2)
# Plot ROC curves
plt.figure(figsize=(10, 8))
plt.plot(fpr_1, tpr_1, color='blue', lw=2, label=f'Without Age (AUC = {auc_1:.2f})')
plt.plot(fpr_2, tpr_2, color='red', lw=2, label=f'With Age (AUC = {auc_2:.2f})')
plt.plot([0, 1], [0, 1], color='gray', lw=1, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate (1 - Specificity)')
plt.ylabel('True Positive Rate (Sensitivity)')
plt.title('ROC Curve')
plt.legend(loc="lower right")
plt.show()References
