Classification Metrics: Accuracy in Jun 2024 by Jatin7 #metrics

Basically accuracy → total correct prediction/ total predictions

• When data is imbalanced → Accuracy is a bad metric
• Fails to capture class-wise (granular → failing to classify spam data) performance

fully understand the confusion matrix for this binary class classification problem, we first need to get familiar with the following terms:

• True Positive (TP) refers to a sample belonging to the positive class being classified correctly.
• True Negative (TN) refers to a sample belonging to the negative class being classified correctly.
• False Positive (FP) refers to a sample belonging to the negative class but being classified wrongly as belonging to the positive class.
• False Negative (FN) refers to a sample belonging to the positive class but being classified wrongly as belonging to the negative class.

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
y_pred = model.predict(X_test)
conf_matrix = confusion_matrix(y_test, y_pred)
conf_matrix 

Precision:-

While the scenario where

True Positives+False Positives=0 is a primary reason for adding a small value epsilon(ϵ)

In many algorithms, precision may become zero or extremely small. Dividing by zero or a minimal number can cause numerical instability or errors. Adding a small ϵ\epsilonϵ ensures that the denominator is never zero.

• High Accuracy: This means that the model is correctly predicting a high percentage of the total instances.
• Low Precision: This indicates that among the positive predictions made by the model, a significant number are actually false positives.

Implications:

• False Positives: The model is predicting many instances as positive, but a substantial portion of these are incorrect.
• Use Case Sensitivity: This scenario might be problematic in situations where false positives are costly or harmful. For example, in medical diagnosis, predicting a disease when it is not present (false positive) can lead to unnecessary stress and treatment for patients.

Example: In spam detection, if the model has high accuracy but low precision, it might label many legitimate emails as spam, which can be frustrating for users.

• Low Accuracy: This indicates that the model is not predicting the total instances correctly very often.
• High Precision: This means that when the model predicts a positive instance, it is usually correct.

Implications:

• False Negatives: The model might be missing a lot of actual positive instances (high false negatives), leading to low overall accuracy.
• Use Case Sensitivity: This might be acceptable in scenarios where false positives are highly undesirable, but missing some positives is acceptable. For example, in fraud detection, you might prefer a system that rarely flags legitimate transactions as fraudulent (high precision) even if it misses some fraudulent ones (lower accuracy).

Example: In cancer detection, if the model has low accuracy but high precision, it may not detect all cancer cases (high false negatives), but when it does detect one, it is usually correct, which might still be useful depending on the context.

• Context Matters: The importance of accuracy versus precision depends on the specific problem and the consequences of false positives versus false negatives.
• Other Metrics: Consider other metrics like recall (sensitivity), F1 score (harmonic mean of precision and recall), and the overall context of the use case.

1. Medical Diagnosis:

• High Precision, Low Accuracy: Better if the cost of false positives is high (e.g., unnecessary treatment).
• High Accuracy, Low Precision: Better if the cost of missing a disease is high (e.g., missing a cancer diagnosis).

2. Email Spam Detection:

• High Precision, Low Accuracy: Preferred to avoid marking legitimate emails as spam.
• High Accuracy, Low Precision: Might mark too many legitimate emails as spam, leading to user frustration.

from sklearn.metrics import recall_score
recall_score(y_test, y_pred)

• Type I Error (False Positive): This error occurs when the model incorrectly predicts a positive outcome for a negative instance. For example, in medical diagnosis, it would mean diagnosing a patient with a disease they do not have.
• Type II Error (False Negative): This error occurs when the model incorrectly predicts a negative outcome for a positive instance. For example, failing to diagnose a patient who actually has a disease.

• Recall (Sensitivity, True Positive Rate):

Recall measures the ability of a model to identify all relevant instances (i.e., it focuses on minimizing Type II errors).

High Recall: Indicates that the model is good at finding most of the positive instances, even if it means including some false positives.

Precision measures the ability of a model to accurately identify positive instances (i.e., it focuses on minimizing Type I errors).

High Precision: Indicates that the model’s positive predictions are mostly correct, even if it misses some true positives.

When Minimizing Type I Errors (False Positives):

• Use Precision: Precision is crucial when the cost of a false positive is high. You want to ensure that when the model predicts a positive, it is very likely to be correct.

Example Scenarios:

• Spam Detection: Marking legitimate emails as spam (Type I error) can be very disruptive.
• Medical Testing for Rare but Serious Conditions: A false positive might lead to unnecessary stress and treatment.

When Minimizing Type II Errors (False Negatives):

• Use Recall: Recall is crucial when the cost of a false negative is high. You want to ensure that the model captures as many positive instances as possible, even if it includes some false positives.

Example Scenarios:

• Cancer Screening: Missing a cancer diagnosis (Type II error) can be life-threatening.
• Security: Missing a threat (e.g., an intruder or malware) can have severe consequences.

Practical Considerations

• High Recall, Low Precision: Suitable when missing a positive instance is more costly than a false positive. For example, in medical diagnostics, you might prefer to err on the side of caution.
• High Precision, Low Recall: Suitable when false positives are very costly or disruptive. For example, in spam detection, you might want to avoid flagging legitimate emails as spam even if it means some spam gets through.

• F1 Score: Sometimes, a balance between precision and recall is desired.

Why do we take harmonic mean in the F1 score instead of
arithmetic mean?
Harmonic Mean penalizes the reduction in Precision and Recall more than
Arithmetic Mean.

• The F1 score is useful when you need a balance between precision and recall, especially when the class distribution is imbalanced.

F1 — Beta:

• When it is more important to optimize recall than precision
• beta = 2
• we might be interested in an F-measure with more attention put on precision,
• beta = 0.5

Some important resource blog :- https://neptune.ai/blog/evaluation-metrics-binary-classification

video unfload data science :- https://youtube.com/watch?v=V-zmQDtd25k

campus x video : — https://www.youtube.com/watch?v=gdW6hj9IXaA&t=3s