Confused with Confusion Matrix?
Can we go ahead with a model having an accuracy score of 90%?
There are a lot of matrices that can help us to judge the performance of a model. Some of the commonly used matrices are Accuracy, Confusion matrix, recall, etc.
Consider a scenario where we have a classification algorithm. We have 80 data for the train and 10 for the test. Once we train the model, we test it on our test data. After executing the model, let's compare the actual data and the data predicted by the model.
Above are the test data and the predicted data. Here we can see or model has given the wrong prediction 2 times(the wrong prediction is marked in red).
Based on the above scenario, we can calculate Accuracy = (Total number of correct predicted values/ Total number of predicted values).
In our case, we have 10 predicted values and there are 2 wrong predicted values, so the remaining correct value is 8. Hence, Accuracy = 8/10 = 0.8. Accuracy score of 0.8 means for every 100 data the model will perform 80 correct predictions.
Confusion Matrix
In order to understand how the Confusion matrix works, let's assume the case of a heart disease patient. Based on the above parameters, let's understand it using 1 and 0 from Actual and predicted values. So 1 in predicted value means heart disease is predicted, while 0 means heart disease is not predicted. Similarly to the Actual value, 1 is the person is having heart disease while 0 is no heart disease.
True Positive: Its position is between 1 of Predicted and 1 of Actual value, i.e it's true for correct prediction and true for Actual value. For true positive, the person has heart disease and the model has performed a correct prediction.
False Negative(Type 2 Error): Positioned between 1 of the Actual and 0 of Predicted, it says the modal has predicted no heart disease for the patient, but he had heart disease.
False Positive(Type 1 Error): Positioned between 0 of Actual and 1 of Predicted, it says the modal has predicted heart disease, but actually they don't have any heart disease.
True Negative: Positioned between 0 of Actual and 0 of Predicted, it says the modal has predicted there is no heart disease and in actual, there was no heart disease.
Accuracy in Confusion matrix = (TP+TN)/(TP+TN+FP+FN)
Let us consider a case wherein we have a confusion matrix available for 2 models having the same accuracy.
In the above 2 models, we can see the Accuracy of both are same. Now in this case, how can we choose the best model? In this case lets consider the value of a False positive. In our earlier case, a false negative does not predict heart disease while the person is suffering from heart disease, hence we will deploy those model which has a minimum number of false negative values.
The above scenario is called Recall where we find out the actual positives which are correctly classified.Recall = TP/(TP+FN). In simple words, out of all the people whose heart disease has been detected, how many have been correctly detected?
Consider the same confusion matrix values above for predicting food orders. Here, a False Positive in the model has predicted the customer will place an order, while in actuality he has not placed any order. So the restaurant will face a loss of food as he is preparing more food and no one is ordering. Hence, in this case, he wants the false positive should be the least. The above scenario is called Precision as it tells what proportion of predicted positives are truly positives.Precision = TP/(TP+FP).
There is one more scenario where we do not know if the True positive should be less or the False positive should be less. In this case, we will use F1 Score. F1 Score = 2 (Recall * Precision)/(Precision+Recall).
Can we go ahead with a model having an accuracy score of 90%?
After going through recall and precision we can understand there are scenarios based on which we can say if we can choose a model with 90% or not. For example, if we are predicting if someone is having cancer or not we cannot go with a 90% accuracy model because it will give 10 wrong results for every 100 patients. While in food ordering scenario we can go ahead with this model.
