How to select between models when AUC scores are similar?
2
$begingroup$
I use two machine learning algorithms for binary classification and I get this result :
Algo 1 :
AUC- Train : 0.75 AUC- Test: 0.65 big Train / overfitting
Algo 2 :
AUC- Train : 0.72 AUC- Test: 0.65 small train / small overfitting
Which one is better?
machine-learning data-mining metric
edited yesterday
Esmailian
1,259112
asked yesterday
amal amalamal amal
202
$endgroup$
add a comment |
2
$begingroup$
I use two machine learning algorithms for binary classification and I get this result :
Algo 1 :
AUC- Train : 0.75 AUC- Test: 0.65 big Train / overfitting
Algo 2 :
AUC- Train : 0.72 AUC- Test: 0.65 small train / small overfitting
Which one is better?
machine-learning data-mining metric
edited yesterday
Esmailian
1,259112
asked yesterday
amal amalamal amal
202
$endgroup$
add a comment |
2
2
2
1
$begingroup$
I use two machine learning algorithms for binary classification and I get this result :
Algo 1 :
AUC- Train : 0.75 AUC- Test: 0.65 big Train / overfitting
Algo 2 :
AUC- Train : 0.72 AUC- Test: 0.65 small train / small overfitting
Which one is better?
machine-learning data-mining metric
edited yesterday
Esmailian
1,259112
asked yesterday
amal amalamal amal
202
$endgroup$
I use two machine learning algorithms for binary classification and I get this result :
Algo 1 :
AUC- Train : 0.75 AUC- Test: 0.65 big Train / overfitting
Algo 2 :
AUC- Train : 0.72 AUC- Test: 0.65 small train / small overfitting
Which one is better?
machine-learning data-mining metric
machine-learning data-mining metric
edited yesterday
Esmailian
1,259112
asked yesterday
amal amalamal amal
202
edited yesterday
Esmailian
1,259112
asked yesterday
amal amalamal amal
202
edited yesterday
Esmailian
1,259112
edited yesterday
Esmailian
1,259112
edited yesterday
Esmailian
1,259112
1,259112
asked yesterday
amal amalamal amal
202
asked yesterday
amal amalamal amal
202
asked yesterday
amal amalamal amal
202
202
add a comment |
add a comment |
3 Answers
3
active
oldest
votes
1
$begingroup$
Based on the AUC score they are the same. It does not really matter if the model is overfitting or not. What matters is how well it performs on new data (test score).
Overfitting is just an indication that there might be room for improvement by making your model more general. But until the test score has increased the model has not improved even if it is overfitting less.
answered yesterday
Simon LarssonSimon Larsson
4316
$endgroup$
$begingroup$
Thanks simon, so if I understand I should always take the biggest test score as the best model without getting any importance to training score?
$endgroup$
– amal amal
yesterday
$begingroup$
Yes, that is correct.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Thanks for your help
$endgroup$
– amal amal
yesterday
$begingroup$
No problem! Don't forget to mark my answer as correct if you got what you asked for.
$endgroup$
– Simon Larsson
yesterday
add a comment |
1
$begingroup$
Algo 2
Between equal test scores choose the one with less difference between training and test scores (Algo 2), since the one with better training score (Algo 1) is more over-fitted. We tolerate a more over-fitted model only if it has a subjectively better test score.
For a better justification, think of how we train a neural network. When validation score stops improving, we stop the training process even though training score will keep improving. If we let the training continue, the model will start making extra assumptions based on the training set that are not scrutinized by the critic (validation set) which makes the model more prone to building false assumptions about the data.
By the same token, a model (Algo 1) that has the same performance based on the critic (test set) but performs better on training set is prone to have made untested assumptions about the data.
answered yesterday
EsmailianEsmailian
1,259112
$endgroup$
$begingroup$
How can you make these assumptions? Test score tells you the generalization ability of the algorithm regardless of the bias/variance. I feel like you can say nothing about which one will perform better on another test set.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Genuinely curious btw, incase you know something I have missed. :)
$endgroup$
– Simon Larsson
yesterday
$begingroup$
@SimonLarsson cool! I made some updates.
$endgroup$
– Esmailian
yesterday
$begingroup$
Thank you for replying! But what I would like to know is how you can assume that one will generalize better than the other on other data when the test score is the same? Just because you know that one model has learned some junk from the training set it does not say that the other model will have learned something useful in its' place.
$endgroup$
– Simon Larsson
yesterday
2
$begingroup$
@SimonLarsson I think fundamentally it's an Occam's Razor thing, with an assumption that the more-overfit model is "more complicated." In specific situations it's easier; e.g., if the data is time-dependent and the test set is out-of-time, then the train/test score discrepancy might indicate degradation over time, so that future performance may degrade faster in the more-overfit model.
$endgroup$
– Ben Reiniger
yesterday
|
show 4 more comments
1
$begingroup$
Just based on this metric you can not find which one is better because AUC could not differentiate these two result. You should use some other metrics such as Kappa or some benchmarks.
Disclaimer:
If you are using Python I suggest PyCM module which get your confusion matrix as input and calculate about 100 overall and class-based metrics.
For using this module at first prepare your confusion matrix and see it's recommended parameters by the following code:
>>> from pycm import *
>>> cm = ConfusionMatrix(matrix={"0": {"0": 1, "1":0, "2": 0}, "1": {"0": 0, "1": 1, "2": 2}, "2": {"0": 0, "1": 1, "2": 0}})
>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]
and then see the value of the metrics focusing on the recommended metrics by the following code:
>>> print(cm)
Predict 0 1 2
Actual
0 1 0 0
1 0 1 2
2 0 1 0
Overall Statistics :
95% CI (-0.02941,0.82941)
Bennett_S 0.1
Chi-Squared 6.66667
Chi-Squared DF 4
Conditional Entropy 0.55098
Cramer_V 0.8165
Cross Entropy 1.52193
Gwet_AC1 0.13043
Joint Entropy 1.92193
KL Divergence 0.15098
Kappa 0.0625
Kappa 95% CI (-0.60846,0.73346)
Kappa No Prevalence -0.2
Kappa Standard Error 0.34233
Kappa Unbiased 0.03226
Lambda A 0.5
Lambda B 0.66667
Mutual Information 0.97095
Overall_ACC 0.4
Overall_RACC 0.36
Overall_RACCU 0.38
PPV_Macro 0.5
PPV_Micro 0.4
Phi-Squared 1.33333
Reference Entropy 1.37095
Response Entropy 1.52193
Scott_PI 0.03226
Standard Error 0.21909
Strength_Of_Agreement(Altman) Poor
Strength_Of_Agreement(Cicchetti) Poor
Strength_Of_Agreement(Fleiss) Poor
Strength_Of_Agreement(Landis and Koch) Slight
TPR_Macro 0.44444
TPR_Micro 0.4
Class Statistics :
Classes 0 1 2
ACC(Accuracy) 1.0 0.4 0.4
BM(Informedness or bookmaker informedness) 1.0 -0.16667 -0.5
DOR(Diagnostic odds ratio) None 0.5 0.0
ERR(Error rate) 0.0 0.6 0.6
F0.5(F0.5 score) 1.0 0.45455 0.0
F1(F1 score - harmonic mean of precision and sensitivity) 1.0 0.4 0.0
F2(F2 score) 1.0 0.35714 0.0
FDR(False discovery rate) 0.0 0.5 1.0
FN(False negative/miss/type 2 error) 0 2 1
FNR(Miss rate or false negative rate) 0.0 0.66667 1.0
FOR(False omission rate) 0.0 0.66667 0.33333
FP(False positive/type 1 error/false alarm) 0 1 2
FPR(Fall-out or false positive rate) 0.0 0.5 0.5
G(G-measure geometric mean of precision and sensitivity) 1.0 0.40825 0.0
LR+(Positive likelihood ratio) None 0.66667 0.0
LR-(Negative likelihood ratio) 0.0 1.33333 2.0
MCC(Matthews correlation coefficient) 1.0 -0.16667 -0.40825
MK(Markedness) 1.0 -0.16667 -0.33333
N(Condition negative) 4 2 4
NPV(Negative predictive value) 1.0 0.33333 0.66667
P(Condition positive) 1 3 1
POP(Population) 5 5 5
PPV(Precision or positive predictive value) 1.0 0.5 0.0
PRE(Prevalence) 0.2 0.6 0.2
RACC(Random accuracy) 0.04 0.24 0.08
RACCU(Random accuracy unbiased) 0.04 0.25 0.09
TN(True negative/correct rejection) 4 1 2
TNR(Specificity or true negative rate) 1.0 0.5 0.5
TON(Test outcome negative) 4 3 3
TOP(Test outcome positive) 1 2 2
TP(True positive/hit) 1 1 0
TPR(Sensitivity, recall, hit rate, or true positive rate) 1.0 0.33333 0.0
answered yesterday
Alireza ZolanvariAlireza Zolanvari
18914
$endgroup$
1
$begingroup$
You should mention that you are an author of the package. (datascience.stackexchange.com/help/behavior)
$endgroup$
– Ben Reiniger
yesterday
$begingroup$
thanks for your reminder.I just edited my answer
$endgroup$
– Alireza Zolanvari
yesterday
$begingroup$
@alirezazolanvari In my opinion, change of measure does not solve the underlying problem. First, choice of measure dependents on task too, we cannot peak and choose independently. More importantly, this problem can happen for any other measure (e.g. Kappa) too, the solution is not to simply change the measure.
$endgroup$
– Esmailian
yesterday
$begingroup$
@Esmailian obviously the evaluation metric is directly related to the task but the researches for finding proper metrics for evaluating a learning algorithm have been focused on clearing the difference between the performance of algorithms in the cases in which the simple metrics such as AUC can not say which one is better. Totally for answering this question many other things should be considered. This answer not a golden key for this problem but can be helpful to solve it.
$endgroup$
– Alireza Zolanvari
yesterday
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47339%2fhow-to-select-between-models-when-auc-scores-are-similar%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
$begingroup$
Based on the AUC score they are the same. It does not really matter if the model is overfitting or not. What matters is how well it performs on new data (test score).
Overfitting is just an indication that there might be room for improvement by making your model more general. But until the test score has increased the model has not improved even if it is overfitting less.
answered yesterday
Simon LarssonSimon Larsson
4316
$endgroup$
$begingroup$
Thanks simon, so if I understand I should always take the biggest test score as the best model without getting any importance to training score?
$endgroup$
– amal amal
yesterday
$begingroup$
Yes, that is correct.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Thanks for your help
$endgroup$
– amal amal
yesterday
$begingroup$
No problem! Don't forget to mark my answer as correct if you got what you asked for.
$endgroup$
– Simon Larsson
yesterday
add a comment |
1
$begingroup$
Based on the AUC score they are the same. It does not really matter if the model is overfitting or not. What matters is how well it performs on new data (test score).
Overfitting is just an indication that there might be room for improvement by making your model more general. But until the test score has increased the model has not improved even if it is overfitting less.
answered yesterday
Simon LarssonSimon Larsson
4316
$endgroup$
$begingroup$
Thanks simon, so if I understand I should always take the biggest test score as the best model without getting any importance to training score?
$endgroup$
– amal amal
yesterday
$begingroup$
Yes, that is correct.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Thanks for your help
$endgroup$
– amal amal
yesterday
$begingroup$
No problem! Don't forget to mark my answer as correct if you got what you asked for.
$endgroup$
– Simon Larsson
yesterday
add a comment |
1
1
1
$begingroup$
Based on the AUC score they are the same. It does not really matter if the model is overfitting or not. What matters is how well it performs on new data (test score).
Overfitting is just an indication that there might be room for improvement by making your model more general. But until the test score has increased the model has not improved even if it is overfitting less.
answered yesterday
Simon LarssonSimon Larsson
4316
$endgroup$
Based on the AUC score they are the same. It does not really matter if the model is overfitting or not. What matters is how well it performs on new data (test score).
Overfitting is just an indication that there might be room for improvement by making your model more general. But until the test score has increased the model has not improved even if it is overfitting less.
answered yesterday
Simon LarssonSimon Larsson
4316
edited yesterday
answered yesterday
Simon LarssonSimon Larsson
4316
answered yesterday
Simon LarssonSimon Larsson
4316
answered yesterday
Simon LarssonSimon Larsson
4316
4316
$begingroup$
Thanks simon, so if I understand I should always take the biggest test score as the best model without getting any importance to training score?
$endgroup$
– amal amal
yesterday
$begingroup$
Yes, that is correct.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Thanks for your help
$endgroup$
– amal amal
yesterday
$begingroup$
No problem! Don't forget to mark my answer as correct if you got what you asked for.
$endgroup$
– Simon Larsson
yesterday
add a comment |
$begingroup$
Thanks simon, so if I understand I should always take the biggest test score as the best model without getting any importance to training score?
$endgroup$
– amal amal
yesterday
$begingroup$
Yes, that is correct.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Thanks for your help
$endgroup$
– amal amal
yesterday
$begingroup$
No problem! Don't forget to mark my answer as correct if you got what you asked for.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Thanks simon, so if I understand I should always take the biggest test score as the best model without getting any importance to training score?
$endgroup$
– amal amal
yesterday
$begingroup$
Thanks simon, so if I understand I should always take the biggest test score as the best model without getting any importance to training score?
$endgroup$
– amal amal
yesterday
$begingroup$
Yes, that is correct.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Yes, that is correct.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Thanks for your help
$endgroup$
– amal amal
yesterday
$begingroup$
Thanks for your help
$endgroup$
– amal amal
yesterday
$begingroup$
No problem! Don't forget to mark my answer as correct if you got what you asked for.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
No problem! Don't forget to mark my answer as correct if you got what you asked for.
$endgroup$
– Simon Larsson
yesterday
add a comment |
1
$begingroup$
Algo 2
Between equal test scores choose the one with less difference between training and test scores (Algo 2), since the one with better training score (Algo 1) is more over-fitted. We tolerate a more over-fitted model only if it has a subjectively better test score.
For a better justification, think of how we train a neural network. When validation score stops improving, we stop the training process even though training score will keep improving. If we let the training continue, the model will start making extra assumptions based on the training set that are not scrutinized by the critic (validation set) which makes the model more prone to building false assumptions about the data.
By the same token, a model (Algo 1) that has the same performance based on the critic (test set) but performs better on training set is prone to have made untested assumptions about the data.
answered yesterday
EsmailianEsmailian
1,259112
$endgroup$
$begingroup$
How can you make these assumptions? Test score tells you the generalization ability of the algorithm regardless of the bias/variance. I feel like you can say nothing about which one will perform better on another test set.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Genuinely curious btw, incase you know something I have missed. :)
$endgroup$
– Simon Larsson
yesterday
$begingroup$
@SimonLarsson cool! I made some updates.
$endgroup$
– Esmailian
yesterday
$begingroup$
Thank you for replying! But what I would like to know is how you can assume that one will generalize better than the other on other data when the test score is the same? Just because you know that one model has learned some junk from the training set it does not say that the other model will have learned something useful in its' place.
$endgroup$
– Simon Larsson
yesterday
2
$begingroup$
@SimonLarsson I think fundamentally it's an Occam's Razor thing, with an assumption that the more-overfit model is "more complicated." In specific situations it's easier; e.g., if the data is time-dependent and the test set is out-of-time, then the train/test score discrepancy might indicate degradation over time, so that future performance may degrade faster in the more-overfit model.
$endgroup$
– Ben Reiniger
yesterday
|
show 4 more comments
1
$begingroup$
Algo 2
Between equal test scores choose the one with less difference between training and test scores (Algo 2), since the one with better training score (Algo 1) is more over-fitted. We tolerate a more over-fitted model only if it has a subjectively better test score.
For a better justification, think of how we train a neural network. When validation score stops improving, we stop the training process even though training score will keep improving. If we let the training continue, the model will start making extra assumptions based on the training set that are not scrutinized by the critic (validation set) which makes the model more prone to building false assumptions about the data.
By the same token, a model (Algo 1) that has the same performance based on the critic (test set) but performs better on training set is prone to have made untested assumptions about the data.
answered yesterday
EsmailianEsmailian
1,259112
$endgroup$
$begingroup$
How can you make these assumptions? Test score tells you the generalization ability of the algorithm regardless of the bias/variance. I feel like you can say nothing about which one will perform better on another test set.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Genuinely curious btw, incase you know something I have missed. :)
$endgroup$
– Simon Larsson
yesterday
$begingroup$
@SimonLarsson cool! I made some updates.
$endgroup$
– Esmailian
yesterday
$begingroup$
Thank you for replying! But what I would like to know is how you can assume that one will generalize better than the other on other data when the test score is the same? Just because you know that one model has learned some junk from the training set it does not say that the other model will have learned something useful in its' place.
$endgroup$
– Simon Larsson
yesterday
2
$begingroup$
@SimonLarsson I think fundamentally it's an Occam's Razor thing, with an assumption that the more-overfit model is "more complicated." In specific situations it's easier; e.g., if the data is time-dependent and the test set is out-of-time, then the train/test score discrepancy might indicate degradation over time, so that future performance may degrade faster in the more-overfit model.
$endgroup$
– Ben Reiniger
yesterday
|
show 4 more comments
1
1
1
$begingroup$
Algo 2
Between equal test scores choose the one with less difference between training and test scores (Algo 2), since the one with better training score (Algo 1) is more over-fitted. We tolerate a more over-fitted model only if it has a subjectively better test score.
For a better justification, think of how we train a neural network. When validation score stops improving, we stop the training process even though training score will keep improving. If we let the training continue, the model will start making extra assumptions based on the training set that are not scrutinized by the critic (validation set) which makes the model more prone to building false assumptions about the data.
By the same token, a model (Algo 1) that has the same performance based on the critic (test set) but performs better on training set is prone to have made untested assumptions about the data.
answered yesterday
EsmailianEsmailian
1,259112
$endgroup$
Algo 2
Between equal test scores choose the one with less difference between training and test scores (Algo 2), since the one with better training score (Algo 1) is more over-fitted. We tolerate a more over-fitted model only if it has a subjectively better test score.
For a better justification, think of how we train a neural network. When validation score stops improving, we stop the training process even though training score will keep improving. If we let the training continue, the model will start making extra assumptions based on the training set that are not scrutinized by the critic (validation set) which makes the model more prone to building false assumptions about the data.
By the same token, a model (Algo 1) that has the same performance based on the critic (test set) but performs better on training set is prone to have made untested assumptions about the data.
answered yesterday
EsmailianEsmailian
1,259112
edited yesterday
answered yesterday
EsmailianEsmailian
1,259112
answered yesterday
EsmailianEsmailian
1,259112
answered yesterday
EsmailianEsmailian
1,259112
1,259112
$begingroup$
How can you make these assumptions? Test score tells you the generalization ability of the algorithm regardless of the bias/variance. I feel like you can say nothing about which one will perform better on another test set.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Genuinely curious btw, incase you know something I have missed. :)
$endgroup$
– Simon Larsson
yesterday
$begingroup$
@SimonLarsson cool! I made some updates.
$endgroup$
– Esmailian
yesterday
$begingroup$
Thank you for replying! But what I would like to know is how you can assume that one will generalize better than the other on other data when the test score is the same? Just because you know that one model has learned some junk from the training set it does not say that the other model will have learned something useful in its' place.
$endgroup$
– Simon Larsson
yesterday
2
$begingroup$
@SimonLarsson I think fundamentally it's an Occam's Razor thing, with an assumption that the more-overfit model is "more complicated." In specific situations it's easier; e.g., if the data is time-dependent and the test set is out-of-time, then the train/test score discrepancy might indicate degradation over time, so that future performance may degrade faster in the more-overfit model.
$endgroup$
– Ben Reiniger
yesterday
|
show 4 more comments
$begingroup$
How can you make these assumptions? Test score tells you the generalization ability of the algorithm regardless of the bias/variance. I feel like you can say nothing about which one will perform better on another test set.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Genuinely curious btw, incase you know something I have missed. :)
$endgroup$
– Simon Larsson
yesterday
$begingroup$
@SimonLarsson cool! I made some updates.
$endgroup$
– Esmailian
yesterday
$begingroup$
Thank you for replying! But what I would like to know is how you can assume that one will generalize better than the other on other data when the test score is the same? Just because you know that one model has learned some junk from the training set it does not say that the other model will have learned something useful in its' place.
$endgroup$
– Simon Larsson
yesterday
2
$begingroup$
@SimonLarsson I think fundamentally it's an Occam's Razor thing, with an assumption that the more-overfit model is "more complicated." In specific situations it's easier; e.g., if the data is time-dependent and the test set is out-of-time, then the train/test score discrepancy might indicate degradation over time, so that future performance may degrade faster in the more-overfit model.
$endgroup$
– Ben Reiniger
yesterday
$begingroup$
How can you make these assumptions? Test score tells you the generalization ability of the algorithm regardless of the bias/variance. I feel like you can say nothing about which one will perform better on another test set.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
How can you make these assumptions? Test score tells you the generalization ability of the algorithm regardless of the bias/variance. I feel like you can say nothing about which one will perform better on another test set.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Genuinely curious btw, incase you know something I have missed. :)
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Genuinely curious btw, incase you know something I have missed. :)
$endgroup$
– Simon Larsson
yesterday
$begingroup$
@SimonLarsson cool! I made some updates.
$endgroup$
– Esmailian
yesterday
$begingroup$
@SimonLarsson cool! I made some updates.
$endgroup$
– Esmailian
yesterday
$begingroup$
Thank you for replying! But what I would like to know is how you can assume that one will generalize better than the other on other data when the test score is the same? Just because you know that one model has learned some junk from the training set it does not say that the other model will have learned something useful in its' place.
$endgroup$
– Simon Larsson
yesterday
$begingroup$
Thank you for replying! But what I would like to know is how you can assume that one will generalize better than the other on other data when the test score is the same? Just because you know that one model has learned some junk from the training set it does not say that the other model will have learned something useful in its' place.
$endgroup$
– Simon Larsson
yesterday
2
2
$begingroup$
@SimonLarsson I think fundamentally it's an Occam's Razor thing, with an assumption that the more-overfit model is "more complicated." In specific situations it's easier; e.g., if the data is time-dependent and the test set is out-of-time, then the train/test score discrepancy might indicate degradation over time, so that future performance may degrade faster in the more-overfit model.
$endgroup$
– Ben Reiniger
yesterday
$begingroup$
@SimonLarsson I think fundamentally it's an Occam's Razor thing, with an assumption that the more-overfit model is "more complicated." In specific situations it's easier; e.g., if the data is time-dependent and the test set is out-of-time, then the train/test score discrepancy might indicate degradation over time, so that future performance may degrade faster in the more-overfit model.
$endgroup$
– Ben Reiniger
yesterday
|
show 4 more comments
1
$begingroup$
Just based on this metric you can not find which one is better because AUC could not differentiate these two result. You should use some other metrics such as Kappa or some benchmarks.
Disclaimer:
If you are using Python I suggest PyCM module which get your confusion matrix as input and calculate about 100 overall and class-based metrics.
For using this module at first prepare your confusion matrix and see it's recommended parameters by the following code:
>>> from pycm import *
>>> cm = ConfusionMatrix(matrix={"0": {"0": 1, "1":0, "2": 0}, "1": {"0": 0, "1": 1, "2": 2}, "2": {"0": 0, "1": 1, "2": 0}})
>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]
and then see the value of the metrics focusing on the recommended metrics by the following code:
>>> print(cm)
Predict 0 1 2
Actual
0 1 0 0
1 0 1 2
2 0 1 0
Overall Statistics :
95% CI (-0.02941,0.82941)
Bennett_S 0.1
Chi-Squared 6.66667
Chi-Squared DF 4
Conditional Entropy 0.55098
Cramer_V 0.8165
Cross Entropy 1.52193
Gwet_AC1 0.13043
Joint Entropy 1.92193
KL Divergence 0.15098
Kappa 0.0625
Kappa 95% CI (-0.60846,0.73346)
Kappa No Prevalence -0.2
Kappa Standard Error 0.34233
Kappa Unbiased 0.03226
Lambda A 0.5
Lambda B 0.66667
Mutual Information 0.97095
Overall_ACC 0.4
Overall_RACC 0.36
Overall_RACCU 0.38
PPV_Macro 0.5
PPV_Micro 0.4
Phi-Squared 1.33333
Reference Entropy 1.37095
Response Entropy 1.52193
Scott_PI 0.03226
Standard Error 0.21909
Strength_Of_Agreement(Altman) Poor
Strength_Of_Agreement(Cicchetti) Poor
Strength_Of_Agreement(Fleiss) Poor
Strength_Of_Agreement(Landis and Koch) Slight
TPR_Macro 0.44444
TPR_Micro 0.4
Class Statistics :
Classes 0 1 2
ACC(Accuracy) 1.0 0.4 0.4
BM(Informedness or bookmaker informedness) 1.0 -0.16667 -0.5
DOR(Diagnostic odds ratio) None 0.5 0.0
ERR(Error rate) 0.0 0.6 0.6
F0.5(F0.5 score) 1.0 0.45455 0.0
F1(F1 score - harmonic mean of precision and sensitivity) 1.0 0.4 0.0
F2(F2 score) 1.0 0.35714 0.0
FDR(False discovery rate) 0.0 0.5 1.0
FN(False negative/miss/type 2 error) 0 2 1
FNR(Miss rate or false negative rate) 0.0 0.66667 1.0
FOR(False omission rate) 0.0 0.66667 0.33333
FP(False positive/type 1 error/false alarm) 0 1 2
FPR(Fall-out or false positive rate) 0.0 0.5 0.5
G(G-measure geometric mean of precision and sensitivity) 1.0 0.40825 0.0
LR+(Positive likelihood ratio) None 0.66667 0.0
LR-(Negative likelihood ratio) 0.0 1.33333 2.0
MCC(Matthews correlation coefficient) 1.0 -0.16667 -0.40825
MK(Markedness) 1.0 -0.16667 -0.33333
N(Condition negative) 4 2 4
NPV(Negative predictive value) 1.0 0.33333 0.66667
P(Condition positive) 1 3 1
POP(Population) 5 5 5
PPV(Precision or positive predictive value) 1.0 0.5 0.0
PRE(Prevalence) 0.2 0.6 0.2
RACC(Random accuracy) 0.04 0.24 0.08
RACCU(Random accuracy unbiased) 0.04 0.25 0.09
TN(True negative/correct rejection) 4 1 2
TNR(Specificity or true negative rate) 1.0 0.5 0.5
TON(Test outcome negative) 4 3 3
TOP(Test outcome positive) 1 2 2
TP(True positive/hit) 1 1 0
TPR(Sensitivity, recall, hit rate, or true positive rate) 1.0 0.33333 0.0
answered yesterday
Alireza ZolanvariAlireza Zolanvari
18914
$endgroup$
1
$begingroup$
You should mention that you are an author of the package. (datascience.stackexchange.com/help/behavior)
$endgroup$
– Ben Reiniger
yesterday
$begingroup$
thanks for your reminder.I just edited my answer
$endgroup$
– Alireza Zolanvari
yesterday
$begingroup$
@alirezazolanvari In my opinion, change of measure does not solve the underlying problem. First, choice of measure dependents on task too, we cannot peak and choose independently. More importantly, this problem can happen for any other measure (e.g. Kappa) too, the solution is not to simply change the measure.
$endgroup$
– Esmailian
yesterday
$begingroup$
@Esmailian obviously the evaluation metric is directly related to the task but the researches for finding proper metrics for evaluating a learning algorithm have been focused on clearing the difference between the performance of algorithms in the cases in which the simple metrics such as AUC can not say which one is better. Totally for answering this question many other things should be considered. This answer not a golden key for this problem but can be helpful to solve it.
$endgroup$
– Alireza Zolanvari
yesterday
add a comment |
1
$begingroup$
Just based on this metric you can not find which one is better because AUC could not differentiate these two result. You should use some other metrics such as Kappa or some benchmarks.
Disclaimer:
If you are using Python I suggest PyCM module which get your confusion matrix as input and calculate about 100 overall and class-based metrics.
For using this module at first prepare your confusion matrix and see it's recommended parameters by the following code:
>>> from pycm import *
>>> cm = ConfusionMatrix(matrix={"0": {"0": 1, "1":0, "2": 0}, "1": {"0": 0, "1": 1, "2": 2}, "2": {"0": 0, "1": 1, "2": 0}})
>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]
and then see the value of the metrics focusing on the recommended metrics by the following code:
>>> print(cm)
Predict 0 1 2
Actual
0 1 0 0
1 0 1 2
2 0 1 0
Overall Statistics :
95% CI (-0.02941,0.82941)
Bennett_S 0.1
Chi-Squared 6.66667
Chi-Squared DF 4
Conditional Entropy 0.55098
Cramer_V 0.8165
Cross Entropy 1.52193
Gwet_AC1 0.13043
Joint Entropy 1.92193
KL Divergence 0.15098
Kappa 0.0625
Kappa 95% CI (-0.60846,0.73346)
Kappa No Prevalence -0.2
Kappa Standard Error 0.34233
Kappa Unbiased 0.03226
Lambda A 0.5
Lambda B 0.66667
Mutual Information 0.97095
Overall_ACC 0.4
Overall_RACC 0.36
Overall_RACCU 0.38
PPV_Macro 0.5
PPV_Micro 0.4
Phi-Squared 1.33333
Reference Entropy 1.37095
Response Entropy 1.52193
Scott_PI 0.03226
Standard Error 0.21909
Strength_Of_Agreement(Altman) Poor
Strength_Of_Agreement(Cicchetti) Poor
Strength_Of_Agreement(Fleiss) Poor
Strength_Of_Agreement(Landis and Koch) Slight
TPR_Macro 0.44444
TPR_Micro 0.4
Class Statistics :
Classes 0 1 2
ACC(Accuracy) 1.0 0.4 0.4
BM(Informedness or bookmaker informedness) 1.0 -0.16667 -0.5
DOR(Diagnostic odds ratio) None 0.5 0.0
ERR(Error rate) 0.0 0.6 0.6
F0.5(F0.5 score) 1.0 0.45455 0.0
F1(F1 score - harmonic mean of precision and sensitivity) 1.0 0.4 0.0
F2(F2 score) 1.0 0.35714 0.0
FDR(False discovery rate) 0.0 0.5 1.0
FN(False negative/miss/type 2 error) 0 2 1
FNR(Miss rate or false negative rate) 0.0 0.66667 1.0
FOR(False omission rate) 0.0 0.66667 0.33333
FP(False positive/type 1 error/false alarm) 0 1 2
FPR(Fall-out or false positive rate) 0.0 0.5 0.5
G(G-measure geometric mean of precision and sensitivity) 1.0 0.40825 0.0
LR+(Positive likelihood ratio) None 0.66667 0.0
LR-(Negative likelihood ratio) 0.0 1.33333 2.0
MCC(Matthews correlation coefficient) 1.0 -0.16667 -0.40825
MK(Markedness) 1.0 -0.16667 -0.33333
N(Condition negative) 4 2 4
NPV(Negative predictive value) 1.0 0.33333 0.66667
P(Condition positive) 1 3 1
POP(Population) 5 5 5
PPV(Precision or positive predictive value) 1.0 0.5 0.0
PRE(Prevalence) 0.2 0.6 0.2
RACC(Random accuracy) 0.04 0.24 0.08
RACCU(Random accuracy unbiased) 0.04 0.25 0.09
TN(True negative/correct rejection) 4 1 2
TNR(Specificity or true negative rate) 1.0 0.5 0.5
TON(Test outcome negative) 4 3 3
TOP(Test outcome positive) 1 2 2
TP(True positive/hit) 1 1 0
TPR(Sensitivity, recall, hit rate, or true positive rate) 1.0 0.33333 0.0
answered yesterday
Alireza ZolanvariAlireza Zolanvari
18914
$endgroup$
1
$begingroup$
You should mention that you are an author of the package. (datascience.stackexchange.com/help/behavior)
$endgroup$
– Ben Reiniger
yesterday
$begingroup$
thanks for your reminder.I just edited my answer
$endgroup$
– Alireza Zolanvari
yesterday
$begingroup$
@alirezazolanvari In my opinion, change of measure does not solve the underlying problem. First, choice of measure dependents on task too, we cannot peak and choose independently. More importantly, this problem can happen for any other measure (e.g. Kappa) too, the solution is not to simply change the measure.
$endgroup$
– Esmailian
yesterday
$begingroup$
@Esmailian obviously the evaluation metric is directly related to the task but the researches for finding proper metrics for evaluating a learning algorithm have been focused on clearing the difference between the performance of algorithms in the cases in which the simple metrics such as AUC can not say which one is better. Totally for answering this question many other things should be considered. This answer not a golden key for this problem but can be helpful to solve it.
$endgroup$
– Alireza Zolanvari
yesterday
add a comment |
1
1
1
$begingroup$
Just based on this metric you can not find which one is better because AUC could not differentiate these two result. You should use some other metrics such as Kappa or some benchmarks.
Disclaimer:
If you are using Python I suggest PyCM module which get your confusion matrix as input and calculate about 100 overall and class-based metrics.
For using this module at first prepare your confusion matrix and see it's recommended parameters by the following code:
>>> from pycm import *
>>> cm = ConfusionMatrix(matrix={"0": {"0": 1, "1":0, "2": 0}, "1": {"0": 0, "1": 1, "2": 2}, "2": {"0": 0, "1": 1, "2": 0}})
>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]
and then see the value of the metrics focusing on the recommended metrics by the following code:
>>> print(cm)
Predict 0 1 2
Actual
0 1 0 0
1 0 1 2
2 0 1 0
Overall Statistics :
95% CI (-0.02941,0.82941)
Bennett_S 0.1
Chi-Squared 6.66667
Chi-Squared DF 4
Conditional Entropy 0.55098
Cramer_V 0.8165
Cross Entropy 1.52193
Gwet_AC1 0.13043
Joint Entropy 1.92193
KL Divergence 0.15098
Kappa 0.0625
Kappa 95% CI (-0.60846,0.73346)
Kappa No Prevalence -0.2
Kappa Standard Error 0.34233
Kappa Unbiased 0.03226
Lambda A 0.5
Lambda B 0.66667
Mutual Information 0.97095
Overall_ACC 0.4
Overall_RACC 0.36
Overall_RACCU 0.38
PPV_Macro 0.5
PPV_Micro 0.4
Phi-Squared 1.33333
Reference Entropy 1.37095
Response Entropy 1.52193
Scott_PI 0.03226
Standard Error 0.21909
Strength_Of_Agreement(Altman) Poor
Strength_Of_Agreement(Cicchetti) Poor
Strength_Of_Agreement(Fleiss) Poor
Strength_Of_Agreement(Landis and Koch) Slight
TPR_Macro 0.44444
TPR_Micro 0.4
Class Statistics :
Classes 0 1 2
ACC(Accuracy) 1.0 0.4 0.4
BM(Informedness or bookmaker informedness) 1.0 -0.16667 -0.5
DOR(Diagnostic odds ratio) None 0.5 0.0
ERR(Error rate) 0.0 0.6 0.6
F0.5(F0.5 score) 1.0 0.45455 0.0
F1(F1 score - harmonic mean of precision and sensitivity) 1.0 0.4 0.0
F2(F2 score) 1.0 0.35714 0.0
FDR(False discovery rate) 0.0 0.5 1.0
FN(False negative/miss/type 2 error) 0 2 1
FNR(Miss rate or false negative rate) 0.0 0.66667 1.0
FOR(False omission rate) 0.0 0.66667 0.33333
FP(False positive/type 1 error/false alarm) 0 1 2
FPR(Fall-out or false positive rate) 0.0 0.5 0.5
G(G-measure geometric mean of precision and sensitivity) 1.0 0.40825 0.0
LR+(Positive likelihood ratio) None 0.66667 0.0
LR-(Negative likelihood ratio) 0.0 1.33333 2.0
MCC(Matthews correlation coefficient) 1.0 -0.16667 -0.40825
MK(Markedness) 1.0 -0.16667 -0.33333
N(Condition negative) 4 2 4
NPV(Negative predictive value) 1.0 0.33333 0.66667
P(Condition positive) 1 3 1
POP(Population) 5 5 5
PPV(Precision or positive predictive value) 1.0 0.5 0.0
PRE(Prevalence) 0.2 0.6 0.2
RACC(Random accuracy) 0.04 0.24 0.08
RACCU(Random accuracy unbiased) 0.04 0.25 0.09
TN(True negative/correct rejection) 4 1 2
TNR(Specificity or true negative rate) 1.0 0.5 0.5
TON(Test outcome negative) 4 3 3
TOP(Test outcome positive) 1 2 2
TP(True positive/hit) 1 1 0
TPR(Sensitivity, recall, hit rate, or true positive rate) 1.0 0.33333 0.0
answered yesterday
Alireza ZolanvariAlireza Zolanvari
18914
$endgroup$
Just based on this metric you can not find which one is better because AUC could not differentiate these two result. You should use some other metrics such as Kappa or some benchmarks.
Disclaimer:
If you are using Python I suggest PyCM module which get your confusion matrix as input and calculate about 100 overall and class-based metrics.
For using this module at first prepare your confusion matrix and see it's recommended parameters by the following code:
>>> from pycm import *
>>> cm = ConfusionMatrix(matrix={"0": {"0": 1, "1":0, "2": 0}, "1": {"0": 0, "1": 1, "2": 2}, "2": {"0": 0, "1": 1, "2": 0}})
>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]
and then see the value of the metrics focusing on the recommended metrics by the following code:
>>> print(cm)
Predict 0 1 2
Actual
0 1 0 0
1 0 1 2
2 0 1 0
Overall Statistics :
95% CI (-0.02941,0.82941)
Bennett_S 0.1
Chi-Squared 6.66667
Chi-Squared DF 4
Conditional Entropy 0.55098
Cramer_V 0.8165
Cross Entropy 1.52193
Gwet_AC1 0.13043
Joint Entropy 1.92193
KL Divergence 0.15098
Kappa 0.0625
Kappa 95% CI (-0.60846,0.73346)
Kappa No Prevalence -0.2
Kappa Standard Error 0.34233
Kappa Unbiased 0.03226
Lambda A 0.5
Lambda B 0.66667
Mutual Information 0.97095
Overall_ACC 0.4
Overall_RACC 0.36
Overall_RACCU 0.38
PPV_Macro 0.5
PPV_Micro 0.4
Phi-Squared 1.33333
Reference Entropy 1.37095
Response Entropy 1.52193
Scott_PI 0.03226
Standard Error 0.21909
Strength_Of_Agreement(Altman) Poor
Strength_Of_Agreement(Cicchetti) Poor
Strength_Of_Agreement(Fleiss) Poor
Strength_Of_Agreement(Landis and Koch) Slight
TPR_Macro 0.44444
TPR_Micro 0.4
Class Statistics :
Classes 0 1 2
ACC(Accuracy) 1.0 0.4 0.4
BM(Informedness or bookmaker informedness) 1.0 -0.16667 -0.5
DOR(Diagnostic odds ratio) None 0.5 0.0
ERR(Error rate) 0.0 0.6 0.6
F0.5(F0.5 score) 1.0 0.45455 0.0
F1(F1 score - harmonic mean of precision and sensitivity) 1.0 0.4 0.0
F2(F2 score) 1.0 0.35714 0.0
FDR(False discovery rate) 0.0 0.5 1.0
FN(False negative/miss/type 2 error) 0 2 1
FNR(Miss rate or false negative rate) 0.0 0.66667 1.0
FOR(False omission rate) 0.0 0.66667 0.33333
FP(False positive/type 1 error/false alarm) 0 1 2
FPR(Fall-out or false positive rate) 0.0 0.5 0.5
G(G-measure geometric mean of precision and sensitivity) 1.0 0.40825 0.0
LR+(Positive likelihood ratio) None 0.66667 0.0
LR-(Negative likelihood ratio) 0.0 1.33333 2.0
MCC(Matthews correlation coefficient) 1.0 -0.16667 -0.40825
MK(Markedness) 1.0 -0.16667 -0.33333
N(Condition negative) 4 2 4
NPV(Negative predictive value) 1.0 0.33333 0.66667
P(Condition positive) 1 3 1
POP(Population) 5 5 5
PPV(Precision or positive predictive value) 1.0 0.5 0.0
PRE(Prevalence) 0.2 0.6 0.2
RACC(Random accuracy) 0.04 0.24 0.08
RACCU(Random accuracy unbiased) 0.04 0.25 0.09
TN(True negative/correct rejection) 4 1 2
TNR(Specificity or true negative rate) 1.0 0.5 0.5
TON(Test outcome negative) 4 3 3
TOP(Test outcome positive) 1 2 2
TP(True positive/hit) 1 1 0
TPR(Sensitivity, recall, hit rate, or true positive rate) 1.0 0.33333 0.0
answered yesterday
Alireza ZolanvariAlireza Zolanvari
18914
edited yesterday
answered yesterday
Alireza ZolanvariAlireza Zolanvari
18914
answered yesterday
Alireza ZolanvariAlireza Zolanvari
18914
answered yesterday
Alireza ZolanvariAlireza Zolanvari
18914
18914
1
$begingroup$
You should mention that you are an author of the package. (datascience.stackexchange.com/help/behavior)
$endgroup$
– Ben Reiniger
yesterday
$begingroup$
thanks for your reminder.I just edited my answer
$endgroup$
– Alireza Zolanvari
yesterday
$begingroup$
@alirezazolanvari In my opinion, change of measure does not solve the underlying problem. First, choice of measure dependents on task too, we cannot peak and choose independently. More importantly, this problem can happen for any other measure (e.g. Kappa) too, the solution is not to simply change the measure.
$endgroup$
– Esmailian
yesterday
$begingroup$
@Esmailian obviously the evaluation metric is directly related to the task but the researches for finding proper metrics for evaluating a learning algorithm have been focused on clearing the difference between the performance of algorithms in the cases in which the simple metrics such as AUC can not say which one is better. Totally for answering this question many other things should be considered. This answer not a golden key for this problem but can be helpful to solve it.
$endgroup$
– Alireza Zolanvari
yesterday
add a comment |
1
$begingroup$
You should mention that you are an author of the package. (datascience.stackexchange.com/help/behavior)
$endgroup$
– Ben Reiniger
yesterday
$begingroup$
thanks for your reminder.I just edited my answer
$endgroup$
– Alireza Zolanvari
yesterday
$begingroup$
@alirezazolanvari In my opinion, change of measure does not solve the underlying problem. First, choice of measure dependents on task too, we cannot peak and choose independently. More importantly, this problem can happen for any other measure (e.g. Kappa) too, the solution is not to simply change the measure.
$endgroup$
– Esmailian
yesterday
$begingroup$
@Esmailian obviously the evaluation metric is directly related to the task but the researches for finding proper metrics for evaluating a learning algorithm have been focused on clearing the difference between the performance of algorithms in the cases in which the simple metrics such as AUC can not say which one is better. Totally for answering this question many other things should be considered. This answer not a golden key for this problem but can be helpful to solve it.
$endgroup$
– Alireza Zolanvari
yesterday
1
1
$begingroup$
You should mention that you are an author of the package. (datascience.stackexchange.com/help/behavior)
$endgroup$
– Ben Reiniger
yesterday
$begingroup$
You should mention that you are an author of the package. (datascience.stackexchange.com/help/behavior)
$endgroup$
– Ben Reiniger
yesterday
$begingroup$
thanks for your reminder.I just edited my answer
$endgroup$
– Alireza Zolanvari
yesterday
$begingroup$
thanks for your reminder.I just edited my answer
$endgroup$
– Alireza Zolanvari
yesterday
$begingroup$
@alirezazolanvari In my opinion, change of measure does not solve the underlying problem. First, choice of measure dependents on task too, we cannot peak and choose independently. More importantly, this problem can happen for any other measure (e.g. Kappa) too, the solution is not to simply change the measure.
$endgroup$
– Esmailian
yesterday
$begingroup$
@alirezazolanvari In my opinion, change of measure does not solve the underlying problem. First, choice of measure dependents on task too, we cannot peak and choose independently. More importantly, this problem can happen for any other measure (e.g. Kappa) too, the solution is not to simply change the measure.
$endgroup$
– Esmailian
yesterday
$begingroup$
@Esmailian obviously the evaluation metric is directly related to the task but the researches for finding proper metrics for evaluating a learning algorithm have been focused on clearing the difference between the performance of algorithms in the cases in which the simple metrics such as AUC can not say which one is better. Totally for answering this question many other things should be considered. This answer not a golden key for this problem but can be helpful to solve it.
$endgroup$
– Alireza Zolanvari
yesterday
$begingroup$
@Esmailian obviously the evaluation metric is directly related to the task but the researches for finding proper metrics for evaluating a learning algorithm have been focused on clearing the difference between the performance of algorithms in the cases in which the simple metrics such as AUC can not say which one is better. Totally for answering this question many other things should be considered. This answer not a golden key for this problem but can be helpful to solve it.
$endgroup$
– Alireza Zolanvari
yesterday
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47339%2fhow-to-select-between-models-when-auc-scores-are-similar%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
