Data prepration for logistic regression : Value either “not available” or a “year”
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked 2 days ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked 2 days ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
2 days ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
2 days ago
add a comment |
$begingroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
asked 2 days ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
I have some data of houses that have been renovated.
In my data there is one column (among others) that captures this information.
It is either "-1" if there has not been yet any renovation, or the information is the year of renovation like "1995" or "2008".
I would like to apply logistic regression.
However, I do not know how to treat this value.
IMHO it looks like a missing value although it is not a missing information.
So, does anybody have an idea how to put these (unordered) values into relation to the ordered years?
On alternative I could think of is binning the information. Like 1990-1995, 1996-2000,...2016-2019.
Any suggestions are highly appreciated.
logistic-regression missing-data
logistic-regression missing-data
asked 2 days ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 2 days ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 2 days ago
toomtoom
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 2 days ago
toomtoom
1112
asked 2 days ago
toomtoom
1112
1112
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
toom is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
2 days ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
2 days ago
add a comment |
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
2 days ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
2 days ago
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
2 days ago
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
2 days ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
2 days ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
2 days ago
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 2 days ago
EsmailianEsmailian
1,651114
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
2 days ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
toom is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47586%2fdata-prepration-for-logistic-regression-value-either-not-available-or-a-yea%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 2 days ago
EsmailianEsmailian
1,651114
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
2 days ago
add a comment |
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 2 days ago
EsmailianEsmailian
1,651114
$endgroup$
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
2 days ago
add a comment |
$begingroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 2 days ago
EsmailianEsmailian
1,651114
$endgroup$
First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression.
Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans like 5 years, 10 years (depends on the context) seems as an improvement. For example:
bins = [1990, 2000], [2000, 2010], [2010, 2020]
[1990, 2000] $rightarrow$ (1, 0, 0)
[2000, 2010] $rightarrow$ (0, 1, 0)
[2010, 2020] $rightarrow$ (0, 0, 1)
This approach also has a tendency to over-fit but over a larger time span. Also note that, this way, your model always has an expiration date, since if we pass the last bin in 2021, there is no bin to cover the year. And if we include [2020, 2030] now, there is no data to learn about this bin. And using [2020, forever] is equally useless for future.
I suggest using the age of construction and renovation which are generalizable. A 5 years old house in 2000 could help us infer about a 5 years old house in 2010, 2020, or 2030. For houses with no renovation, age could be set to -1, which works fine with logistic regression (experiment with 0 too). So as a final example:
renovation (has renovation, renovation age)
-1 (0, -1)
2010 in 2019 (1, 9)
Note that repetitive time features are OK. For example, "Spring", "Monday", or "8:00PM", etc.
answered 2 days ago
EsmailianEsmailian
1,651114
edited 2 days ago
answered 2 days ago
EsmailianEsmailian
1,651114
answered 2 days ago
EsmailianEsmailian
1,651114
answered 2 days ago
EsmailianEsmailian
1,651114
1,651114
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
2 days ago
add a comment |
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
2 days ago
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
2 days ago
$begingroup$
Great advice. Thanks for this help. Makes sense :)
$endgroup$
– toom
2 days ago
add a comment |
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
toom is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47586%2fdata-prepration-for-logistic-regression-value-either-not-available-or-a-yea%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
If you change value of "year of renovation" to be same "year of construction" for houses that have not been renovated; does it makes sense for this problem ?
$endgroup$
– Shamit Verma
2 days ago
$begingroup$
Yeah, this is a of course a good idea and it also makes sense. In my case this is however, not applicable. Thanks for mentioning
$endgroup$
– toom
2 days ago