Mutual information of continuous variables
$begingroup$
I think I am misunderstanding the notion of mutual information of continuous variables. Could anyone help me clear up the following?
Let $X sim N(0, sigma^2) $ and $Y sim N(0, sigma^2) $ denote Gaussian random variables. If $X$ and $Y$ are correlated with a coefficient $rho$, then the mutual information between $X$ and $Y$ is given by
(reference: https://en.wikipedia.org/wiki/Mutual_information).
begin{equation}
I(X; Y) = -frac{1}{2} log (1-rho^2).
end{equation}
Here, I thought $I(X; Y) rightarrow infty$ when $rho rightarrow 1$ (for $X = Y$, $rho = 1$). I considered this another way.
I considered $Y = X$.
In this case, I would obtain $ I (X; Y) = H(X) - H(Y|X) = H(X) $.
For the Gaussian random variable $X$, $H(X)$ is bounded as follows (reference: https://en.wikipedia.org/wiki/Differential_entropy):
begin{equation}
H(X) leq frac{1}{2} log ( 2 pi e sigma^2).
end{equation}
Thus, $ I (X; Y) leq frac{1}{2} log ( 2 pi e sigma^2)$.
Here is my question.
I obtained two different results on $ I (X; Y)$ for $X = Y$.
What could be some mistakes in my understanding?
Thank you in advance.
information-theory entropy
edited Jun 6 '18 at 9:21
Perspectiva8
16212
asked Jun 6 '18 at 8:40
Inkyu BangInkyu Bang
295
$endgroup$
add a comment |
$begingroup$
I think I am misunderstanding the notion of mutual information of continuous variables. Could anyone help me clear up the following?
Let $X sim N(0, sigma^2) $ and $Y sim N(0, sigma^2) $ denote Gaussian random variables. If $X$ and $Y$ are correlated with a coefficient $rho$, then the mutual information between $X$ and $Y$ is given by
(reference: https://en.wikipedia.org/wiki/Mutual_information).
begin{equation}
I(X; Y) = -frac{1}{2} log (1-rho^2).
end{equation}
Here, I thought $I(X; Y) rightarrow infty$ when $rho rightarrow 1$ (for $X = Y$, $rho = 1$). I considered this another way.
I considered $Y = X$.
In this case, I would obtain $ I (X; Y) = H(X) - H(Y|X) = H(X) $.
For the Gaussian random variable $X$, $H(X)$ is bounded as follows (reference: https://en.wikipedia.org/wiki/Differential_entropy):
begin{equation}
H(X) leq frac{1}{2} log ( 2 pi e sigma^2).
end{equation}
Thus, $ I (X; Y) leq frac{1}{2} log ( 2 pi e sigma^2)$.
Here is my question.
I obtained two different results on $ I (X; Y)$ for $X = Y$.
What could be some mistakes in my understanding?
Thank you in advance.
information-theory entropy
edited Jun 6 '18 at 9:21
Perspectiva8
16212
asked Jun 6 '18 at 8:40
Inkyu BangInkyu Bang
295
$endgroup$
$begingroup$
One usually writes $h(X)$ for the differetial entropy, not $H(X)$, and with good reason, it reminds you that it's not a "true" entropy, so that you don't into the trap of assuming $h(X |X)=0$ (as with the true entropy). Actually $h(X|X)=-infty$. See eg my answer here
$endgroup$
– leonbloy
Nov 30 '18 at 23:37
add a comment |
$begingroup$
I think I am misunderstanding the notion of mutual information of continuous variables. Could anyone help me clear up the following?
Let $X sim N(0, sigma^2) $ and $Y sim N(0, sigma^2) $ denote Gaussian random variables. If $X$ and $Y$ are correlated with a coefficient $rho$, then the mutual information between $X$ and $Y$ is given by
(reference: https://en.wikipedia.org/wiki/Mutual_information).
begin{equation}
I(X; Y) = -frac{1}{2} log (1-rho^2).
end{equation}
Here, I thought $I(X; Y) rightarrow infty$ when $rho rightarrow 1$ (for $X = Y$, $rho = 1$). I considered this another way.
I considered $Y = X$.
In this case, I would obtain $ I (X; Y) = H(X) - H(Y|X) = H(X) $.
For the Gaussian random variable $X$, $H(X)$ is bounded as follows (reference: https://en.wikipedia.org/wiki/Differential_entropy):
begin{equation}
H(X) leq frac{1}{2} log ( 2 pi e sigma^2).
end{equation}
Thus, $ I (X; Y) leq frac{1}{2} log ( 2 pi e sigma^2)$.
Here is my question.
I obtained two different results on $ I (X; Y)$ for $X = Y$.
What could be some mistakes in my understanding?
Thank you in advance.
information-theory entropy
edited Jun 6 '18 at 9:21
Perspectiva8
16212
asked Jun 6 '18 at 8:40
Inkyu BangInkyu Bang
295
$endgroup$
I think I am misunderstanding the notion of mutual information of continuous variables. Could anyone help me clear up the following?
Let $X sim N(0, sigma^2) $ and $Y sim N(0, sigma^2) $ denote Gaussian random variables. If $X$ and $Y$ are correlated with a coefficient $rho$, then the mutual information between $X$ and $Y$ is given by
(reference: https://en.wikipedia.org/wiki/Mutual_information).
begin{equation}
I(X; Y) = -frac{1}{2} log (1-rho^2).
end{equation}
Here, I thought $I(X; Y) rightarrow infty$ when $rho rightarrow 1$ (for $X = Y$, $rho = 1$). I considered this another way.
I considered $Y = X$.
In this case, I would obtain $ I (X; Y) = H(X) - H(Y|X) = H(X) $.
For the Gaussian random variable $X$, $H(X)$ is bounded as follows (reference: https://en.wikipedia.org/wiki/Differential_entropy):
begin{equation}
H(X) leq frac{1}{2} log ( 2 pi e sigma^2).
end{equation}
Thus, $ I (X; Y) leq frac{1}{2} log ( 2 pi e sigma^2)$.
Here is my question.
I obtained two different results on $ I (X; Y)$ for $X = Y$.
What could be some mistakes in my understanding?
Thank you in advance.
information-theory entropy
information-theory entropy
edited Jun 6 '18 at 9:21
Perspectiva8
16212
asked Jun 6 '18 at 8:40
Inkyu BangInkyu Bang
295
edited Jun 6 '18 at 9:21
Perspectiva8
16212
asked Jun 6 '18 at 8:40
Inkyu BangInkyu Bang
295
edited Jun 6 '18 at 9:21
Perspectiva8
16212
edited Jun 6 '18 at 9:21
Perspectiva8
16212
edited Jun 6 '18 at 9:21
Perspectiva8
16212
16212
asked Jun 6 '18 at 8:40
Inkyu BangInkyu Bang
295
asked Jun 6 '18 at 8:40
Inkyu BangInkyu Bang
295
asked Jun 6 '18 at 8:40
Inkyu BangInkyu Bang
295
295
$begingroup$
One usually writes $h(X)$ for the differetial entropy, not $H(X)$, and with good reason, it reminds you that it's not a "true" entropy, so that you don't into the trap of assuming $h(X |X)=0$ (as with the true entropy). Actually $h(X|X)=-infty$. See eg my answer here
$endgroup$
– leonbloy
Nov 30 '18 at 23:37
add a comment |
$begingroup$
One usually writes $h(X)$ for the differetial entropy, not $H(X)$, and with good reason, it reminds you that it's not a "true" entropy, so that you don't into the trap of assuming $h(X |X)=0$ (as with the true entropy). Actually $h(X|X)=-infty$. See eg my answer here
$endgroup$
– leonbloy
Nov 30 '18 at 23:37
$begingroup$
One usually writes $h(X)$ for the differetial entropy, not $H(X)$, and with good reason, it reminds you that it's not a "true" entropy, so that you don't into the trap of assuming $h(X |X)=0$ (as with the true entropy). Actually $h(X|X)=-infty$. See eg my answer here
$endgroup$
– leonbloy
Nov 30 '18 at 23:37
$begingroup$
One usually writes $h(X)$ for the differetial entropy, not $H(X)$, and with good reason, it reminds you that it's not a "true" entropy, so that you don't into the trap of assuming $h(X |X)=0$ (as with the true entropy). Actually $h(X|X)=-infty$. See eg my answer here
$endgroup$
– leonbloy
Nov 30 '18 at 23:37
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
Differential entropy can actually be negative, and thus the upper bound on your information is not correct. Indeed, if they are the same random variable on a continuous domain, then you would hope that the mutual information between them would be infinite (and if they are the same Gaussian, indeed that is the case).
EDIT: I guess I should have clarified: In differential entropy sense, H(Y | X) is not 0; it is negative infinity if X = Y. Any singularity in differential entropy has negative infinite relative uncertainty to any quantized uniform distribution.
answered Jun 6 '18 at 8:54
E-AE-A
2,1221414
$endgroup$
$begingroup$
Thank you for the comment. However, the upper bound ($H(X) leq frac{1}{2} log ( 2 pi e sigma^2 )$) is in en.wikipedia.org/wiki/Differential_entropy. Also, $ -frac{1}{2} log ( 1 - rho^2 ) $ is based on this formular.
$endgroup$
– Inkyu Bang
Jun 6 '18 at 14:29
$begingroup$
See edit above; I realized I did not actually write the negative quantity.
$endgroup$
– E-A
Jun 6 '18 at 19:45
1
$begingroup$
Okay, I got your point. So my assumption on $H ( Y | X ) = 0 $ only holds when $X$ and $Y$ are discrete and $Y=X$. In case of continuous R.Vs, $H ( Y | X ) = - infty $ for $Y=X$. Thus, $ I(X; Y) = H(X) - (-infty) = infty $ for $X=Y$. Thank you!
$endgroup$
– Inkyu Bang
Jun 7 '18 at 1:46
$begingroup$
@InkyuBang Your assumption on $H(X|X)=0$ only holds for the true (Shannon) entropy. It does not apply to differential entropy, which is a different beast. The (true/Shannon) entropy of a (non degenerate) continuous variable is $+infty$, because (among other things) you need (on average) an infinite amount of bits to represent its value.
$endgroup$
– leonbloy
Nov 30 '18 at 23:40
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2809880%2fmutual-information-of-continuous-variables%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Differential entropy can actually be negative, and thus the upper bound on your information is not correct. Indeed, if they are the same random variable on a continuous domain, then you would hope that the mutual information between them would be infinite (and if they are the same Gaussian, indeed that is the case).
EDIT: I guess I should have clarified: In differential entropy sense, H(Y | X) is not 0; it is negative infinity if X = Y. Any singularity in differential entropy has negative infinite relative uncertainty to any quantized uniform distribution.
answered Jun 6 '18 at 8:54
E-AE-A
2,1221414
$endgroup$
$begingroup$
Thank you for the comment. However, the upper bound ($H(X) leq frac{1}{2} log ( 2 pi e sigma^2 )$) is in en.wikipedia.org/wiki/Differential_entropy. Also, $ -frac{1}{2} log ( 1 - rho^2 ) $ is based on this formular.
$endgroup$
– Inkyu Bang
Jun 6 '18 at 14:29
$begingroup$
See edit above; I realized I did not actually write the negative quantity.
$endgroup$
– E-A
Jun 6 '18 at 19:45
1
$begingroup$
Okay, I got your point. So my assumption on $H ( Y | X ) = 0 $ only holds when $X$ and $Y$ are discrete and $Y=X$. In case of continuous R.Vs, $H ( Y | X ) = - infty $ for $Y=X$. Thus, $ I(X; Y) = H(X) - (-infty) = infty $ for $X=Y$. Thank you!
$endgroup$
– Inkyu Bang
Jun 7 '18 at 1:46
$begingroup$
@InkyuBang Your assumption on $H(X|X)=0$ only holds for the true (Shannon) entropy. It does not apply to differential entropy, which is a different beast. The (true/Shannon) entropy of a (non degenerate) continuous variable is $+infty$, because (among other things) you need (on average) an infinite amount of bits to represent its value.
$endgroup$
– leonbloy
Nov 30 '18 at 23:40
add a comment |
$begingroup$
Differential entropy can actually be negative, and thus the upper bound on your information is not correct. Indeed, if they are the same random variable on a continuous domain, then you would hope that the mutual information between them would be infinite (and if they are the same Gaussian, indeed that is the case).
EDIT: I guess I should have clarified: In differential entropy sense, H(Y | X) is not 0; it is negative infinity if X = Y. Any singularity in differential entropy has negative infinite relative uncertainty to any quantized uniform distribution.
answered Jun 6 '18 at 8:54
E-AE-A
2,1221414
$endgroup$
$begingroup$
Thank you for the comment. However, the upper bound ($H(X) leq frac{1}{2} log ( 2 pi e sigma^2 )$) is in en.wikipedia.org/wiki/Differential_entropy. Also, $ -frac{1}{2} log ( 1 - rho^2 ) $ is based on this formular.
$endgroup$
– Inkyu Bang
Jun 6 '18 at 14:29
$begingroup$
See edit above; I realized I did not actually write the negative quantity.
$endgroup$
– E-A
Jun 6 '18 at 19:45
1
$begingroup$
Okay, I got your point. So my assumption on $H ( Y | X ) = 0 $ only holds when $X$ and $Y$ are discrete and $Y=X$. In case of continuous R.Vs, $H ( Y | X ) = - infty $ for $Y=X$. Thus, $ I(X; Y) = H(X) - (-infty) = infty $ for $X=Y$. Thank you!
$endgroup$
– Inkyu Bang
Jun 7 '18 at 1:46
$begingroup$
@InkyuBang Your assumption on $H(X|X)=0$ only holds for the true (Shannon) entropy. It does not apply to differential entropy, which is a different beast. The (true/Shannon) entropy of a (non degenerate) continuous variable is $+infty$, because (among other things) you need (on average) an infinite amount of bits to represent its value.
$endgroup$
– leonbloy
Nov 30 '18 at 23:40
add a comment |
$begingroup$
Differential entropy can actually be negative, and thus the upper bound on your information is not correct. Indeed, if they are the same random variable on a continuous domain, then you would hope that the mutual information between them would be infinite (and if they are the same Gaussian, indeed that is the case).
EDIT: I guess I should have clarified: In differential entropy sense, H(Y | X) is not 0; it is negative infinity if X = Y. Any singularity in differential entropy has negative infinite relative uncertainty to any quantized uniform distribution.
answered Jun 6 '18 at 8:54
E-AE-A
2,1221414
$endgroup$
Differential entropy can actually be negative, and thus the upper bound on your information is not correct. Indeed, if they are the same random variable on a continuous domain, then you would hope that the mutual information between them would be infinite (and if they are the same Gaussian, indeed that is the case).
EDIT: I guess I should have clarified: In differential entropy sense, H(Y | X) is not 0; it is negative infinity if X = Y. Any singularity in differential entropy has negative infinite relative uncertainty to any quantized uniform distribution.
answered Jun 6 '18 at 8:54
E-AE-A
2,1221414
edited Jun 6 '18 at 19:46
answered Jun 6 '18 at 8:54
E-AE-A
2,1221414
answered Jun 6 '18 at 8:54
E-AE-A
2,1221414
answered Jun 6 '18 at 8:54
E-AE-A
2,1221414
2,1221414
$begingroup$
Thank you for the comment. However, the upper bound ($H(X) leq frac{1}{2} log ( 2 pi e sigma^2 )$) is in en.wikipedia.org/wiki/Differential_entropy. Also, $ -frac{1}{2} log ( 1 - rho^2 ) $ is based on this formular.
$endgroup$
– Inkyu Bang
Jun 6 '18 at 14:29
$begingroup$
See edit above; I realized I did not actually write the negative quantity.
$endgroup$
– E-A
Jun 6 '18 at 19:45
1
$begingroup$
Okay, I got your point. So my assumption on $H ( Y | X ) = 0 $ only holds when $X$ and $Y$ are discrete and $Y=X$. In case of continuous R.Vs, $H ( Y | X ) = - infty $ for $Y=X$. Thus, $ I(X; Y) = H(X) - (-infty) = infty $ for $X=Y$. Thank you!
$endgroup$
– Inkyu Bang
Jun 7 '18 at 1:46
$begingroup$
@InkyuBang Your assumption on $H(X|X)=0$ only holds for the true (Shannon) entropy. It does not apply to differential entropy, which is a different beast. The (true/Shannon) entropy of a (non degenerate) continuous variable is $+infty$, because (among other things) you need (on average) an infinite amount of bits to represent its value.
$endgroup$
– leonbloy
Nov 30 '18 at 23:40
add a comment |
$begingroup$
Thank you for the comment. However, the upper bound ($H(X) leq frac{1}{2} log ( 2 pi e sigma^2 )$) is in en.wikipedia.org/wiki/Differential_entropy. Also, $ -frac{1}{2} log ( 1 - rho^2 ) $ is based on this formular.
$endgroup$
– Inkyu Bang
Jun 6 '18 at 14:29
$begingroup$
See edit above; I realized I did not actually write the negative quantity.
$endgroup$
– E-A
Jun 6 '18 at 19:45
1
$begingroup$
Okay, I got your point. So my assumption on $H ( Y | X ) = 0 $ only holds when $X$ and $Y$ are discrete and $Y=X$. In case of continuous R.Vs, $H ( Y | X ) = - infty $ for $Y=X$. Thus, $ I(X; Y) = H(X) - (-infty) = infty $ for $X=Y$. Thank you!
$endgroup$
– Inkyu Bang
Jun 7 '18 at 1:46
$begingroup$
@InkyuBang Your assumption on $H(X|X)=0$ only holds for the true (Shannon) entropy. It does not apply to differential entropy, which is a different beast. The (true/Shannon) entropy of a (non degenerate) continuous variable is $+infty$, because (among other things) you need (on average) an infinite amount of bits to represent its value.
$endgroup$
– leonbloy
Nov 30 '18 at 23:40
$begingroup$
Thank you for the comment. However, the upper bound ($H(X) leq frac{1}{2} log ( 2 pi e sigma^2 )$) is in en.wikipedia.org/wiki/Differential_entropy. Also, $ -frac{1}{2} log ( 1 - rho^2 ) $ is based on this formular.
$endgroup$
– Inkyu Bang
Jun 6 '18 at 14:29
$begingroup$
Thank you for the comment. However, the upper bound ($H(X) leq frac{1}{2} log ( 2 pi e sigma^2 )$) is in en.wikipedia.org/wiki/Differential_entropy. Also, $ -frac{1}{2} log ( 1 - rho^2 ) $ is based on this formular.
$endgroup$
– Inkyu Bang
Jun 6 '18 at 14:29
$begingroup$
See edit above; I realized I did not actually write the negative quantity.
$endgroup$
– E-A
Jun 6 '18 at 19:45
$begingroup$
See edit above; I realized I did not actually write the negative quantity.
$endgroup$
– E-A
Jun 6 '18 at 19:45
1
1
$begingroup$
Okay, I got your point. So my assumption on $H ( Y | X ) = 0 $ only holds when $X$ and $Y$ are discrete and $Y=X$. In case of continuous R.Vs, $H ( Y | X ) = - infty $ for $Y=X$. Thus, $ I(X; Y) = H(X) - (-infty) = infty $ for $X=Y$. Thank you!
$endgroup$
– Inkyu Bang
Jun 7 '18 at 1:46
$begingroup$
Okay, I got your point. So my assumption on $H ( Y | X ) = 0 $ only holds when $X$ and $Y$ are discrete and $Y=X$. In case of continuous R.Vs, $H ( Y | X ) = - infty $ for $Y=X$. Thus, $ I(X; Y) = H(X) - (-infty) = infty $ for $X=Y$. Thank you!
$endgroup$
– Inkyu Bang
Jun 7 '18 at 1:46
$begingroup$
@InkyuBang Your assumption on $H(X|X)=0$ only holds for the true (Shannon) entropy. It does not apply to differential entropy, which is a different beast. The (true/Shannon) entropy of a (non degenerate) continuous variable is $+infty$, because (among other things) you need (on average) an infinite amount of bits to represent its value.
$endgroup$
– leonbloy
Nov 30 '18 at 23:40
$begingroup$
@InkyuBang Your assumption on $H(X|X)=0$ only holds for the true (Shannon) entropy. It does not apply to differential entropy, which is a different beast. The (true/Shannon) entropy of a (non degenerate) continuous variable is $+infty$, because (among other things) you need (on average) an infinite amount of bits to represent its value.
$endgroup$
– leonbloy
Nov 30 '18 at 23:40
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2809880%2fmutual-information-of-continuous-variables%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
One usually writes $h(X)$ for the differetial entropy, not $H(X)$, and with good reason, it reminds you that it's not a "true" entropy, so that you don't into the trap of assuming $h(X |X)=0$ (as with the true entropy). Actually $h(X|X)=-infty$. See eg my answer here
$endgroup$
– leonbloy
Nov 30 '18 at 23:37