Under what conditions does the function C = f(A,B) satisfy H(C|A) = H(B)?
$begingroup$
Suppose we have a function $f$,
$$
C = f(A,B),
$$
where $A$, $B$ and $C$ are random variables.
I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:
$$
H(C|A) = H(B),
$$
where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.
Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above holds.
information-theory
$endgroup$
add a comment |
$begingroup$
Suppose we have a function $f$,
$$
C = f(A,B),
$$
where $A$, $B$ and $C$ are random variables.
I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:
$$
H(C|A) = H(B),
$$
where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.
Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above holds.
information-theory
$endgroup$
1
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
Mar 28 at 8:04
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
Mar 28 at 8:11
add a comment |
$begingroup$
Suppose we have a function $f$,
$$
C = f(A,B),
$$
where $A$, $B$ and $C$ are random variables.
I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:
$$
H(C|A) = H(B),
$$
where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.
Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above holds.
information-theory
$endgroup$
Suppose we have a function $f$,
$$
C = f(A,B),
$$
where $A$, $B$ and $C$ are random variables.
I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:
$$
H(C|A) = H(B),
$$
where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.
Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above holds.
information-theory
information-theory
edited 2 days ago
hklel
asked Mar 28 at 7:20
hklelhklel
1255
1255
1
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
Mar 28 at 8:04
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
Mar 28 at 8:11
add a comment |
1
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
Mar 28 at 8:04
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
Mar 28 at 8:11
1
1
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
Mar 28 at 8:04
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
Mar 28 at 8:04
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
Mar 28 at 8:11
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
Mar 28 at 8:11
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.
Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$
Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.
$endgroup$
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
Mar 28 at 8:45
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:03
add a comment |
$begingroup$
Note
begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}
so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.
For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.
$endgroup$
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:04
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
Mar 28 at 10:34
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "419"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcs.stackexchange.com%2fquestions%2f106156%2funder-what-conditions-does-the-function-c-fa-b-satisfy-hca-hb%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.
Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$
Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.
$endgroup$
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
Mar 28 at 8:45
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:03
add a comment |
$begingroup$
The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.
Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$
Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.
$endgroup$
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
Mar 28 at 8:45
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:03
add a comment |
$begingroup$
The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.
Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$
Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.
$endgroup$
The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.
Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$
Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.
edited Mar 28 at 10:12
answered Mar 28 at 8:28
Yuval FilmusYuval Filmus
195k15184349
195k15184349
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
Mar 28 at 8:45
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:03
add a comment |
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
Mar 28 at 8:45
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:03
1
1
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
Mar 28 at 8:45
$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
Mar 28 at 8:45
3
3
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:03
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:03
add a comment |
$begingroup$
Note
begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}
so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.
For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.
$endgroup$
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:04
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
Mar 28 at 10:34
add a comment |
$begingroup$
Note
begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}
so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.
For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.
$endgroup$
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:04
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
Mar 28 at 10:34
add a comment |
$begingroup$
Note
begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}
so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.
For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.
$endgroup$
Note
begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}
so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.
For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.
edited Mar 28 at 10:34
answered Mar 28 at 8:38
xskxzrxskxzr
4,07921033
4,07921033
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:04
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
Mar 28 at 10:34
add a comment |
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:04
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
Mar 28 at 10:34
2
2
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:04
$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:04
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
Mar 28 at 10:34
$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
Mar 28 at 10:34
add a comment |
Thanks for contributing an answer to Computer Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcs.stackexchange.com%2fquestions%2f106156%2funder-what-conditions-does-the-function-c-fa-b-satisfy-hca-hb%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
Mar 28 at 8:04
$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
Mar 28 at 8:11