Under what conditions does the function C = f(A,B) satisfy H(C|A) = H(B)?












3












$begingroup$


Suppose we have a function $f$,



$$
C = f(A,B),
$$



where $A$, $B$ and $C$ are random variables.



I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:



$$
H(C|A) = H(B),
$$



where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.



Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above holds.










share|cite|improve this question











$endgroup$








  • 1




    $begingroup$
    The function needs to be injective with respect to its second argument.
    $endgroup$
    – Yuval Filmus
    Mar 28 at 8:04










  • $begingroup$
    @YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
    $endgroup$
    – hklel
    Mar 28 at 8:11
















3












$begingroup$


Suppose we have a function $f$,



$$
C = f(A,B),
$$



where $A$, $B$ and $C$ are random variables.



I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:



$$
H(C|A) = H(B),
$$



where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.



Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above holds.










share|cite|improve this question











$endgroup$








  • 1




    $begingroup$
    The function needs to be injective with respect to its second argument.
    $endgroup$
    – Yuval Filmus
    Mar 28 at 8:04










  • $begingroup$
    @YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
    $endgroup$
    – hklel
    Mar 28 at 8:11














3












3








3





$begingroup$


Suppose we have a function $f$,



$$
C = f(A,B),
$$



where $A$, $B$ and $C$ are random variables.



I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:



$$
H(C|A) = H(B),
$$



where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.



Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above holds.










share|cite|improve this question











$endgroup$




Suppose we have a function $f$,



$$
C = f(A,B),
$$



where $A$, $B$ and $C$ are random variables.



I notice that when the random variables are binary (${0, 1}$) and $f$ is the XOR operation, we have the following identity:



$$
H(C|A) = H(B),
$$



where $H(B)$ is the entropy of $B$ and $H(C|A)$ is the conditional entropy of $C$ given $A$.



Obviously this is not true for a general $f$. What I am interested to know is, is there a set of conditions on $f$ and $A,B,C$, under which the identity above holds.







information-theory






share|cite|improve this question















share|cite|improve this question













share|cite|improve this question




share|cite|improve this question








edited 2 days ago







hklel

















asked Mar 28 at 7:20









hklelhklel

1255




1255








  • 1




    $begingroup$
    The function needs to be injective with respect to its second argument.
    $endgroup$
    – Yuval Filmus
    Mar 28 at 8:04










  • $begingroup$
    @YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
    $endgroup$
    – hklel
    Mar 28 at 8:11














  • 1




    $begingroup$
    The function needs to be injective with respect to its second argument.
    $endgroup$
    – Yuval Filmus
    Mar 28 at 8:04










  • $begingroup$
    @YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
    $endgroup$
    – hklel
    Mar 28 at 8:11








1




1




$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
Mar 28 at 8:04




$begingroup$
The function needs to be injective with respect to its second argument.
$endgroup$
– Yuval Filmus
Mar 28 at 8:04












$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
Mar 28 at 8:11




$begingroup$
@YuvalFilmus Ah that makes sense! I didn't know the term "injective". Do you want to elaborate a bit and write an answer so I can upvote it?
$endgroup$
– hklel
Mar 28 at 8:11










2 Answers
2






active

oldest

votes


















4












$begingroup$

The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.



Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$

Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.






share|cite|improve this answer











$endgroup$









  • 1




    $begingroup$
    $H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
    $endgroup$
    – xskxzr
    Mar 28 at 8:45








  • 3




    $begingroup$
    The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
    $endgroup$
    – Emil Jeřábek
    Mar 28 at 10:03





















8












$begingroup$

Note



begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}



so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.



For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.






share|cite|improve this answer











$endgroup$









  • 2




    $begingroup$
    The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
    $endgroup$
    – Emil Jeřábek
    Mar 28 at 10:04










  • $begingroup$
    @EmilJeřábek Thanks, fixed.
    $endgroup$
    – xskxzr
    Mar 28 at 10:34












Your Answer





StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "419"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcs.stackexchange.com%2fquestions%2f106156%2funder-what-conditions-does-the-function-c-fa-b-satisfy-hca-hb%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























2 Answers
2






active

oldest

votes








2 Answers
2






active

oldest

votes









active

oldest

votes






active

oldest

votes









4












$begingroup$

The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.



Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$

Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.






share|cite|improve this answer











$endgroup$









  • 1




    $begingroup$
    $H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
    $endgroup$
    – xskxzr
    Mar 28 at 8:45








  • 3




    $begingroup$
    The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
    $endgroup$
    – Emil Jeřábek
    Mar 28 at 10:03


















4












$begingroup$

The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.



Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$

Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.






share|cite|improve this answer











$endgroup$









  • 1




    $begingroup$
    $H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
    $endgroup$
    – xskxzr
    Mar 28 at 8:45








  • 3




    $begingroup$
    The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
    $endgroup$
    – Emil Jeřábek
    Mar 28 at 10:03
















4












4








4





$begingroup$

The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.



Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$

Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.






share|cite|improve this answer











$endgroup$



The following answer assumes that $A,B$ are independent, and that $A,B$ have full support on their respective domains (the latter is without loss of generality). For the general case, see the other answer.



Let's write your equation in a slightly different way:
$$
H(B) = H(f(A,B)|A) = operatorname*{mathbb{E}}_{a sim A} H(f(a,B)).
$$

Clearly $H(f(a,B)) leq H(B)$, with equality if and only if $f(a,b_1) neq f(a,b_2)$ whenever $b_1 neq b_2$. We deduce that $H(B) = H(f(A,B)|A)$ if and only if $f$ is injective in its second argument, i.e., for all $a$ and $b_1 neq b_2$, we have $f(a,b_1) neq f(a,b_2)$.







share|cite|improve this answer














share|cite|improve this answer



share|cite|improve this answer








edited Mar 28 at 10:12

























answered Mar 28 at 8:28









Yuval FilmusYuval Filmus

195k15184349




195k15184349








  • 1




    $begingroup$
    $H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
    $endgroup$
    – xskxzr
    Mar 28 at 8:45








  • 3




    $begingroup$
    The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
    $endgroup$
    – Emil Jeřábek
    Mar 28 at 10:03
















  • 1




    $begingroup$
    $H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
    $endgroup$
    – xskxzr
    Mar 28 at 8:45








  • 3




    $begingroup$
    The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
    $endgroup$
    – Emil Jeřábek
    Mar 28 at 10:03










1




1




$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
Mar 28 at 8:45






$begingroup$
$H(f(A,B)|A)=mathbb{E}_aH(f(a,B)|A=a)$, and $H(f(a,B)|A=a)$ is different from $H(f(a,B))$ since $A$ and $B$ may be dependent.
$endgroup$
– xskxzr
Mar 28 at 8:45






3




3




$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:03






$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:03













8












$begingroup$

Note



begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}



so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.



For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.






share|cite|improve this answer











$endgroup$









  • 2




    $begingroup$
    The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
    $endgroup$
    – Emil Jeřábek
    Mar 28 at 10:04










  • $begingroup$
    @EmilJeřábek Thanks, fixed.
    $endgroup$
    – xskxzr
    Mar 28 at 10:34
















8












$begingroup$

Note



begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}



so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.



For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.






share|cite|improve this answer











$endgroup$









  • 2




    $begingroup$
    The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
    $endgroup$
    – Emil Jeřábek
    Mar 28 at 10:04










  • $begingroup$
    @EmilJeřábek Thanks, fixed.
    $endgroup$
    – xskxzr
    Mar 28 at 10:34














8












8








8





$begingroup$

Note



begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}



so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.



For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.






share|cite|improve this answer











$endgroup$



Note



begin{align}
0&=H(C|A,B)\
&=H(A,B,C)-H(A,B)\
&=H(B|A,C)+H(C|A)+H(A)-H(A,B)quadtext{(chain rule)}\
&=H(B|A,C)+H(C|A)-H(B|A),
end{align}



so $H(C|A)=H(B)$ is equivalently $H(B|A,C)+H(B)-H(B|A)=0$. Also note $H(B|A,C)ge 0$ and $H(B)ge H(B|A)$, your condition is equivalently $H(B|A,C)=0wedge H(B)=H(B|A)$.



For a human-readable explanation, $H(B|A,C)=0$ means $B$ is determined by $A$ and $C$, that is, for any fixed $a$ in the support of $A$, $f(a,b)$ as a function of $b$ with domain ${bmid mathrm{Pr}{A=a, B=b}>0}$ is an injection. $H(B)=H(B|A)$ means $A$ and $B$ are independent of each other.







share|cite|improve this answer














share|cite|improve this answer



share|cite|improve this answer








edited Mar 28 at 10:34

























answered Mar 28 at 8:38









xskxzrxskxzr

4,07921033




4,07921033








  • 2




    $begingroup$
    The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
    $endgroup$
    – Emil Jeřábek
    Mar 28 at 10:04










  • $begingroup$
    @EmilJeřábek Thanks, fixed.
    $endgroup$
    – xskxzr
    Mar 28 at 10:34














  • 2




    $begingroup$
    The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
    $endgroup$
    – Emil Jeřábek
    Mar 28 at 10:04










  • $begingroup$
    @EmilJeřábek Thanks, fixed.
    $endgroup$
    – xskxzr
    Mar 28 at 10:34








2




2




$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:04




$begingroup$
The conclusion that $f$ is injective in the second argument is only correct if $Pr(A=a)>0$ and $Pr(B=b)>0$ for all $(a,b)inoperatorname{dom}(f)$.
$endgroup$
– Emil Jeřábek
Mar 28 at 10:04












$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
Mar 28 at 10:34




$begingroup$
@EmilJeřábek Thanks, fixed.
$endgroup$
– xskxzr
Mar 28 at 10:34


















draft saved

draft discarded




















































Thanks for contributing an answer to Computer Science Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


Use MathJax to format equations. MathJax reference.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcs.stackexchange.com%2fquestions%2f106156%2funder-what-conditions-does-the-function-c-fa-b-satisfy-hca-hb%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Plaza Victoria

In PowerPoint, is there a keyboard shortcut for bulleted / numbered list?

How to put 3 figures in Latex with 2 figures side by side and 1 below these side by side images but in...