Csdrhrt

How to calculate $frac{partialTheta}{partial L}$ if I know $frac{partial L}{partialTheta}$?

Suppose I have a halved sum of squared errors loss:

$$L(Theta)=frac{1}{2}sum^{M}(y-h(XcircTheta))^2$$

with constant inputs $Xinmathbb{R}^{Mtimes{in}}$, parameters $Thetainmathbb{R}^{intimes out}$ and ground truth/hypothesis ${y, h(z)}inmathbb{R}^{Mtimes out}$

Then according to this article, $frac{partial L}{partialTheta}$ can be computed by multiplying all partial derivatives in the path between $L(Theta)$ and $Theta$.

So, if I give a name to all computations, and write down the partial derivative between it and its inputs like:

$$f=frac{1}{2}etexttt{ and } frac{partial f}{partial e}=frac{1}{2}$$

$$e=sum^M dtexttt{ and } frac{partial e}{partial d}=1$$

$$d=c^2texttt{ and } frac{partial d}{partial c}=2c$$

$$c=y-btexttt{ and } frac{partial c}{partial b}=-1$$

$$b=h(a)texttt{ and } frac{partial b}{partial a}=h'(a)$$

$$a=Xcirc Thetatexttt{ and } frac{partial a}{partial Theta}=X$$

(Note: $frac{partial}{partial Y}Xcirc Y=X$ from Matrix Cookbook rule (38))

Then

$$frac{partial}{partialTheta}L(Theta)=frac{partial f}{partial e}frac{partial e}{partial d}frac{partial d}{partial c}frac{partial c}{partial b}frac{partial b}{partial a}frac{partial a}{partial Theta}$$

If replaced I get:

$$frac{partial}{partialTheta}L(Theta)=dcirc-h'(a)circ X$$

Which is:

$$frac{partial}{partialTheta}L(Theta)=(y-h(XcircTheta))circ -h'(XcircTheta)circ X$$

Now after adding some transpositions and multiplying everything, I get a shape of $frac{partial}{partialTheta}L(Theta)inmathbb{R}^{Mtimes in}$, which is different from the shape for $Thetainmathbb{R}^{intimes out}$. But everything I can find online, tells me, that in order to perform a weight update with Gradient Descent, the shape of $frac{partial}{partialTheta}L(Theta)$ should be the same as $Theta$.

What am I doing wrong? I think I actually want $frac{partialTheta}{partial L}$ but I'm not sure if it makes even sense at all.
I'd guess it has to do with, that I treat $Theta$ as a single value, which it is not.

Ok, if i understand correctly what you are asking then $L(Theta)$ should be written as
$$L(Theta)=sum_{1leq ileq M, 1leq jleq text{out}}(Y-h(XTheta))_{i,j}=||Y-h(XTheta)||^2_F=(Y-h(XTheta))cdot(Y-h(XTheta))$$
Where $L(Theta)in R; Y,h(XTheta)in R^{Mtimestext{out}}; Xin R^{Mtimestext{in}}, Thetain R^{text{in}timestext{out}}$, $||cdot||_F$ - Frobenius matrix norm; and elementwise matrix product $cdot$, i.e. $Acdot B=sum_{i,j}A_{i,j}B_{i,j}$ . So, using some rules of matrix differential calculus, which you can find here - Practical Guide to Matrix Calculus for Deep Learning - Andrew Delong, for example.

In particular, we will need rule (6) from the paper - $Acdot(BC)=B^TAcdot C$, rule (13) for elementwise matrix product - $d(Acdot B)=dAcdot B+Acdot dB$, rule (12) for ordinary matrix product - $d(AB)=dAB+AdB$ and the fact $(*)$ that differential of a matrix is the matrix of differentials, i.e $(df(A))_{i,j}=df_{i,j}(A), f(X)in R^{mtimes n}$ . So
$$dL(Theta)=d[(Y-h(XTheta))cdot(Y-h(XTheta))]=\d(Y-h(XTheta))cdot(Y-h(XTheta))+(Y-h(XTheta))cdot d(Y-h(XTheta))=\-dh(XTheta)cdot(Y-h(XTheta))+(Y-h(XTheta))cdot(-dh(XTheta))=\-2(Y-h(XTheta))cdot dh(XTheta)$$
So we have $dL(Theta)=-2(Y-h(XTheta))cdot dh(XTheta)$. Now, if there are no further restrictions on the form of the function $h$, then we can do only something like this using $(*)$ and (6)
$$dL(Theta)=-2(Y-h(XTheta))cdot dh(XTheta)=sum_{1leq ileq M, 1leq jleq text{out}}(2h(XTheta)-2Y)_{i,j}dh_{i,j}(XTheta)=sum_{1leq ileq M, 1leq jleq text{out}}(2h(XTheta)-2Y)_{i,j}h'_{i,j}(XTheta)cdot d(XTheta)=\sum_{1leq ileq M, 1leq jleq text{out}}(2h(XTheta)-2Y)_{i,j}h'_{i,j}(XTheta)cdot XdTheta=\sum_{1leq ileq M, 1leq jleq text{out}}(2h(XTheta)-2Y)_{i,j}X^Th'_{i,j}(XTheta)cdot dTheta=\
[sum_{1leq ileq M, 1leq jleq text{out}}(2h(XTheta)-2Y)_{i,j}X^Th'_{i,j}(XTheta)]cdot dTheta$$
Where each $dh_{i,j}in R$, consequently the derivative $h'_{i,j}in R^{Mtimestext{out}}$, i.e. of dimensions of $XTheta$ . Note tthat now the differential is exactly of the form (17) in the paper, i.e. $dL(Theta)=Dcdot dTheta$, and $D$ is the sum of the matrices, shown in the previous derivation. This means that the needed derivative is exactly
$$frac{partial}{partialTheta}L(Theta)=D=sum_{1leq ileq M, 1leq jleq text{out}}(2h(XTheta)-2Y)_{i,j}X^Th'_{i,j}(XTheta)$$
Note also that $(2h(XTheta)-2Y)_{i,j}in R, X^Tin R^{text{in}times M}, h'_{i,j}in R^{Mtimestext{out}}$, so $(2h(XTheta)-2Y)_{i,j}X^Th'_{i,j}(XTheta)in R^{text{in}timestext{out}}$ and thus their sum $Din R^{text{in}timestext{out}}$ has exactly the same dimensions as $Theta$.