Csdrhrt

I have defined function A as A: D*E --> F and function B as B: S*T --> R.

Now, I want to define function C which gets an instance of A functions and an instance of B functions as inputs and returns a real number.

What is the correct notation for this function?

Is the following notation correct?

C: A(.)*B(.) --> R

Mathematical function notation describes the domain and range of the function in the abstract. So when you have a function $$A: D*E rightarrow {mathbb R}$$ this is a function defined on some product of sets $D*E$ (the domain) taking values in the (ordinary) real numbers (the range).

If your function is taking the output of functions $A$ and $B$ then it is acting on two real numbers, and so $C:{mathbb R}times{mathbb R} rightarrow {mathbb R}$.

If, however, you have an operator $C$ that operates on the domains of functions $A$ and $B$, the domain of $C$ must be jointly the domains of $A$ and $B$. So with your notation,
$$ C: D*E times S*T rightarrow {mathbb R}$$
where $times$ is the Cartesian product of the two domains.

$C:A*B rightarrow {mathbb R}$ isn't right in general because $A$ and $B$ are functions, not sets. ("In general": sometimes notation is abused so that a set is defined by a function, e.g. $[f=alpha]$ to describe a hyperplane.)

Since this is a mathematics site, let's use mathematical notation.

There is no widely accepted mathematical notation that looks like this:
A: D*E --> F.
However, there is a notation that looks like
$A: D times E to F,$
which says that $A$ is a function to which we give an element of the set $D$ and an element of the set $E,$ upon which $A$ produces an element of the set $F.$

Note that in this notation, $D,$ $E,$ and $F$ are sets that may contain many different elements, but $A$ is the name of a particular function.
If you want to describe a function that takes other functions as input, it does not usually make sense to use the name $A$ as part of the "type" of the new function,
any more than it makes sense to use the name of the number $3$ as part of the "type" of a function. What I mean is, you would not write
$f: 3 times mathbb R to mathbb R.$
Technically, you could write $f: {3} times mathbb R to mathbb R,$
but that's rather silly; it says the first argument to the function must always be $3,$ which kind of defeats the purpose of giving the argument to the function in the first place.

You certainly can have a function that acts on functions. Here is an example:
$$ C: (mathbb R times mathbb R to mathbb R) times
(mathbb R times mathbb R to mathbb R) to mathbb R.$$
This says that $C$ is a function that takes two arguments, each of which itself is a functions that takes two real numbers arguments, and $C$ returns a real number as its result.
As a more specific example, we could say that $C$ is the function such that
$$ C(f,g) = f(0,0) + g(1,1) $$
for any two functions $f: mathbb R times mathbb R to mathbb R$
and $g: mathbb R times mathbb R to mathbb R.$