Csdrhrt

I would like to see that $texttt{Nil}_n$ is isomorphic to $h^{mathbb{Z}[x]/(x^n)}$ as categories.

$texttt{Nil}_n: texttt{Rings} longrightarrow texttt{Sets}$ is the functor that sends a ring $R$ to ${x in R | x^n = 0}$.

$h^{mathbb{Z}[x]/(x^n)}: texttt{Mod}_R longrightarrow texttt{Mod}_R$ is the functor where $mathbb{Z}[x]/(x^n)$ is an $R$-module and it sends an $R$-module $N$ to $Hom(mathbb{Z}[x]/(x^n),N)$.

I know that two categories are isomorphic if when there exists two functors as above, they satisfy also the above conditions.

I want to see that if there exists two functors $F: texttt{Nil}_n longrightarrow h^{mathbb{Z}[x]/(x^n)}$ and $G: h^{mathbb{Z}[x]/(x^n)} longrightarrow texttt{Nil}_n$, then $FG=id_{h^{mathbb{Z}[x]/(x^n)}}$ and $GF=id_{texttt{Nil}_n}$.

But my problem comes when I have to find / see which are these functors. Does $F$ and $G$ even exist? How can I find them? Are they known functors?

I want to see that in order to see that $texttt{Nil}_n$ is represented by the ring $h^{mathbb{Z}[x]/(x^n)}$ and the natural equivalence

$tau: h^{mathbb{Z}[x]/(x^n)} longrightarrow texttt{Nil}_n$

given (for each ring $R$) by

$tau_R: h^{mathbb{Z}[x]/(x^n)}(R) longrightarrow texttt{Nil}_n(R): f ↦ f(bar{x})$.

I was thinking on first showing that $texttt{Nil}_n$ and $h^{mathbb{Z}[x]/(x^n)}$ are isomorphic and then showing that there exists a natural iso $mu: h^{mathbb{Z}[x]/(x^n)} longrightarrow texttt{Nil}_n$ determined by $mu_{mathbb{Z}[x]/(x^n)}(Id_{mathbb{Z}[x]/(x^n)})$ and it is equal to $tau$.

Ok, several problems. As Kevin Carlson points out in the comments, $newcommandNil{operatorname{Nil}}Nil_n$ and (if we define $A=Bbb{Z}[x]/(x^n)$) $h^A$ are functors, not categories.

Thus we need to show that they are isomorphic as functors.

Isomorphisms of functors

Recall, then, the definition of natural transformation, which is the notion of a morphism between functors. Let $F,G:Cto D$ be functors from a category $C$ to a category $D$. Then a natural transformation $theta: Fto G$ is a family of maps, $theta_x: F(x)to G(x)$ for each $xin C$ satisfying the following commutative diagram for every morphism $f:xto y$ in $C$:
$$require{AMScd}begin{CD}F(x) @>F(f)>> F(y) \ @Vtheta_xVV @Vtheta_yVV\G(x) @>>G(f)> G(y) end{CD}$$

An isomorphism of functors (usually called a natural isomorphism) then is a natural transformation all of whose components are isomorphisms. (Check that this is equivalent to the natural transformation having an inverse natural transformation)

How to apply this to your question

First of all, you have the domains and codomains of your functors wrong. Well $Nil_n$ is correct. It is a functor from $newcommandRing{mathbf{Ring}}newcommandSet{mathbf{Set}}Ring$ to $Set$, but $h^A(R)=newcommandHom{operatorname{Hom}}Hom_Ring(A,R)$ is actually also a functor from $Ring$ to $Set$, not from right $R$-modules to right $R$-modules.

Thus you want to find a family of maps for every ring $RinRing$, $tau_R:h^A(R)to Nil_n(R)$ with $tau_R$ an isomorphism of sets (i.e. a bijection) for every ring $R$.
You should also check that $tau_R$ is natural. I.e. that the following diagram commutes for every map $phi :Rto S$ of rings:
$$begin{CD}h^A(R) @>h^A(phi)>> h^A(S) \ @Vtau_RVV @Vtau_SVV\Nil_n(R) @>>Nil_n(phi)> Nil_n(S) end{CD}$$

Note: I've used $tau_R$, because you've already identified the correct family of maps $tau_R$ in your question, you just have to prove it has the right properties.

Edit: In response to your comment asking how is $tau_R$ defined exactly.

Well, as you mentioned, the elements of $h^A(R)$ are ring homomorphisms $f:Ato R$, and the elements of $Nil_n(R)$ are elements $r$ of $R$ with $r^n=0$. Thus $tau_R$ should take ring homomorphisms $f:Ato R$ and produce appropriate elements of $R$.

Well, $A=Bbb{Z}[x]/(x^n)$, so let $bar{x}$ be the image of $xin Bbb{Z}[x]$ in the quotient, $A$. Then if $f:Ato R$ is a ring homomorphism, $r=f(bar{x})$ is an element of $R$, and moreover, $r^n=f(bar{x})^n=f(bar{x}^n)=f(0)=0$, so $r$ is in $Nil_n(R)$.

Hint: there is a universal property of $mathbb Z[x]$ as the free ring on one generator. This means there is a functorial identifcation $$Hom_{Rings}(mathbb Z[x], R) = Hom_{Sets}({x},R) = {xin R} = R$$

Now describe this equality in terms of an isomorphism of functors and adapt the proof to your situation. (Yes the functors natural transformations exist and are well known.)