In The Lord of the Rings: The Return of the King when Pippin took the Palantír from Gandalf and Sauron saw him, why did Aragorn faint when he took it but later in the movie he challenged Sauron via the Palantír. Why did he faint before?

The scene in question occurs differently in the books (The Two Towers)

Closer and closer he [Pippin] bent, and then became rigid; his lips moved soundlessly for a while. Then with a strangled cry he fell back and lay still.

The cry was piercing. The guards leapt down from the banks. All the camp was soon astir.

[...]

'I think all will be well now,' answered Gandalf. 'He was not held long, and hobbits have an amazing power of recovery. The memory, or the horror of it, will probably fade quickly. Too quickly, perhaps. Will you, Aragorn, take the Orthanc-stone and guard it? It is a dangerous charge.'

'Dangerous indeed, but not to all,' said Aragorn. 'There is one who may claim it by right. For this assuredly is the palantir of Orthanc from the treasury of Elendil, set here by the Kings of Gondor. Now my hour draws near. I will take it.'

Gandalf looked at Aragorn, and then, to the surprise of the others, he lifted the covered Stone, and bowed as he presented it.

_{The Lord of the Rings, The Two Towers, The Palantir}

Aragorn never touches the stone at that time in the books, so we can't tell why he fainted in the movie. Aragorn is the rightful owner of the stone, and proves that he is able to use it in the latter scene you mentioned (also occurs in the books). Why he fainted in the movie is probably just to make the scene more dramatic, but a more definite answer would be because...

He did not, at that time, expect Sauron to be on the other end

Before Pippin used the stone, neither he nor Gandalf knew the stone's purpose. Only when Pippin used it were they able to find out who the stone was linked to. In the aforementioned latter scene where Aragorn wrests control of the palantir from Sauron, he already knows what he's up against.

'You forget to whom you speak,' said Aragorn sternly, and his eyes glinted. 'What do you fear that I should say to him? Did I not openly proclaim my title before the doors of Edoras? Nay, Gimli, I am the lawful master of the Stone, and I had both the right and the strength to use it, or so I judged. The right cannot be doubted. The strength was enough - barely.'

He's described as 'aged rapidly' when he comes out of the Hornburg after using the Stone. That shows the mental strain of using the stone, coupled with the fact that he was not prepared during the scene in question.

Mental strain of using a Palantir

The Unfinished Tales has a section dedicated to the palantiri.

The use of the palantiri was a mental strain, especially on men of later days not trained to the task, and no doubt in addition to his anxieties his strain contributed to Denethor's "grimness".

_{Unfinished Tales, The Palantiri, Note 13}

So simply touching the stone and also not knowing who's on the other end are likely reasons that caused Aragorn to lose consciousness.