Players

The players are new and possibly overwhelmed by having too many options. Thankfully, this is easily solved by talking to them and reiterating that the game you play is more open world.

Make sure to ask the players what they would feel would be a good prompt for them to make new decisions.

You can encourage them to do just that by describing thing in more detail. The castle seemed to be in one piece and standing just fine. So, instead the door should have had arrows in it, scorched marks, and there should be crows everywhere. The walls look impenetrable but there's a massive crack in one as if a sally was done -- clue for caves, assault, and of course another mean to get in.

GM

You are doing a little of rail roading: You planed the castle exploration, getting into the caves, and interesting encounters. Then the players did not follow your plan. That is your fault for having a plan.

A fix is instead of saying "no", say "yes, and …" or "yes, but … ", or "yes. How? …". This leads the players to feel (rightly) that they have agency in the world. It allows you, as the GM, to add complications to ensure that the story moves in the right direction.

So, in your example: they spend time trying to climb up. Great, let them. Now, they are in the castle, ready to explore it. However, while climbing they made such a racket that the orcs heard them. Cue: Drums! Drums in the Deep!.

As for clues, remember that you know what things are clues and what are not. To the players, all your say have equal value. Therefore, if you say something like "fluff, fluff, clue, fluff" all your players hear "thing, thing, thing, thing". Which one is important? This is where the 3 clue rule applies: make sure any vital clue is shown at least three times.

If they players do something weird, it is quite alright to question it. For example, the players knock on the door. You could say "You knock. After a few heart beats, you hear bunch of birds take off from within the castle. As you look up, crows stream out towards the sky. Something wet drops on X's face. It is a drop of blood, probably carried by a crow overhead. There is no response from the castle. What do you do?"

When dealing with new players there are three key things that I'd use to help them understand that the game is open to options and ideas rather than following a single course and encourage them to think for themselves and get ideas rather than just follow a course.

Talk to them out of game

Getting them to understand what and how a game works out of character, either during the game or before it to help them understand how things work - for new players who are used to games with defined rule sets (like Settlers of Catan) or stories where there are only a single path this can be a bit of a confusing concept; you can help this by prompting them out of character when they first played and laying the options out for them clearly in out of character comments, they also may not be listening or remembering other options.

You're already telling them this after they've been failing for a bit, but try to lay the options out for them so they realise there are paths earlier.

GM: "You bang on the door and it's very loud, you're pretty sure that no one could have missed that noise who was on the other side. Now players, just to remind you; there's a low balcony that you could possibly climb up to if you were athletic, the door itself could be battered down with an axe if you had a mind and a lot of time, and there's also that grate down at the bottom of the moat that could be crawled past, there are other possibilities, if you can think of something you can try it."

Spoon-feeding ideas and options can help them, and then you can move into playing it through descriptions...

Use visual cues

The options above can then be woven into descriptions rather than telling the players openly what their options are as they get more experienced, describe other doors, loose bricks, low windows and so on - lowering the amount of OOC info you give so that they'll hopefully start picking up ideas from your description, remind them to listen to the descriptions and repeat if necessary - it can be a bit tedious at first, but they're learning so you need to be patient :)

Sidekick helpers

I also find it helpful if players have a familiar, sidekick, or any sort of animal (my MERP group had a Rohan warhorse that was the smartest person in the group) that can help feed these ideas or quest for ideas to them without breaking into OOC.

GM: "You've hammered at the door to the keep for ten minutes now without any help. Siggy the Warhorse has wandered off and is chomping down at a large patch of tasty green grass right next to a large crack in the wall about thirty feet away, looks like it might be wide enough to squeeze through."

New Rule: One Scene, One Try

There are several exhaustive answers here, but I wish to focus on a particular issue I often encounter:

they spent almost all of the remainder of the session trying to get a grappling-hook style thing up through the little hole and then climb out.

Did it go something like this?

DM: You see a small opening in the top of the cave with light shining through.

Player 1: my character takes out his grappling hook and 50ft of rope and throws it up at the opening.

DM: Ok. Make a roll.

Player 1 rolls dice.

DM: Your grappling hook soars through the air but misses the opening.

Player 2: Wait wait wait. My character picks up the grappling hook and tries.

Player 2 rolls dice.

DM: The grappling hook again soars through the air, misses the opening and strikes the cave wall with a loud clank.

Player 1: Let's try again, but this time I have my character assist the other character.

Player 2 rolls dice and with assistance bonus.

DM: The grappling hook misses. Again.

Player 3: What if we try it from on top of this rock over here?

Player 4: Spider climb could help...
etc..

In a dice based game this play style is guaranteed to produce:

1. A high enough dice roll to tackle any problem (i.e. a natural 20)


2. Waste an inordinate amount of time on small tasks.

If you have players that fall into this play style, you need to be more firm with not allowing continuous retries. I prefer the one scene one try rule: the players, collectively, get one try per scene* and must abide by the results. The above example would play out like:

DM: You see a small opening in the top of the cave with light shining through.

Player 1: my character takes out his grappling hook and 50ft of rope and throws it up at the opening.

DM: Hold on. The 'one try' rule applies to this scene. Take five minutes and discuss the best way you can work together on this roll.

Players discuss strategies for five minutes.

DM: Ok time is up. What is the plan?

Player 1: Player 3 will have their character cast spider climb on my character. They then walk as far up the side of the cave as possible while carrying the grappling hook. If I am unable to reach the opening, my character will attempt to throw the hook at the opening. Player 2's character will give me an assistance bonus by helping me aim the throw and keep the other end of the rope from tangling.

DM: Great! If everyone agrees with this plan, cast spider climb and make a roll.

Player 1 rolls dice.

DM: Here's what happens..

*A scene is defined as a part of the game that is larger than an 'encounter' but smaller than a 'session'. A night spent drinking at a tavern, a visit to an abandoned castle, a walk through a creepy forest are all examples of what I call scenes. A single session may have several scenes. From the description of your session, I would say the visiting of the castle is a scene that includes the cave. The characters could not just leave the cave and go back in and expect a retry (it's the same "scene"). However, if they left and traveled into the forest only to be chased by owl bears back into the cave, that would be a new scene and merit a retry. As with most things in tabletop RP, the DM will need to use their judgement when a scene ends for the purposes of re-rolls.

One of my favorite experiences was when our party was presented with a quest and we went "outside" of the story line: we were asked to be couriers of a letter, and a convoy was traveling to the said destination tomorrow. I turned to the group and said, "Guys, let's go today, deliver the message, and come back by tonight." My group went for it, and it completely circumvented a MASSIVE amount of the story line.

The cool part: my GM rolled with it (no pun intended).

Instead of requiring us to stay overnight and go with the convoy, he completely designed a new story arc for us. We ended up fighting the same bad guy in the end, but an entire branch of his design went out the window (because we had prevented the death of a city mayor that was supposed to be assassinated while we were with the convoy). After we had finished the campaign, the party had enjoyed it because we got to do what we wanted (which was circumvent 'the bad guy'), and he enjoyed it because it allowed him to do more design between sessions and think more on his feet.

It may be outside of your play style, but if the party wants to do something different, roll with it. Come up with new content, even if it means that your original story goes out the window, but keep it real with the story line that you had planned. If the castle is abandoned, leave it abandoned. If there are supposed to be goblins in the cave, have them show up while the party is half way up the grappling rope. If going into the castle now would prevent a thief from looting some magic item from the castle which would be used later in the story, change the story in accordance with how the party acquiring the object would affect things.

It's difficult to pull off sometimes, but it makes for an awesome play experience in the end, which is the whole purpose of playing the game in the first place.

the PCs never seem to look for clues and just expect everything to be
there because they are supposed to 'use' it

That's not what is happening. I remember being a novice. In fact I never went on to play seriously because I found the games too rule bound.

Novices get enthralled by being submerged in a fantasy land. That is in itself immensely enjoyable. They don't think things are there "because they are supposed to use them", They think things are there because they are there - just like in the real world.

In the real world throwing a grappling hook repeatedly seems like an obvious plan. Keep going until someone succeeds.

You can just say that the opening is 100 feet up and the rope isn't long enough or no-one is strong enough to throw it that high. A simple in-world explanation related to people's real life experiences will satisfy them and they'll move on.

P.S. The opening may not be meant to be used but the grappling hook clearly is. Why wouldn't they want to try and use it?

In addition to the other answers, one of the things I remember seeing a while back about GMing well was giving the players multiple things to focus in on as potentially important in order to disguise the thing the players actually need.

For example:

(And yes, I yoinked example this from wherever I heard it, but I'm having to make up my own details)

You open the door and stir up a thick layer of dust on the floor of this old study. A desk in one corner next to a bookshelf has a thick tome resting on it, the red velvet cover standing out despite the dust that fills the room and covers everything. A shaft of light from the room's only window falls across the messy desk and glints off the book's brass corners.

What object here is important?

The red velvet book. Clearly.

Your players are keying in on a single detail of your area descriptions and taking the fact that nothing else was described in the same level of detail as the obvious clue, when for you it was just a random scene element.

Lets try this room again, only this time with a few more details...

You open the door and stir up a thick layer of dust on the floor of this old study. An elegant many-drawered desk covered in bits of old decayed parchment sits in one corner of the room under the room's single, narrow window. A shaft of light catches on the brass corners of a thick tome resting in one corner of the desk, the red velvet cover standing out despite the dust that fills the room and covers everything. Next to the desk stands a tall and sturdy, if otherwise ordinary looking bookshelf that holds dozens of tomes in a variety of sizes; some are even bound in what looks like dragonscale. On a high shelf, the ends of several scroll cases peek out, having avoided the worst of the accumulating dust. Meanwhile the corner of a poster betrays its presence, otherwise almost completely hidden behind the room's furniture.

Now which object is worth further investigation?

Gosh, take your pick. The desk has a bunch of drawers, the red velvet book might be important, who knows what those scrolls have on them, and what's up with that poster?

Each of your players will pick up on something different and scatter. It doesn't matter that the only important bit of plot is the book. Maybe one of the scrolls has a useful spell on it, and maybe the poster came off its mountings and fell down and isn't important at all. By giving the players several options, they'll naturally split up and investigate different things and someone will find the clue you wanted them to find. Anything else you drop, as a reward for following up on the red-herrings, is a bonus and makes everyone feel like they contributed. Who knows when that +1 throwing dagger that they found, lost under a bed, will come in handy (and gosh, aren't I going to look under the bed more often)?

When they are repetitively trying something like banging on the door, just ask them how long they plan on continuing to try this. Then tell them, "After knocking for [half an hour], there's still no answer".

For something like spending ages trying to roll a natural 20 or whatever it takes to get a grappling hook into the hole, just tell them "You continue to try for half an hour, and you eventually get it hooked on." then when they climb up, they find nothing interesting and it's clear that this is not the way into the castle.