I'm planning a new campaign where the villain will force the party to do his bidding, starting from level 3.

The opening will be an encounter with the villain, where he will "ask" them to do not-so-nice things. A good-natured (or even neutral) predictably will refuse to do so, because it involves a massacre of innocent village, for instance.

I want to intimidate them into submission, simply by showing how great the difference in strength between them and this villain, but not kill them. This villain stat should stay the same across every encounter, but slowly become beatable later. Think of it like a Darth Vader, I guess?

I will be using 5e, and this is part where I believe the problem is: the bounded accuracy means that you can't really show how great the power difference is. I mean, the players having a +4/5 attack bonus vs the villain having +7/8 will still allow them to think "well, we are 5 people, we can still try". Ramping up the AC has its limits.

At first, I was planning to "reveal" the attack bonus, "okay, so that's a 7 plus 8. I believe that's a hit?" to freak them out, but I have doubt that it will deliver the message. A very damaging spell/ability may freak them out alright, but at level 3, I fear killing them out outright (I almost always roll in open and never fudge an open roll)

Is there a way for me to accomplish this without railroading them into the choice? Or is 5e the wrong system to do this?

You don't

I usually hate this type of answer but unfortunately this sort of thing is rarely done well at tables and should be avoided. Particularly in a first session this will be considered railroading.

No matter how strong the initimidation is the players will still think they have a chance to win and are extremely unlikely to give in. This results in three options:

1. TPK, the players will refuse to do what is asked and would rather fight to the death.


2. Your players micraculously win and destroy your carefully planned campaign.


3. The PCs run away. This might be a welcome outcome for you but it is still unlikely they will do what you want them too.

... the villain will force the party to do his bidding ...

This is almost a definition for railroading. You shouldn't be forcing your party to do anything. You can present the villian as evil and describe his actions but you should never dictate how your PCs respond to it. For experience it is extremely unlikely this scenario will go the way you want without railroading.

Some Alternatives

Straight up forcing your players to do something through intimidation is a bad idea. But there are other ways that you could achieve a similar result.

• 
  Blackmail them. Hold a town to ransom, capture the PCs little sister, something suitably evil and compel the PCs that way. This may still not get the result you want but will portray the villain as suitably evil.


• 
  Trick them. Have the villain pose as the quest giver, give them vague or inaccurate reasons for what he wants done. Maybe the people they are sent to kill "killed my son", which is of course a lie. A villain lying to the party is totally ok and something I have done in the past. This is the most likely approach to get the players to commit terrible acts, make them think they weren't terrible while they were doing them. Warning some players may not takes this well. You will need to judge based on your own group of players.


• 
  Run it as a cutscene. Before the game starts you describe the situation, the players are in the midst of the preparation to commit this terrible act. Describe how the powerful enemy inspired terror in them and they choose to do his bidding. Most importantly they regain control of their character before they commit the act. They can then choose to not do it if they want to. Never force a PC to do something they don't want to, if you do you are denying them their agency.


• 
  Have it part of the campaign setting. Estabilish the dominance of the villain as part of your campaign setting, the PCs are already living under the thumb of the bad guy. Have it part of the characters backstory how they came to be under his control. _{Thanks to Nitsua60 in chat for the idea}
  

Session 0 and Campaign Expectation

It is possible that all the advice I have given is totally inaccurate, your players may enjoy this kind of play. If so they wouldn't be like any of the players I have ever met but that doesn't mean it is wrong.

If this is something you believe your players will be onboard with you need to set it up in the campaign expectation during session 0. Explicitly explain that you would like to be able to force the characters to do thing through intimidation. If they aren't onboard then you shouldn't do it. I expect this will be the case, if I'm wrong however, I wish you luck.

Fish with a worm and a hook rather than with a stick of dynamite

Is there a way for me to accomplish this without railroading them into
the choice?

No. What you described is a railroad. If your players are OK with a railroad game, then press on. If not, stop now and change your approach. Put the hard choices in front of them and see how they respond. Then riff off of their response.

Or is 5e the wrong system to do this?

A lot of systems are the wrong system for this.

I would advise you that you can try to influence them to go in a particular direction, but if they resist, you need to be flexible enough to see where those decisions by the players takes the game. You will most likely find some great new things happening based on how they try to get out of this mess / difficult choice.

(This approach served me well in numerous games and in numerous campaigns as a DM).

Let their efforts to evade or overcome this compulsion become the new "plot" such as it is.

Fun all around.

Is the plot or the story more important to you, as the DM?

I think you might want to answer that question before the next session. The story that comes from your game, your campaign, is what the players did when confronted with hard choices, dangers, challenges, etc, and also the consequences of their decisions.

Have you and your players discussed the chance that this campaign may end in grand and glorious failure? In other words, is the story about what they did, even if in the end their greatest efforts weren't enough? (Feanor's story was like that).

That's a story too. Some of the most memorable D&D campaigns I was in had catastrophe as the end play. The trick to making this successful and fun is in making this about the choices the players make, and maybe a few unlucky rolls if the dice, rather than about your domination of their characters. You own all of the cards in the deck: what's the point in dealing them a losing hand?

Mike Mearls has this to say about how fickle the dice(Fate) can be:

the dice will be cruel to you, but you will soldier on (PHB, p. 4)