Let me introduce you to Dale. Dale tried a bunch of different approaches. This is his story.
First, Dale increased the AC of the monsters in the battle. This made it harder for the PCs to hit them (duh). It did take longer for the PCs to kill the monsters in the battle, which increased the chances that the monsters would take them down, but the players hated it.
After a few minutes’ thought, Dale realized why: the PCs were hitting less. Misses aren’t fun. He was adding to the challenge by making the game less fun for the players.
Increase Hit Points
For the next battle, Dale went back to the same AC and increased the Hit Points of the monsters in the battle by about 20%. The monsters stayed alive longer, soaking up more damage before dying, providing more opportunities to wear down the PCs.
This worked out better, but the players still weren’t satisfied. It wasn’t any more fun. With a few more minutes’ thought, Dale realized that this approach simply prolongs the fight. Combat in D&D usually consumes the most time out of all the activities in the game. In other words, turning a quick fight into a slog increases the chances that the PCs will go down, but it’s just the same fight prolonged. While a valid strategy, it’s not the best or most interesting way to challenge the players.
Next, Dale went back to standard monsters, but increased the damage that they did.
Now that added some excitement! The first orc packed a wallop, and the players reacted. A little fear showed in their faces. They sat up and began strategizing about more effective ways to approach the fight.
Dale grinned. He’d found his solution!
Then, in round two, two orcs each took down PCs with lucky hits. Dale backpedaled, secretly fumbling a few rolls and scaling back damage.
The danger of increasing damage lies in building monsters that wipe the party quickly. If you’re going this route, increase the damage bonus, not the die. For example, if a monster normally does 1d8+4 damage, increase it to 1d8+8, not 2d8. That eliminates the chances of a swingy battle.
Dale’s players definitely remembered this fight more than most, but they weren’t excited about it. They remembered the fact that the dice rolled against them, not that the fight was fun. Players don’t like taking damage, so Dale decided he could do this occasionally, but not all the time.
Next, Dale went back to the drawing board, and decided that his problem was variety. His tactics up to this point were to increase numbers that prolonged the battle. What if he could make the actions in the battle itself more interesting?
So he added extra abilities to the monsters in his next fight. Some got +2 on attacks if ganging up on a PC, the leader could move an extra 30 feet as a bonus action, and the sneaky shaman got an extra spell.
Now the players sat up and took notice. Their usual tactics didn’t work. They had to think and strategize.
Dale grinned again. He figured now he’d found his solution.
Then things fell apart again. There was no reason for the monsters to not gang up on an enemy, so that +2 applied all the time, making them hit way more often than he thought. They took down the barbarian in 2 rounds, and moved on to the other PCs, dealing huge amounts of damage with nothing much the PCs could do to stop it.
Dale discovered that abilities can interact in subtle and surprising ways, giving monsters more of an advantage than you expect. So, while also a valid strategy, he vowed to be much more judicious about this.
Increase the Number of Monsters
Dale began to despair. Would he ever find a solution?
However, Dale was a good DM. He kept searching.
For his next fight, he increased the number of monsters from 4 to 6. The players took notice. They adjusted their strategy. The monsters could do a lot more, attacking more targets, holding some monsters in reserve, sending one or two around towards the ranged PCs in the back.
This was it.
Since you already know how to run a given monster, it’s easy for DMs to add one or two more.
Why is this effective? Every extra monster gets an extra attack (or two!) every round. This can significantly increase the damage output of the enemies.
However, as the DM, you control all these creatures during combat. You can decide that some of them stay back as reserves, or some of them go for the spellcaster. Some of them might initially go for the spellcaster, then retreat if things get hard. More monsters give you flexibility without adding complexity.
Now Dale had his solution (and so do you): If the PCs find fights cakewalks, add 1 more enemy than you think you’ll need to the next fight. If the PCs trounce those enemies, then in the next fight, add 2 more enemies than you think you’ll need. Continue increasing until you find the right challenge for your players.