I am making an open world in Unity 5. But the engine is not important in this thread. What I want to know is how to make a good open world that looks good from every angle.

So far I have a few mountains, a lake, some trees and alot of grass. It's your typical open world rpg with spells, weapons, monsters, and quests. The map isn't on the scale of Just Cause 2 or Oblivion, but it is pretty big.

When I make a mountainous area, it feels too cluttered. When I make an open grassy field with sloping hills, it feels too empty. When I try a bit of both it feels too inconsistent.

What is a good way to build up a fantastical, yet believable world? Is there an order or style?

This might help, though admittedly I didn't browse much yet.

http://www.worldofleveldesign.com/

I imagine a lot of your inspiration should also come from games you've played. I'm thinking along the lines of Skyrim. There's a lot to learn there by just looking around.

Could you share images so we have a better understanding on what you call "cluttered" or "empty"?

Would certainly help spotting what could be the problem in your case.

You'll just have to observe landscapes and try to duplicate what interests you.

Picking the brains of landscape architects seems to very helpful, this was done for The Witness. Luis Antonio, one of the artists for The Witness has done a GDC talk and has an art dump where he talks about their approaches to creating the world which you might be interested in.

So far I have a few mountains, a lake, some trees and alot of grass. It's your typical open world rpg

Yep. Certainly sounds like a typical RPG world.

What's influencing the world's design? I mean, apart from every generic fantasy game you've played.

How's the lore, cultures/civilizations, and the plot shaping the world while at the same time being shaped around the world?

Be careful that you're not just taking all the fantasy games and books you've consumed, and distilling it to their common elements, and just using those common elements, resulting in a cliched and bland world, forcing you to try and spraypaint on top of it some "uniqueness" back in, after having already removed what was unique from those other worlds?

Or let me phrase this another way: How is your world different and unique from, say, Lord of the Rings or [insert your favs here]? I'm not talking geometry. I'm not talking characters. I'm not talking cliched plot or blatantly-ripped-from-mythology-or-real-life cultures.

What (hopefully dozen or more things) makes your world different? Make a list for yourself.

Then, use that list to shape your world's geometry, lore, cultures, people, social laws, natural laws, and mechanics, and so on, while also using all that to recursively feed back into the world's uniqueness again. Your design should influence your design should influence your design.

The map isn't on the scale of Just Cause 2 or Oblivion, but it is pretty big.

Why is it big? Is it big merely to be big, or do you actually have content to fill it?
Are you merely creating a large empty world and then trying to fill it with content, sprinkling it around, making the content spread out and watered down?

Personally, I suggest you shape the world around your content at the same time as shaping your content around your world, letting them recursively influence each other, along with being recursively influenced by, and recursively influencing, the story, lore, music, art, enemies, and so on.

Otherwise, what you get is basically empty geometry with annoying enemies bothering you while you travel, with a poisson-distribution of points of interest with too much "nothing" between, regardless of how beautiful that nothing may look.

I suggest you play Minerva: Metastasis (an single-player FPS mod of Half-life 2 for the PC), and King's Field: The Ancient City (a PS2 first-person action-RPG). I guess a more modern version would be Dark Souls 2 (from the same people who made King's Field), though I haven't personally played that.

These games do an excellent job of hyper-compressing their world, to fit as much content into as small a world as possible. That goes counter to some people's goal of being able to brag about "My world is 50 square miles in area! It takes you two hours to walk from one end to the other!! It's so empty and bland walking though it though, that I had to add quick travel to compensate!!! Isn't that amazing!!!!".

Even if you don't hyper-compress your world as compactly as they did, you can still learn better area design from them, and if you practice hyper-area-compression like this, you could then apply the lessons learned to the level of area-compression that you think best fits what you're going for.