Alright I've got it...

The entrance on the map doesn't match the actual entrance, it's east of where it looks like you can get in from the map.

I would have noticed this earlier if I didn't pay attention to the quest saying "It's easy to miss" and to "head east" which takes you into mountains...

It's not easy to miss, there are guards standing outside ffs

I found it annoying as well.  It makes it look like the entrance is on the side you come from, so you try climbing and finding the right path while having to fight endless annoying creatures, but you do get some decent quests once you finally find the place.

Start at the TOP path of desolace follow the path and keep looking left,You should see a gap leading up,and maybe some guards guarding,Thats nijel's point.Dont forget to pick up FP