I spent a good deal of time in the keep looking for him as 'he's inside the building' isn't exactly helpful. 

Grimmord is sitting on the floor to the right of the door of the blacksmith's shop, if you enter via the front door. (Front door has two guards posted beside it, the back does not.) If you enter the rear he'll be on the left side near the front door.