He's immune to fear, so a priest would have no way to stop him beating on the squishy cloth.  You're a paladin in Plate armour with tonnes more armour than a priest, he does Physical damage which you can easily mitigate. 

Engage brain before posting please.