the insignia will drop from anyone in Skull Rock except the pets. It can drop on the first guy in the entrance to the caves (where I got my first insignia btw). The only reason I've found to continue into the caves are for difficult and dangerous xp, and to get the burning eye from the named one in the end of the caverns.