I was thinking along the lines of just DR for creatures but that might work too.
Normal PC shouldn't outright destroy a castle . Castles should have to take several hits before being destroyed .