Definitly watch out for invading armies and mercenary bands.... (and your own king aswell), they might very well like to capture that keep.
Whatever you say, it just isn't a mere inn, its HUGE!.. I'd like to see examples of inns back in the days being that large, I have a feeling they'd be non-existing or housed inside actual (ex-)keeps

You sir, are insane, have a cookie.