Stop enemy spawning inside your castle

It's really annoying when you've built a pretty large castle with lots of resources inside, and enemy tribes just spawn inside and steal from the storages. Even if you destroy them you cannot recover your resources, so this is making the whole fortification idea a bit worthless. Can you fix this?