Your comments

I feel like farming and castle-building are very natural expansions this game could have, but it will impact the gameplay

if it were balanced so that some rabbits were replaced with cows, and the cows had to be milked by a new job, say a farmer, and that would only generate 1 food with a cooldown period, and the cows once tamed follow your totem, that might be something the game really needs, a food supply that you could take with you.