I believe the panning is caused by an encounter with enemies that stay alive after your encounter with them, causing your troops to follow the enemies until your troops or their troops die. I find that picking up the totem sets them to following your totem for a moment, and you can use this to play semi-normally, but the constant aggro to the enemies will cause your troops to lose more battles and harvest resources much less quickly.