This is somewhat of a minor detail that might add a lot. Currently you click once to open the worker choices on a building, and then click again to pick one. A better solution might be (and it would need testing) to open the selection choices on click and make the choice by releasing click on the choice you want to add.
Saves a click, increases smoothness of the UI.
I found out today you can click on the workers on the left hand side then click super fast on all the buildings, it is much faster you don't even need to have the building panel open.