I noticed idle workers in a Homestead once there was nothing more to harvest during Clearance. There were products in the Homestead's storage to deliver. The workers should deliver the goods instead of staying idle.
Awesome idea, I would probably even take it one step further: idle workers should become builders (maybe make this feature optional for each building with a checkbox). It might work like this: if worker is idle and there is stuff to build they take a building resource, do a trip and after that they check if they are still idle in the building they're attached to, if so, they continue building, if not they come back to the building of origin.