We have events already where villagers "fight" monsters like the fishermen. Rather than an event, why not a settlement goal (how the plan for a settlement doing a Seal is to close the seal)? So you and an enemy spawn in glades opposite the active map and compete for some goal. The fishermen are collecting resources and cutting through meadows to get to you, and you are simply trying to gather resources to equip your villagers to fend off the fishermen. That would add a whole new suspense factor.