An important part of the early playstyle is to use the available villagers and mechanical parts to build camps, which you later scrap as your economy develops. With the increased building cost modifier (Prestige 5?) this is no longer a viable playstyle. While this certainly makes sense for other buildings, camps ought to stay an early game "wide-strategy" which later develop into a more complicated "tall-strategy"
As someone mantioned in the comments - the P5 modifier is already taken into accoutn when refunding goods for all buidlings. The only situation where you could lose resources is by playign next to the Corrosive Torrent map modifier.
Building refunds already take into account the full materials spent, even after the P5 modifier. There is no loss of Parts or other components when building and recycling a Camp or any other building.