Buidlings can be downgraded as well as upgraded. Downgrading reduces the amount of resources the building requires. Ore used to upgrade becomes waste on the downgrade. It does not return the resources otherwise.  There is also a cost in happiness equal to the amount of food the upgrade cost.