However, there is one other thing about demolishing buildings that will affect happiness, that is food. You get happiness every time you build something just by the nature of spending food. So when you build a building, the food cost is converted into happiness (provided you have 4 types of food). To ensure that nobody is able to exploit this by building and demolishing buildings, the food spent to construct the building will be subtracted from your happiness when you demolish it.