This is 'conventional wisdom', but like a lot of thermal building advice, I suspect it may be crap.
Castles, which have massive thermal inertia in their stone or concrete walls, have notoriously cold interiors that are very difficult to heat. On the other hand, cave homes at Coober Pedy stay...