A lot of things contributed to that. Not the least of which is the difference between what we'd consider a day off and what they'd consider a day off. Not to mention how they paid taxes and what was actually required of the medieval peasant.

Taxes could be paid in labor or produce. The guys doing the manual labor building a castle were likely to be paying taxes. They did that for up to a third of the year. The rest of the year was theirs to do with as they pleased, and the majority of that time would have been spent growing, gathering, hunting, or maintaining. Guild artisans had the closest thing to jobs that we'd think of them. Coopers made barrels, ropers roped. You had masons and blacksmiths and carpenters sure. Most people were growing and raising food, and maintaining their home. A day off was likely spent doing those things. They had so many partially because that time was needed intermittently.

They worked harder than we do. Every part of their life was harder, required more energy, and took more time.

Taking a day off to relax would have been exceedingly rare and probably maddeningly boring. Though they did party hard.