If you're going to divide people in "100% free people" & "slaves"" then you're right, and then we had slavery at least up to ~1991(Soviet-Union).
For a more nuanced view; serfs belonged to the land, and it wasn't as much that they weren't allowed to leave, as it was that other lords weren't allowed to poach them for their own. They had rights, which slowly increased, and the abilitie to rebel(albeit not always successfully)
Historically, the medieval times(or dark ages) were a time of upheaval and survival, the old (Roman) system had fallen and anyone not able to protect his own was bound to go under, the new system was made on-the-spot everywhere, based on what was there previously(both tribalism & civilization, but also on both slavery & coöperation) the rights of individuals were details that had little priority for rulers, but the basics(like rights & obligations come together) were understood (at least by the more competent rulers)
@Externettrade without the medium of money is called bartering.
I suspect most craftsmen working in a castle were just paid in food & shelter with some luxuries(like more clothes)