At least eight of our 44 presidents were slave owners. Note that John Adams (2) is missing. Adams is actually famous for being against slavery when it was a mainstay institution of the American economy.

1. Washington: 316 slaves

3. Jefferson: 187 slaves

4. Madison: 106 slaves

7. Jackson: 160 slaves

10. Tyler: 70 slaves

11. Polk: unspecified number of slaves

12. Taylor: 100 slaves

18. Grant: at least 1 slave

source: U.S. Slave

As for some of the other early presidents, I speculate that they were simply not rich enough to be able to afford slaves.