Some presidential statistic I found:
Of the first 18 presidents, only 6 did not own slaves at some point or another.
Adams, Quincy Adams, Fillmore, Pierce, Buchanan, and Lincoln.
Buchanan is a bit of a fuzzy case, as he did have some indented servants, and that was similar to slavery.
Also, yes, Ulysses S. Grant did, in fact own some slaves at some point, though it was (obviously) before the war and it was through marriage. Probably doesn't get brought up much, what with him being the Union General and such.
Here's a list of how many slaves each slaveowning president probably had:
- GEORGE WASHINGTON (between 250-350 slaves)
- THOMAS JEFFERSON (about 200)
- JAMES MADISON (more than 100)
- JAMES MONROE (about 75)
- ANDREW JACKSON (fewer than 200)
- Martin Van Buren (one)
- William Henry Harrison (eleven)
- JOHN TYLER (about 70)
- JAMES POLK (about 25)
- ZACHARY TAYLOR (fewer than 150)
- Andrew Johnson (probably eight)
- Ulysses S. Grant (probably five)