The first census was done in 1790. The population of the U.S. was found to be 3,929,214 people. In 1776, at the time of the Declaration of Independence, historians estimate the population to have been about 2.5 million people. That is less than the population of San Diego County today.
It’s also interesting to compare population and population growth of the white versus African-American population, including the percentage of black slaves: