Caesar, Sr. was the son of Romulus and Rose (Rozanna) of Shelburne, Massachusetts. He married Phillis, who was the daughter of a woman enslaved by the Reverend Roger Newton of Greenfield, Massachusetts. The couple moved to Shutesbury, Massachusetts, where their first child, Caesar Augustus, Jr. was born around 1800. By about 1802, the Fenimores were living in Leicester, Massachusetts. They had 11 more children. Four of them died by the age of two, six died between the ages of four and 11, and only Caesar Augustus, Jr. and his sister Louisa lived to adulthood. Caesar appears in the 1810 federal census for Leicester with nine people in the "All Other Free persons" column.