Spencer was born in Hatfield, Massacusetts, to John and Nabby Wheaton Brown. In A History of Hatfield, Massachusetts (Daniel White Wells & Reuben Field Wells, 1910, p. 284) "Spence" is described as one of two "colored boys" who lived with Mr. John Allis while he attended a brick schoolhouse in Hatfield.
Spencer married Elizabeth Cecilia Wilson on June 30, 1818, in Enfield, Connecticut. They had a daughter, Sally Ann, born in April of 1819. The family appears in the 1820 federal census for Enfield with Spencer and Cecilia in the columns for "free colored" males and females between the ages of 26 and 44, and one free colored female under the age of 14.
By the 1850 census, Spencer and "Elizabeth" were living in Springfield, Massachusetts. They are both listed as being Black, and no mention is made of Sally Ann, as she was probably married or living elsewhere by then. Spencer gave his occupation as laborer.
Cecilia died in 1863, and on September 23, 1866, Spencer married 78-year-old Melita Paine in Amherst, Massachusetts. He was 80 years old.
The next census where Spencer appears is the one taken in 1870 for Springfield. His age is given as 84, he is noted as being a "pauper", and is living with 10 other people between the ages of five and 70. None appear to be related to him, so he might have been living in a poorhouse.
Spencer Brown died in Springfield, Massachusetts on Nov. 8, 1875.