Joseph and his wife Esther are recorded in the Deerfield, Massachusetts church records as having lost a 13-month-old child in 1808. Also, he was probably the father of a 5-year-old child that died in 1807. It is likely that Joseph's surname was Salisbury, and it appears that the couple had six children: Eunice (1797-1805), Simon (1801-1824), the 5-year-old (1802-1807), Schuyler (1805-1832), the 13-month-old (1806-1808), and Calvin (1807-1843).
A "Joseph Negro" appears in numerous entries for 1808 in Orlando Ware's day book for his store in Deerfield, Massachusetts. He was indebted for items and services, among which were ferriage, use of a horse to go to nearby Montague, corn, rye, rum, potatoes, a turkey, a load of wood, beef, a loaf of bread, and having fabric cut for a pair of pantaloons. Two White men, Lemuel Barnard and Henry Bardwell, bought fabric, thread, a whetstone, and a skein of yarn for Joseph and delivered the items to him. Two other White men, Solomon and Thomas Williams (who were distantly related,) were in debt to Ware for borrowing cash to pay Joseph. He paid his debts by digging a ditch, cradling (harvesting) grain, chopping wood, providing and axe, and general farm labor.
There is an 1809 entry in Ware's day book that refers to "old Joe". Esther was 48 years old when she had their last child, Calvin. If old Joe and Joseph are one in the same, he might have been substantially older than Esther.