Edward Coles, Reluctant Slaveholder

A human inheritance

Coles’ father died in February 1808. When the family gathered for the funeral, Coles told them he intended to immediately manumit the enslaved people included in his inheritance.

The entire family objected to Coles’ plans. Besides being offended by his rejection of his father’s provision, they questioned whether he could afford to forfeit the value of his slaves and the free labor they provided.1 They also dreaded the social embarrassment of having a family member who broke with Virginian norms. Coles’ slaveholding neighbors worried that his actions would raise expectations of freedom and/or foster hostility among those remaining enslaved.

Coles considered these arguments and developed an alternate plan: to travel to the Northwest Territory with any of the people he inherited who were willing and able, and manumit them there.

As his share of the estate, Coles received 782 acres of land on the Rockfish River in Amherst County, Virginia. He also inherited a $500 debt and about twenty enslaved people.2 Coles began looking for a place to live where he could implement his emancipation plans. He put the Rockfish Plantation up for sale. But his career path suddenly changed, distracting him from his objective.

Working for President Madison

President James Madison

Portrait of President James Madison painted by John Vanderlyn in 1816. Courtesy of the White House Historical Association (White House Collection).

Edward Coles’ older brother Isaac had a position as President Madison’s personal secretary.3 When Isaac Coles resigned in December 1809, Madison invited his former secretary’s younger brother to take his place.

Coles accepted. He moved to Washington, D.C., and hired a manager to run Rockfish Plantation in his absence. Coles decided not to sell his land in the depressed market. Unable to pay off the inherited $500 debt, Coles struggled to avoid getting further in the hole. He remained a slaveholder.

Coles resigned as Madison’s secretary in the spring of 1815. Now 28 years old, he refocused on his plan to leave Virginia, free his slaves, and resettle them and himself in an antislavery state.

Coles traveled to the northwest frontier to scout out a suitable place. Ralph Crawford, an enslaved coachman in his early forties, accompanied Coles on the trip. They liked the fertile and inexpensive land in Illinois. Coles purchased some land in an attractive spot close to the relatively civilized atmosphere of St. Louis.

A request from the president again distracted Coles from his plans to move west. Madison asked Coles to act as his personal envoy to the Russian czar. It seems that the prosecution of a sex scandal involving the Russian consul in Philadelphia had threatened diplomatic relations between the two countries. Coles accomplished the mission and returned to the United States in the fall of 1817. That same year, Coles’ brother Walter purchased Rockfish Plantation. This gave Coles the freedom to start his new life in Illinois.


1Even if he could afford it, Coles couldn’t legally keep the freed workers on as paid laborers: Virginia’s Removal Act stipulated that all enslaved people emancipated after 1806 had to leave the state within a year or risk re-enslavement.

2The division of real property in the will included the enslaved people who worked the land, rather than designating specific individuals. Furthermore, lists of enslaved people did not always include small children. Coles’ plans to manumit his human inheritance remained delayed for several years: in the meantime, people died and were born. We do know the exact number of Illinoisans who eventually realized their freedom through Coles’ action, and their names. But we know very few details about the enslaved people Coles first inherited in 1808.

3Isaac Coles’ appointment may have had something to do with the fact that his (and Edward’s) father and First Lady Dolley Madison’s mother had a double-first-cousin relationship. In a typical first-cousin relationship, one of the parents of one cousin is the sibling of one of the parents of the other cousin. In other words, the children of siblings are first cousins. In a double-first-cousin relationship, the cousin’s other parents are also siblings. For example, when two sisters marry two brothers, their children are double-first-cousins; this was the case with double-first-cousins John Coles II and Dolley Madison’s mother, Mary.