This compact plantation house was built circa 1793 for William Madison, member of the Virginia House of Delegates for seven consecutive terms and brother of President James Madison.
In 1793 James Madison asked Thomas Jefferson to supply plans for a house for his brother. Jefferson, a close friend of the president, suggested a floor plan for a seven-room house in a geometric configuration that is a hallmark of Jefferson's residential designs.
James Madison later wrote to Jefferson saying that William had adopted the plans. No Jefferson drawings have been positively identified as the Madison design, but the correspondence authenticates the Jefferson connection. The original, unacademic two-column portico suggests, however, that Jefferson was not involved in the execution.
In 1870 the property was purchased by Robert Stringfellow Walker, who remodeled the house in 1884. It was here that Walker founded Woodberry Forest School in 1889, naming it after the Madison plantation. The house was renamed The Residence and became the headmaster's house. Walker hired a tutor to educate his six sons and neighboring children.
The first classes were taught in a room of The Residence. Additions made in 1884 changed the architectural character of the house from Palladian to Victorian. Other renovations in 1948 created a large drawing room by eliminating partitions between three rooms, one of which was used as a back porch by the Madison family.