In 1870 the property was purchased by Robert Stringfellow Walker, who remodeled the house in 1884. It was here that Walker founded Woodberry Forest School in 1889, naming it after the Madison plantation. The house was renamed The Residence and became the headmaster's house. Walker hired a tutor to educate his six sons and neighboring children.
The first classes were taught in a room of The Residence. Additions made in 1884 changed the architectural character of the house from Palladian to Victorian. Other renovations in 1948 created a large drawing room by eliminating partitions between three rooms, one of which was used as a back porch by the Madison family.