The courthouse was first established in 1762, on land donated by Dr. Thomas Walker. The grounds of the courthouse once held a whipping post, pillory and stocks, as was typical of the time.
In addition to serving as the courthouse and sole place to vote in Albemarle County, religious services for Episcopal, Presbyterian, Methodist and Baptist worshippers were held in this space, which Jefferson called "The Common Temple".
Jefferson, Madison and Monroe were frequent visitors here, especially after Jefferson retired in 1809. In 1859 a front addition was constructed in the Gothic Revival style, with octagonal stair towers flanking the entrance.