The town was visited by Lafayette on his farewell tour of the U.S. in 1824 and by President Andrew Jackson numerous times. After the decline of the American grain trade in the 1820s, the rise of railroads, and increasing racial segregation, the turnpike and the town, whose inhabitants and artisans had included many free African-Americans, became less successful in the years leading up to the Civil War.

On October 19, 1863 Union cavalry under Maj. Gen. J. Kilpatrick pursued Maj. Gen. J.E.B. Stuart's cavalry along the Warrenton Turnpike but were lured into an ambush near Buckland Mills. The Federal troopers were scattered and chased five miles in an affair that came to be known as the Battle of Buckland Mills or "Buckland Races." The battle was among the Confederate cavalry's last victories of the Civil War.