There was no settlement at what is now downtown Middleton until 1856 when the Milwaukee & Mississippi Railroad selected that location for its new rail line and depot. Development then shifted from the Pheasant Branch area, which had been a thriving settlement. Businesses spurred by the depot included three hotels, stockyards and warehouses. Until the 1870s, large quantities of wheat were shipped to market from the Middleton station. The present building, which was built in 1895, is the third depot – the first two were lost to fires. The depot continued as a passenger station for the Chicago, Milwaukee, St. Paul Railroad until 1960 and as a freight depot until 1975. In that year, Middleton purchased the building for a senior citizens center. The depot was listed on the National Register of Historic Places in 1995.
Today, the depot is home to the Middleton Visitors Center and Visit Middleton offices and also includes a curated display of railroad photographs and items by the local Middleton Historical Society and just a short walk from the Rowley House Museum. The depot is open to the public from 8:30 a.m. to 4:30 p.m., Monday-Friday.