The Whittlesey House was the original residence for Mr. Thomas Tucker Whittlesey, and was built from 1853-1855, taking nearly two years to complete construction. The Whittlesey House, was considered a mansion in its time with 19 rooms, and was constructed from Milwaukee Bricks, giving the house its signature yellow color.

T.T. Whittlesey was a prominent Wisconsin State Senator, member of the U.S. Congress, Chairman of the Town of Middleton, probate judge and attorney, and also a farmer. He founded and platted the Pheasant Branch community then north of Middleton. Whittlesey is locally recognized for his dedication to his country, his land, and various agricultural pursuits. Present day University Avenue was originally named Whittlesey Street, in honor of Mr. T.T. Whittlesey.