The Spencer Residence Hall, designed by W. C. Holleyman of Greensboro, North Carolina, consists of two connecting buildings: North Spencer Residence Hall, which opened in 1904, and South Spencer Residence Hall, which opened in 1907.
Spencer was constructed after the Brick Dormitory fire of 1904 destroyed one of the two residence halls on campus. Within three days of the Brick Dormitory fire, the state legislature promised $80,000 for the construction of a new dormitory and president Charles Duncan McIver and the faculty began planning its features. The result of this was a two-story, very long building (492 feet) divided into fireproof sections. The new building would also house a dining hall. Construction began in April 1904 along College Avenue, and, in October, the Spencer Residence Hall was completed.
The building was named in honor of Cornelia Phillips Spencer, a North Carolina poet, social historian, and journalist. In 1949 until 1953, the campus post office was located in the basement of South Spencer Residence Hall.