The Wilton Free Public Library Association was incorporated in 1901, and was formerly housed in the Holmes Block on Main Street in downtown Wilton.

In 1915, Goodspeed Memorial Library was built using funds donated by brothers Frank O. and George F. Goodspeed in memory of their parents, Flavius and Angnes Ingalls Goodspeed. The building was designed by the Lewiston architect Harry S. Coombs and was constructed by the B.F. Dunning Company of Massachusetts at a cost of $10,000.

In the 1950s, the childrens room was moved to the basement to better serve the youngest library patrons in their own space. 

The library was placed on the National Register of Historic Places in 1989.

In 2008, major renovations upgraded the library to improve accessibility. A ground level entrance, an elevator, and an accessible bathroom were added. The renovations also included the addition of the mezzanine floor with a multipurpose room, storage, and space to house additional book stacks. This project expanded the footprint of the building by 20%.

In 2016, the installation of a perimeter fence was completed. This allows the library to utilize its multipurpose outdoor space for events and everyday use.