After an 1853 fire destroyed part of Oswego's east side, abolitionist and philanthropist Gerrit Smith approached eight civic leaders with a unique proposal. Smith would give $25,000 (about $785,000 today) to construct a library, on two conditions:
- The library must be sited east of the Oswego River; and
- It must be open to all, regardless of race or economic circumstance.
Smith was, at the time, New York state's richest man and the owner of much of the Oswego waterfront. The eight leaders he approached accepted the offer and became the founding trustees of the Oswego Public Library. Most would serve until their deaths.
The building was designed and constructed in 1855–56. It exemplified the then-popular Norman Revival style with "castellated" parapets resembling battlements. The library opened in the spring of 1857.
The library followed through on Smith's requirement that it be open to all. Prominent among its early patrons were children of escaped slaves who had relocated to the area, most directly aided by Smith and his Oswego business agent John B. Edwards. One early African American user was George Franklin Grant, who became the second African American to graduate from Harvard's dental school, later joining its faculty. He pioneered a treatment for cleft palate and, unrelated, invented the golf tee.
Smith continued to support the library's operations, donating $1,000 in 1862 and another $4,000 in 1868.
Until the end of the twentieth century, the library was supported by the city and the Oswego school system. (Early on, the trustees were personally liable for building upkeep, which led to significant maintenance issues.) In 1999, the Oswego School District discontinued its support for the library. Voters overwhelmingly approved a referendum to establish the library as an independent civic institution, which it remains today.
The Oswego Public Library is the only main public library in the United States still occupying its original historic building. (A contemporary-style annex was added early in the twenty-first century, positioned so as not to be seen when one views Oswego's "Castle on the Hill" from the front.)