Southwold was mentioned in the Domesday Book as an important fishing port, and it received its town charter from Henry VII in 1489. Over the following centuries a shingle bar built up across the harbour mouth, preventing the town from becoming a major port. Southwold was the home of a number of Puritan emigrants to the Massachusetts Bay Colony in the early seventeenth century. Richard Ibrook, born in Southwold and a former bailiff of the town, emigrated to Hingham, Massachusetts, along with Rev. Peter Hobart, son of Edmund Hobart of Hingham, Norfolk. Rev. Hobart was formerly an assistant vicar of Southwold's St. Edmunds Church after his graduation from Magdalene College, Cambridge. The immigrants to Hingham were led by Robert Peck, vicar of St. Andrews' Church in Hingham and a native of Beccles, Suffolk.

In 1659 a fire devastated most of the town and damaged St Edmunds Church, whose original structure dated from the 12th century. The fire created a number of open spaces within the town which were never rebuilt. Today these greens, and the restriction of expansion because of the surrounding marshes, have preserved its genteel appearance. On the green just above the beach, descriptively named Gun Hill, the six eighteen-pounder cannon commemorate the Battle of Sole Bay, fought in 1672 between English and French fleets on one side and the Dutch (under Michiel de Ruyter) on the other. The battle was bloody but indecisive and many bodies were washed ashore. Southwold Museum has a collection of mementos of the event. It has occasionally been held that the cannons were actually captured from the Scots at Culloden and given to the town by the Duke of Cumberland but they are much larger than those used in that campaign.