Dunwich is on the E coast of Suffolk between Swold and Aldeburgh. It was a Roman site and an important Anglo-Saxon port town, and from the late 11thc. until the middle of the 14thc. it was a nationally important seaport. By 1225 it was a mile from N to S, with an area similar to London's at that date, and had seven parishes with 19 churches and chapels, Franciscan and Dominican houses and two hospitals, including this one. By 1242 Dunwich was the largest port in Suffolk, but this changed dramatically after the great storms of 1287 and especially 1328. The latter completely silted up the harbour mouth and flooded the quays, effectively ruining the town as a port, although some fishing survived. The church of St Leonard disappeared into the sea and only 12 houses in that parish were left standing. The parishes of St Martin and St Nicholas lost 225 out of 300 houses between them. Many of the inhabitants left in search of a livelihood elsewhere and this, in combination with another great storm in 1347, when another 400 houses were lost to the sea, further reduced the size of the town. In another storm twenty years later, the churches of St Martin and St Nicholas were lost. The sea continued to erode the coastline, reaching the market place in 1540. The inhabitants stripped the churches and other buildings of their lead roofs and valuables as the sea reached them, and by 1717 St John's, the church of the Knights Templar, the market cross, St Peter's, the Blackfriars monastery and the town gaol were all lost in this way. The last of the medieval churches to go was All Saints. The parish boundaries had been redrawn to bring what remained of Dunwich into its parish, but there were not enough parishioners left to support it, and its last rector left in 1755. It remained in occasional use until the new church of St James was built alongside the Hospital in 1832, after which it was abandoned. It went into the sea between 1904 and 1919, and its last buttress was moved into St James's churchyard before the sea could claim it.

Dunwich is now a small village with houses, a 19thc. church and a public house on a triangle of streets at the edge of the sea. Inland is Dunwich Heath, to the S, and Dunwich Forest and the road to Westleton on the W. The Hospital of St James was built for lepers at the W edge of the town, outside the 12thc. town wall and the earlier Pales Dyke. Its position outside the town and far from the sea saved its chapel from the fate shared by all of Dunwich's other medieval buildings except the Franciscan Friary (Greyfriars) that now stands in ruins on the clifftop. This was founded in 1277 but moved further inland in 1289.

St James's Hospital chapel is now the only 12thc. structure in the village. It stands in the churchyard of St James's, a parish church of 1832 that was built alongside it. The 12thc. chapel is now ruinous and roofless. The E end survives, consisting of a rectangular chancel and an apse, and between them the jambs only of the apse arch. The apse had three windows originally, of which only the N retains its ashlar work. At ground level are the remains of an internal blind arcade, arches only and a few capitals, originally of 12 bays. Ashlar work survives on five bays and part of a sixth on the N, and two bays and part of a third on the S. The central bays are gone, along with the window above them, and all the losses have been consolidated with distinctive flintwork. Only the rubble cores of the apse arch jambs survive. The chancel had blind arcading along the N and S walls too, but of this there are only slight remains at the W end of the N wall and the E end of the S wall. The chancel has one surviving window on the N wall, with cut stonework still in place inside and out.

There is evidence of Roman occupation, in the form of a road from Bury St Edmunds, and finds of coins, pottery and weapons. It became the seat of the East Anglian diocese with the appointment of Felix as bishop in the early 7thc. Felix died in 647 and in 670 the see was split between Dunwich and North Elmham. When the see was reunited by Bishop Wilfrid in 870, North Elmham was chosen as the seat. Eadric of Laxfield held Dunwich as a manor before the Conquest. In his time there was 1 carucate of ploughland but by 1086, when it had passed to Robert Malet, there was only one, the other having been carried off by the sea. The town was large, with 236 burgesses, 178 poor men, 2 bordars and 24 Frenchmen (with 40 acres of land) in 1086. There was one church before the Conquest and three by 1086. The manor was worth £50 and gave 60,000 herrings as a gift. There were also three holdings of an acre each from Robert Malet. After the fall of Malet in 1100, Dunwich became a royal town, and remained faithful to the king when it was besieged by rebel barons led by Hugh Bigod in 1173. By 1200, when Dunwich received its charter of privileges from King John, there was a merchant gild in existence. Concerning the Hospital of St James, the earliest documentary account of any detail is a charter of Walter de Riboff granting land at Brandeston and elsewhere, the tithes of his mills, various gifts of wheat, bread and ale and a pension to the chaplain, to the church of St James and the house of lepers at Dunwich, and to Hubert the chaplain and his successors. This dates from c.1200 and the gift was made for the good of the souls of Henry de Cressie and Walter himself. In 1631 the church was described by Weever as great, large and fair, but greatly decayed owing to the actions of covetous masters who sold off its possessions. By 1754 the only buildings remaining were one poor house, accommodating three or four indigent people, and the shells of the church and the chapel (Gardner).

The documentary history takes the chapel back to c.1200, but it is clearly older than this. The sculpture is too worn for a detailed analysis, but the combination of cushion capital variants, hollow chamfered imposts, shafted windows and arches with heavy angle rolls and face hollows indicate to the present author that it was unlikely to have been built before c.1130 or after c.1160.