In the fireplace of the ground floor lounge and in the cellar there are a series of reused carved 12th century stones from Devizes Castle.

Devizes, a borough by prescription, lies almost exactly in the centre of the county. Deemed a hundred in itself in Richard I's reign, a part of Cannings hundred in 1280, and a part of Bishop's Rowborough hundred in 1316, it has since 1592 been claimed as a liberty within the hundred of Potterne and Cannings. Speed, however, marked it (1610) within Swanborough hundred and the boundary of that hundred, as he traced it, was considered to have some authority even in 1839.

Devizes is a distinguished example of a medieval town whose defences were integral with those of the castle abutting it. At an unknown date a bishop of Salisbury, perhaps Osmund, built a castle upon certain boundaries (divise), which gave the castle and adjacent town their name. The fortified area, as a document of 1149 shows, was carved out of the manor of Bishop's Cannings. Devizes castle is first mentioned in 1106, when Robert of Normandy was imprisoned in it.

A town grew up below the castle walls and by 1141 was called a 'borough'. To this in course of time town lands were added. Presumably it was the combination of castle, town, and town lands that formed the lordship of Devizes, or 'manor' as it is actually called on eight occasions between 1217 and 1248.

The fragments found in houses in Devizes seem to originate from the castle. They include a lot of surface decoration, mainly a shallow chevron pattern and an imbricated pattern, the latter being similar to the pattern found in the chancel of St John’s church. This suggests that the castle built by Bishop Roger was lavish, like the buildings he erected at Old Sarum and Sherborne.