The castle at Devizes has a reset arch containing 12th century stones and another smaller one with reused stones. However, despite having been the site of a major Norman castle, these stones originated at St John’s church. They were part of the west façade which was replaced in 1863. In addition to these stones there are the famous wooden heads which may have originated from the castle but equally could have come from another major building in the town or the area. They have been suggested to be 12th century in date, but are probably more plausibly 13th century in date. In addition to the stones found at the castle, a number of buildings in the town contain reset fragments from the castle, such as at 1 St John's Court.

A motte and bailey castle with a wooden keep appears to have been built soon after 1066 by either Bishop Herman or Bishop Osmund. A fire in 1113 destroyed the castle which was soon rebuilt by Bishop Roger. Just before his downfall in 1139 he appears to have been rebuilding the castle again. Roger also built castles at Sherborne, Malmesbury, Old Sarum and perhaps Kidwelly, as well as the east end of the cathedral of Old Sarum. By 1121 Devizes Castle was sufficiently complete to hold an ordination there. In plan the new, stone castle appears to have followed the overall layout of the earlier wooden one. With Bishop Roger’s fall from grace, the castle passed to King Stephen but it was subsequently taken over by the Empress Maud, and then the future Henry II. This situation was ratified in 1157 and thereafter governors were installed to look after it.

During the reign of Henry III the hall of the castle was rebuilt. John Leland visited Devizes in 1545 and described the castle as follows:

‘There is a castell on the southe west side of the toune stately avaunced upon an high ground, defendyd partly by nature and partly with dykes, the yere [earth] whereof is cast up a slope, and that of a greate height to defence of the waulle.

This castle was made in henry the first dayes by one Roger Byshope of Salisbyrye, Chauncelar and Treasurer to the Kynge. Such a piece of castle worke so costly ans strongly was never afore nor sence set up by any Byshope of England. The kepe or dungeon of it, set upon an hille cast by hand, is a peace of worke of an incredible coste.

There appere in the gate of it 6 or 7 places for porte colacis, and muche goodly buyldying was in it. It is now in ruine, and parte of the front of the towres of the gate of the kepe and chapel in it were carried, full unprofitably, onto the buyldynge of Master Bainton’s place at Bromeham scant 3 myles of.

There remayne dyvers goodly towres yet in the utter walle of the Castle, but all goynge to ruine. The principall gate that ledithe in to the toune is yet of a great strengthe, and hathe places for 7 or 8 porte colices.’

By 1596 the castle was described as ‘utterly ruinated and decayed, and the walls fallen down for the most part’, yet it proved a formidable fortification during the civil war. Therefore, after its capture Parliament ruled that it should be slighted and by 1648 its demolition had taken place. The castle appears to have been used as a quarry by the townspeople and by the early 18th century two windmills had been established on the site. In the 1830s the site passed to Valentine Leach who c1840 built the round tower. After his death in 1842, it passed to Robert Valentine Leach who built the current building between 1860 and 1880. In 1863 the west façade of St John’s church was rebuilt, and some of the displaced Norman stones were reused to form ‘The Bishop’s Gate’ in the garden, named in honour of Bishop Roger.

Fragments found in houses in Devizes seem to originate from surface decoration, predominantly a shallow chevron pattern and an imbricated pattern, similar to the pattern found in the chancel of St John’s church. This suggests that the castle built by Bishop Roger was lavish, like the buildings he erected at Old Sarum and Sherborne.

The reset arch, the Bishop’s Gate, has a decorated shaft which is suggestive of patterns found at Ely and elsewhere in the east of England, rather than in Wiltshire. Bishop Roger’s nephew Nigel was the Bishop of Ely but it is unclear whether this was the route of the influence. Perhaps Old Sarum employed such shafts, which have not survived.