A motte and bailey castle with a wooden keep appears to have been built soon after 1066 by either Bishop Herman or Bishop Osmund. A fire in 1113 destroyed the castle which was soon rebuilt by Bishop Roger. Just before his downfall in 1139 he appears to have been rebuilding the castle again. Roger also built castles at Sherborne, Malmesbury, Old Sarum and perhaps Kidwelly, as well as the east end of the cathedral of Old Sarum. By 1121 Devizes Castle was sufficiently complete to hold an ordination there. In plan the new, stone castle appears to have followed the overall layout of the earlier wooden one. With Bishop Roger’s fall from grace, the castle passed to King Stephen but it was subsequently taken over by the Empress Maud, and then the future Henry II. This situation was ratified in 1157 and thereafter governors were installed to look after it.
During the reign of Henry III the hall of the castle was rebuilt. John Leland visited Devizes in 1545 and described the castle as follows:
‘There is a castell on the southe west side of the toune stately avaunced upon an high ground, defendyd partly by nature and partly with dykes, the yere [earth] whereof is cast up a slope, and that of a greate height to defence of the waulle.
This castle was made in henry the first dayes by one Roger Byshope of Salisbyrye, Chauncelar and Treasurer to the Kynge. Such a piece of castle worke so costly ans strongly was never afore nor sence set up by any Byshope of England. The kepe or dungeon of it, set upon an hille cast by hand, is a peace of worke of an incredible coste.
There appere in the gate of it 6 or 7 places for porte colacis, and muche goodly buyldying was in it. It is now in ruine, and parte of the front of the towres of the gate of the kepe and chapel in it were carried, full unprofitably, onto the buyldynge of Master Bainton’s place at Bromeham scant 3 myles of.
There remayne dyvers goodly towres yet in the utter walle of the Castle, but all goynge to ruine. The principall gate that ledithe in to the toune is yet of a great strengthe, and hathe places for 7 or 8 porte colices.’
By 1596 the castle was described as ‘utterly ruinated and decayed, and the walls fallen down for the most part’, yet it proved a formidable fortification during the civil war. Therefore, after its capture Parliament ruled that it should be slighted and by 1648 its demolition had taken place. The castle appears to have been used as a quarry by the townspeople and by the early 18th century two windmills had been established on the site. In the 1830s the site passed to Valentine Leach who c1840 built the round tower. After his death in 1842, it passed to Robert Valentine Leach who built the current building between 1860 and 1880. In 1863 the west façade of St John’s church was rebuilt, and some of the displaced Norman stones were reused to form ‘The Bishop’s Gate’ in the garden, named in honour of Bishop Roger.