Ludlow Castle is sited on the western edge of the town, and to its west the land falls away towards the River Teme. It consists of an approximately square enclosure surrounded by a curtain wall, and in the NW corner of this is the Inner Bailey, surrounded on its south and east by the castle ditch, and inside this the Inner Bailey wall. The oldest building on the site is the massive gatehouse keep, or Great Tower. This dates in its original form from the late 11thc., but has been considerably modified. In the early 12thc. the gatehouse was increased in height to four storeys, with a two-storey hall on the first and second floors. In the late 12thc. the entrance was blocked, and a new arch cut through the inner bailey wall immediately E of the keep. Finally, in the 15thc. it was reduced in size, the N wall was rebuilt and floors were inserted in the hall to create new apartments. Some Romanesque windows survive in the keep, as does the original entrance passage (now blocked) with blind arcading on its side walls and the remains of a doorway in the E wall. The main castle buildings form the N range of the Inner Bailey, and consist of the late-13thc. Great Hall with a solar wing to the west, both begun by Geoffrey de Geneville (d. 1314) and his son Peter. The western solar was completed by Roger Mortimer, who took possession in 1308, and he also built a second solar block east of the hall, with a garderobe tower in the outer wall. The kitchen is detached from the Great Hall, standing in the Inner Bailey with its doorway facing that of the hall. No doubt a wooden passageway linked the two. Returning to the N range, to the east of Roger Mortimer's solar is a set of lodgings built in the 16thc., usually called the Tudor block. The chapel, with a circular nave, stands to the E of the Inner Bailey and is the subject of a separate report. Finally, E of the present Inner Bailey are the Judge's lodgings, completed in 1581.

From 1066 the manor of Stanton, in which Ludlow Castle, was situated, was held by the Walter de Lacy as tenant of Roger Montgomery. Although some of the fabric dates from the late 11thc., there is no written reference to the castle before 1138. The castle was held by the de Lacys until the death of Walter de Lacy II in 1241, when his lands were divided between his two surviving heirs; his granddaughters Margery and Maud. Ludlow Castle passed to Maud, and through her to her second husband Geoffrey de Geneville. They had a son, Peter and he had a daughter, Joanna, but Peter died before Geoffrey so that on his death in 1314 the estate passed to Joanna and her husband, the infamous Roger Mortimer. The castle remained in the Mortimer family until the death of Edmund Mortimer in 1425, when it passed to his sister Anne, married to Richard, earl of Cambridge, and thence to their son Richard Duke of York, ending up in royal hands when his son Edward came to the throne as Edward IV. It came to the Earls of Powis, the present owners, in 1811 when Edward Clive, the second earl, bought it from the Crown.

The gatehouse dates from the end of the 11thc. This kind of gatehouse keep is unusual but not unique, being found also at Richmond Castle (Yorkshire) and Rougemont Castle, Exeter (Devon). It was raised to a four-storey keep in the early 12thc. The Gatehouse was blocked at the end of the 12thc. and (according to Lloyd) was turned into a prison. The similar base profiles of the window and blind arcading confirm that these two belonged to the same campaign.