The keep of Lancaster Castle is a four-storey tower, which is 20 m high (albeit the upper storey may have been heightened or added at a later date) and has shallow pilaster-like buttresses at the corners and at the centre of each side. The outer walls are approximately 3 m thick. There is a spine wall running E-W which divides the keep internally.

The original main entrance of the keep was via a long set of stone stairs to the 1st floor on the S side, still extant until the 18thc. This elevation is now hidden by the Debtors' Wing of 1796. The W side of the keep is also hidden by the building of the new Shire Hall and Crown Court, begun after 1796. The 1796 prison for male felons is built on the N side of the keep. These additions were all designed by Thomas Harrison, albeit the Shire Hall and Crown Court were finished by Joseph Gandy.

Adrian's (also referred to as Emperor Hadrian's) tower, in the S-W corner of the Castle, has 13th-c style masonry internally, but was entirely refaced externally at the end of the 18thc.

The imposing gatehouse was built by Henry IV after 1399. It has been suggested that the lower levels of the gatehouse contain earlier work from the early 13thc. However, nothing relevant in terms of Romanesque sculpture was observed. The same is true of the Well Tower to the E of the gatehouse, which was largely built in the 15thc. Some structural timbers were dated by Oxford Archaeology North using dendrochronology to a primary phase after 1265, and later refurbishment or building to the late 14thc or early 15thc. Any evidence of earlier building from the 13thc in the lower levels did not include any Romanesque sculpture.

Lancaster Castle was established in 1093 when Roger of Poitou was granted land by King William of Rufus following the capture of Calisle from Scottish forces. Roger placed his military headquarters in Lancaster near the site of the old Roman fort on the hill overlooking the River Lune, and the main road to Scotland at that time, running along the Cumbrian coast. Previous suggestions that it was Roger who built the keep are no longer the consensus view.

After William of Rufus' death, Roger supported Robert Duke of Normandy as the next king and so had his lands confiscated from him by Henry I in 1102. During the Anarchy, after Henry I's death, Stephen of Blois, crowned king in 1135, allowed King David of Scotland to occupy the Honour of Lancaster in 1141. By 1164, the Honour was brought back under the direct control of the king, by then Henry II. Until 1182, the County of Lancaster was administered together with Northumbria. The building of the keep is currently dated to the mid-12thc based on size and style, but no actual records of its building or who paid for it are known to survive.

Royal accounts show that King John spent more than £630 between 1209 and 1211 on building works at the Castle. He is thought to have built Adrian's tower in the S-W corner of the Castle.

After the Civil War and Restoration of Charles II, the N half of the keep was a roofless shell but the S half stayed in use as the Shire Hall until the building of the new Crown Court and Shire Hall at the end of the 18thc, completed at the beginning of the 19thc. The predecessor to the 1816 County Lunatic Asylum was beneath the old Shire Hall.