Whaddon was held by Edward Cild, a thegn of King Edward, in 1066, and by Walter Giffard in 1086. It was assessed at 10 hides, 5 of which were in demesne. The manor also included meadow for 10 ploughs and woodland for 100 pigs.
The manor was attached to the honour of Giffard, and reverted to the crown at the death without issue of Walter Giffard, 2nd Earl of Buckingham, the son of the Domesday holder in 1164. Before 1175 the king granted the tenancy to Richard de Humetis, Constable of Normandy. Richard died c.1179 and the manor (and the office of Constable) passed to his son William, but he forfeited Whaddon on account of his rebelliousness, and King John gave it to Peter de Stoke in or before 1205. This grant was short-lived, and by 1207 it had passed to Anselm de Morvil and then to William Daubeney, Earl of Arundel. It only remained in this line until 1240, when Hugh Daubeney, the 8th earl, was forced to surrender it to King Henry III.
The church was granted by the Domesday landholder to the Priory of St Faith at Longueville in Normandy. During the French wars the church was retained by the cell at Newton Longville, but in 1441 this alien priory was granted with all its possessions to New College, Oxford.