Before the Conquest Charlton Adam was held by three thegns with one clerk, and in 1086 by Reginald de Vautort from Robert, Count of Mortain. It was assessed at 5 hides with 50 acres of meadow and 40 acres of pasture and 20 acres of scrubland.

The overlordship presumably passed from Robert of Mortain to his son William, who forfeited his estates to the crown in 1106. By 1160-61 the tenancy had passed to John, son of Hamon whose family had been Domesday tenants of Count Robert at Buckhorn Weston (Dorset). It stayed in this direct line until 1243 when, on the death of Henry, son of Richard, it was divided between five coheirs. Arrangements between these heirs resulted in a moiety of the manor passing to Geoffrey de Mandeville. The complex later history of the manor may be found in VCH. The church along with the rectory estate was granted by John, son of Hamon to Bruton Priory, between 1142 and 1166. A vicarage had been ordained by 1291, and the advowson descended with the rectory and its lands until at least 1626 (i.e. with Bruton Priory until the Reformation). The benefices of Charlton Adam and Charlton Mackrell were united in 1921.