There were two manors in Chaddleworth recorded in Domesday. The larger was held as 2 manors by 2 free men from Countess Gytha and her son Gyrth and was assessed at 16 hides in 1066. By 1086 it had come into the possession of the Abbey of Winchester, and was assessed at 10 hides. Odo, the abbot gave it to Hugh de Port's steward, Robert. It subsequently passed to Roger d'Ivry and was included in the Honour of St Valery. Bernard of St Valery founded the Abbey of Lieu Dieu in France, which was certainly in possession in the 13thc when they rented out 5 hides to one Robert de Chaddleworth. Lieu Dieu sold their holdings here to Netley Abbey in 1247.
The second Domesday manor was held by Edward in 1066, and by Robert d'Oilly in 1086, as 4 hides at both dates. In the reign of Henry I it was held by Ralph Basset, Chief Justice of England, who granted it to Abingdon Abbey, a gift confirmed by the Pope in 1146. The abbey held the manor until the Dissolution, but Robert de Chaddleworth was holding it from the abbot in 1182.