Byton was held by Osbern FitzRichard in 1086, and was assessed at 2 hides. He also held the manor before the Conquest, as he was the son of Richard Scrupe, a Norman who settled in Herefordshire before the Conquest and who built the castle which bears his name.