There was a pre-Conquest church, a cross-fragment being among the 12th-century pieces reset. Once the Percys had acquired the manor, and stone was being shipped here, rebuilding would have been likely. About 1189, the church was given to Sawley Abbey, a foundation of the Percys (Anker 1991, 16).
The HER schedule 1017407 says 'The castle became neglected from the 12th century when the Percy family ceased to have a dwelling in Tadcaster.' The area of the motte and bailey is marked as wooded on the OS map.
The church was burnt and sacked by the Scots in 1318 (Anker, 1991, 6). VCH Yorkshire III, 404, describes an incursion of the Scots from Berwick to Pontefract and back through Craven in that year, but does not mention Tadcaster.