The village lies about 4 miles S of Hungerford. This small parish church was built in 1855-1856 of rubble with freestone dressings to designs by Robert Jewell Withers on the site of the former church within the earthworks of a medieval village. The font is probably a 19thc reworking of a 12thc piece.

The Domesday Book records that in 1066 Buttermere was held by Godwin; in 1086 it passed to Ascelin, Waleran the Hunter, and Arnulf of Hesdin. From the 11thc the manor of Buttermere was also held by St Swithun's priory at Winchester. The church was first mentioned in 1284 as St. Swithun's priory confirmed the bishop of Winchester's right to collate rectors of Buttermere. It was dedicated to St James only in 1763.

Before being rebuilt due to its poor conditions, the church was depicted in an 1806 watercolour by John Buckler (Devizes, Wiltshire Museum): the building appears as a simple structure featuring a N porch and a wooden W turret.