Little St Mary's stands on Trumpington Street in the city centre. The present church consists of a single-unit mid-14thc. rectangular church with no aisles, a S vestry with a crypt below and a stair above linking it to Peterhouse College, a S chapel of 1931 by T. H. Lyon , a 19thc. S porch at the W end of the nave, a 19thc. NW porch, and a W choir vestry, added in 1892-93 and now used as a parish room. The E window tracery reveals workshop connections with the Lady Chapel at Ely. There was a major restoration by Sir George Gilbert Scott in 1857-62, and the interior was whitewashed in the 1950s by Stephen Dykes-Bower. None of the present fabric is any earlier than the 14thc., but there is a nook shaft with chevron ornament set into a corner of the NW porch, presumed to be from a 12thc. church on the site.

Originally St Peter-without-Trumpington-Gate, the church and its endowments were made over to scholars lodged in houses south of the churchyard by Hugh of Balsham, Bishop of Ely, in 1284. This marks the foundation of Peterhouse, the first Cambridge college. It was rebuilt c.1350, and dedicated to St Mary of Grace in 1352. In 1632, when a new Peterhouse chapel was consecrated, St Mary's was made over largely for parish use, although Fellows of Peterhouse continued to serve as ministers until the 19thc., and the college is still the patron of the living.

Similar point-to-point chevron is found on the W doorway and the nave arcade of the Holy Sepulchre, Cambridge.