Harlowbury is on the E edge of the former parish of Harlow, now Old Harlow, itself at the NE edge of the New Town conurbation. The chapel is alongside Harlowbury Manor, originally built by the Abbot of Bury using timbers felled in 1220-25. The present manor how encases the medieval one and dates from c.1860. The chapel is a single cell gabled building of flint rubble with clunch and brick dressings. It is dateable to the 12thc by plain lancet windows: 1 in the E gable, 3 in the W and 2 in the S wall, flanking a late-12thc doorway. Brick buttresses are a later addition; diagonal at the angles and regular on the side walls. The only Romanesque feature described here is the N doorway.

Four manors were recorded in Harlow in 1086. The first was a manor, originally of 1½ hides, that was held by Bury St Edmunds abbeybefore and after the Conquest. To this manor, KIng William added 3 hides that had belonged to 5 free men in King Edward's time. Half a hide that Beorhtmaer, a free man, held before the Conquest was held in 1086 by Geoffrey from Count Eustace, and Turgis held a manor of 1 hide and 3 virgates from Eudo the Steward that Godwine held in 1066. Finally 1 hide in Harlow was held as a Berewick of Roydon in 1086. The tenant in chief was Ranulf, brother of Ilger and Richard held it from him.

The relevant manor for Harlowbury was clearly that belonging to Bury St Edmunds, known as Harlow or Harlowbury manor, which remained in the abbey's hands until its Dissolution in 1539.

The body of the chapel is of a single build, dateable by diagnostic features of the N doorway. Bettley and Pevsner offer late-12thc, while RCHME suggests c.1180. A date in the 1180s or 90s is preferred here on account of the use of keeled mouldings as well as the developed waterleaf.