Lawrence began building on a remarkable scale soon after 1281 with the completion of his work indicated by the issuance of a "license to crenellate" obtained from Edward I in 1291. Outside the walls, the manor house was surrounded by a water-filled moat fed from a nearby pond. Despite crenellations, curtain walls and moats, Stokesay was always regarded as a manor house rather than a castle. As head of a wool consortium that was raising money for Edward's war with France, Lawrence commanded a fleet of ships carrying wool to the Low Countries but was shipwrecked and drowned off Aldeburgh in 1294.

After 1563, the manor house passed through many different hands until the Craven family assumed ownership in 1620. Soon after the purchase it was let to Charles Baldwyn who probably built the ornate Gatehouse. Charles' son, Samuel, was involved in the only siege of the castle in 1645 during the Civil War. 500 Parliamentary foot and 300 horse soldiers came upon Stokesay on their way to take Ludlow. After a very short siege the castle was taken and subsequently ordered to be leveled but little more seems to have been done than the demolition of the curtain walls in 1647.