It is not known whether a motte and bailey existed at RagIan when the Normans moved into the area around 1071. The detached Keep-the "Yellow Tower of Gwent"-stands on a mound of natural marl while the curving line of the courtyard buildings would also seem to indicate previous building on the site. About the year 1174, RagIan was granted to Walter Bloet whose male heirs held the estate until the late 14th century, when, on the death of Sir John Bloet, it passed to his only daughter, Elizabeth. It was during Elizabeth's second marriage to William ap Thomas that the Great Keep was built. Ap Thomas, knighted by Henry VI in 1426, saw service in France where he was undoubtedly influenced by the local military architecture incorporating such features as prominent machiolations, the polygonal keep and the double drawbridge into Raglan.

On ap Thomas' death in 1445, his son William, a man of considerable influence during the early years of the reign of Edward IV and a great supporter of the House of York during the War of the Roses, built Raglan into its present appearance. However, he was not to enjoy his palatial castle for long; being defeated and subsequently executed, along with his brother, for supporting the losing side at the battle of Edgecote in July 1469.

In June and July 1646, during the English Civil War (1642-1651), Colonel Morgan besieged the castle. When Morgan placed mortar batteries within 60 yards of the castle walls, the defenders realized that further resistance was futile and surrendered on August 19, 1646 ending the first phase of the English Civil War.

The deliberate destruction of RagIan commenced:

"The Great Tower, after tedious battering the top thereof with pickaxes, was undermined, the weight of it propped with the timber whilst the two sides of the six were cut through: the timber being burned it fell down in a lump, and so still remains firmly to this day".