George Henry the 4th Duke was famous for dressing with a tall hat, brightly coloured frock coat and tightly strapped trousers and always carrying a tightly rolled umbrella under his arm. He was known for his dry wit and sense of humour. He was a Knight of the Garter, and an M.P. for 29 years, managing to escape the political intrigue that had dogged his father's political career. Other posts he held were Lord Lieutenant of Suffolk and Ranger of St. James Park and Hyde Park.

He did displease his father when in 1784 he married by licence Charlotte Maria Waldegrave, a charming and attractive woman, who was Horace Walpole's niece, and granddaughter of Sir Edward Walpole by his mistress, Dorothy Clements. He died in 1844 and was succeeded by his son Henry.

The 4th Duke continued the racing and breeding successes of the Euston Stud forward to the 1830s. Between them, the 3rd and 4th Dukes won approximately £250,000 in public stakes in the years their horses ran, largely between 1800 and 1831.

John Wastell managed their stables at Newmarket and when he died, his duties were assumed by the 4th Duke's half-brother, Rev. Lord Henry FitzRoy, described as, "a fine specimen of the sporting parson", for a very productive dozen years, until his death in 1823.

After Henry FitzRoy's death in 1823 and their trainer, Robert Robson's, retirement in 1828, the Grafton racing stables gradually ceased to send out winners, and by the 1840s few were left in the racing stud. The 5th Duke of Grafton had no interest in racing, and the dynasty established by the two Dukes was carried on by others, breeding the descendants of their famous mares and horses.