I don't know when the Crawley family line began, nor when they became titled nobility with the Earldom of Grantham, nor when 'Downton Abbey' was built. But it could be that the family fortune was buttressed by ill-gotten pirate booty at some point.
The title of Earl of Grantham and the inheritance of all the lands associated with the title was handed down to the next male heir in the Crawley family. That meant even going outside the established bloodline should there be only girls in the current Earl's family as we have seen during the reign of George V. Suitable marriages would have to be arranged for the daughters of the Earls and so the family tree would branch out in all directions.
One of those daughters, back in the middle of the 17th Century, may have been married off to an Englishman of noble birth named Avery. (The name itself means "nobility".) If they had children, only the oldest son would inherit the Avery estate; the others would have to find other employ.
They may have had a younger son named Henry, probably born around 1651. In order to make a name for himself and provide for his future, he joined the Royal Navy. Eventually he married, or at the very least, he fathered a son named Toby (Hey now!) by a woman who later died when the lad was about twelve years of age.
Something went wrong with his career choice and Henry Avery became a privateer, commander of the good ship "Fancy". And in April of 1699, his path crossed that of the Doctor and his Companions when their ship was becalmed and being attacked by an alien siren.
(There's also something about Hugh Bonneville in a beard that kept reminding me of Dabney Coleman, so who knows? Perhaps former Fernwood, Ohio, mayor Merle Jeeter can trace his lineage back to the sea captain.....)
'Doctor Who' - "The Curse Of The Black Spot"
'Mary Hartman, Mary Hartman'