The Castle of Mey was built by George, the 4th Earl of Caithness, for his second son William Sinclair. When visiting the family seat Girnigoe Castle in1573, William was murdered by his older brother John, who had been imprisoned there for about six years by his cruel father. John had been planning an escape but William found out about it and told their father. John was in turn murdered and the castle went to the third son, George Sinclair, who founded the family of the Sinclairs of Mey and succeeded to the Earldom in 1789. He changed the name of the castle to Barrogill Castle. The castle then became the seat of the Earls of Caithness for the next one hundred years. In 1819 the twelfth Earl commissioned the architect, William Burn, to make various ambitious alterations to the castle. This was when the grand entrance and the dining room were added. His son, Alexander, was responsible for erecting the monument, now known as 'Lady Fanny's seat as a tribute to his friend, Charles John Canning, who later became the first Viceroy of India. George, the fifteenth Earl died at the age of 30; he had never married and having no children, left the castle to his friend F G Heathcote, on condition that he changed his name to Sinclair. His widow eventually sold it to Captain F B Imbert-Terry, who subsequently sold it to Her Majesty Queen Elizabeth The Queen Mother in 1952.