The castle was built around 1530 by the Elphinstone family and leased to the Forbes of Towie. In 1571 it was burned by their enemy, Adam Gordon of Auchindoun, resulting in the deaths of Margaret Campbell, Lady Forbes, her children, and numerous others, 26 in total, and giving rise to the ballad Edom o Gordon[dubious – discuss].
In May 1607 the castle was captured from Alexander, 4th Lord Elphinstone by Alexander Forbes of Towie and his companions, including a piper called George McRobie. They used hammers and battering rams to break down the gate, then fortified the house with a garrison of “Highland thieves and limmers”.
In 1626 it was acquired by the Earl of Mar. In 1645 it was used as an assembly point by the troops of the Marquis of Montrose. It was burned again in both 1689 and 1716 by Jacobite supporters. It was resettled by the Forbes family