The Castle was fortified by the 1100s, and from around 1200 it was adopted as the main residence of the bishops and archbishops of St Andrews. Many changes were made in the following years before it finally fell to ruin in the late 1600's.