The fortress was founded by Erik Axelsson Tott in 1475 under the name Sankt Olofsborg. The area was chosen from the point of view of defense: steep, rocky islet emerges from a strong stream at the meeting point of two waterways.
Before the building of the actual stone castle could be started, it was necessary to build a wooden fortification to protect the builders, who straight from the beginning had become the victims of Russians’ attacks.
The three-towered keep was completed in 1485, and the construction of the outer curtain walls with two towers was initiated immediately. They were completed at the end of the15th century. The castle is roughly a truncated rhomboid with keep on the western side of the island and the curtain walls and outer bailey to east. One of the towers, has a bad foundation and has since collapsed. One of the towers of Bailey, the Thick Tower, exploded in the 18th century. A bastion has been built on its place. The castle was converted into a Vaubanesque fort in the late 18th century with bastions.