Building started under Napoleon Bonaparte in 1801 and was completed in 1857. Engineers Ferregeau and Armand Samuel de Marescot, and Vice-Admiral François Étienne de Rosily-Mesros designed a fort to be built on the bank. To facilitate the work, a port was established on île d’Oléron.
The project was suspended in 1809. Construction resumed in 1837, under Louis-Philippe, following renewed tensions with the United Kingdom. The fortifications were completed in 1857, with sufficient room for a garrison of 250 men.
After 1871, Fort Boyard was briefly used as a military prison, and abandoned in 1913, after which the unmaintained fort slowly deteriorated and crumbled into the sea. In 1950 it was made a listed building, and in 1961 was sold to Charente Maritime Regional Council.
Fort Boyard is stadium-shaped, 68 meters long and 31 meter wide. The walls were built 20 meter high. At the center is a yard, and the ground floor provided stores and quarters. The floor above contained case mates for the emplacements of guns and further quarters.