Joseph Elsby Martin, he began coaching Cassius Clay in 1954 , who later became a three-time world heavyweight champion under the name of Muhammad Ali. At the time the future Muhammad Ali began training, Columbia Gym was racially integrated, unlike other Louisville boxing gyms of that period.

Ali was first directed toward boxing by Joseph Elsby Martin. In a tale that has become a key piece of Louisville history, 12-year-old Ali, then known as Cassius Clay, attended a Louisville Service Club event at the building and parked his new red Schwinn bicycle outside. When he came back out, the bike had been stolen. An angry Clay informed Martin about the theft and told him that he planned to “whup” whoever took the bike.

Martin told Clay that before he tried to do that, he better learn to fight first, leading him to start training in the Columbia Gym and begin a storied amateur career.