Footage showed the three-storey wooden stand filled with spectators falling at a stadium in El Espinal, Tolima department.
A bull continued to roam the ring as people fled the wreckage.
The traditional "corraleja" event involved members of the public entering the ring to engage the bulls.
The dead included two women, a man and a child, Tolima Governor Jose Ricardo Orozco said. The region's health official said hospitals had treated 322 people of whom four were in intensive care.
Sunday's event was part of celebrations of the popular San Pedro festival.
Outgoing President Ivan Duque said there would be an investigation, while President-elect Gustavo Petro urged local officials to ban such events.
"I ask mayors not to allow more events involving the death of people or animals," he said.
It was not the first time such an accident had happened, he added.
On Saturday several people were injured during corralejas in El Espinal. Earlier this month one person died after being gored by a bull during a corraleja in the town of Repelon.
Mr Orozco said his regional authority would move to ban the corralejas, saying they were dangerous and cruel to animals.
Incoming leader Mr Petro banned bullfights in the main bullring of Colombia's capital Bogotá during his stint as the city's mayor.