A caller who said he had knowledge of the situation told The Associated Press the misconduct involved prostitutes in Cartagena, site of the Summit of the Americas. A Secret Service spokesman did not dispute that.
A U.S. official, who was not authorized to speak publicly on the matter and requested anonymity, put the number of agents sent home at 12. Secret Service was not releasing the number of personnel involved.
The incident threatened to overshadow Obama's economic and trade agenda at the summit and embarrass the U.S. The White House had no comment, but also did not dispute the allegations.
In Washington, Secret Service spokesman Ed Donovan would not confirm that prostitution was involved, saying only that there had been "allegations of misconduct" made against Secret Service personnel in Cartagena for the summit.
Donovan said the allegations of misconduct were related to activity before the president's arrival Friday night and did not impact security plans for Obama's trip.
Obama attended a leaders' dinner Friday night at Cartagena's historic Spanish fortress and was due to attend summit meetings with regional leaders Saturday and Sunday.
The Washington Post reported that Jon Adler, president of the Federal Law Enforcement Officers Association, said the accusations related to at least one agent having involvement with prostitutes in Cartagena. The association represents federal law enforcement officers, including the Secret Service.
Adler later told the AP that he had heard that there were allegations of prostitution, but he had no specific knowledge of any wrongdoing.
The agents were staying at Cartagena's Hotel Caribe, which is also hosting members of the White House staff and press corps during the summit
A hotel employee, speaking on the condition of anonymity for fear of losing his job, said the agents arrived at the beachfront hotel about a week ago. The employee described the agents as drinking heavily during their stay.
The employee said the agents left the hotel Thursday, a day before Obama and other regional leaders arrived for the weekend summit.
The hotel's public relations chief had no comment.
Those involved had been sent back to their permanent place of duty and were being replaced by other agency personnel, Donovan said. The matter was turned over to the agency's Office of Professional Responsibility, which handles the agency's internal affairs.
Associated Press writers Libardo Cardona and Pedro Mendoza contributed to this report.