"If they aren’t out of there before the monsoons hit, their lives and health will be in serious danger," said the group's Asia director, Brad Adams.
The government has said it is holding the displaced Tamils to screen them for former Tamil Tiger fighters -- the separatist force that was finally defeated in May after decades of bloody warfare in north and east Sri Lanka.
Adams said the screening had become "a ruse to hold as many Tamils for as long as possible."
Most of the detainees are held at Manik Farm, a vast complex of makeshift shelters, which aid agencies and reporters are unable to visit freely.
Last month Sri Lankan troops opened fire as dozens of detainees tried to escape from the camp, injuring at least three.
The monsoon is expected to start within weeks.