Officials said the pre-dawn naval battle erupted in waters off Mullaittivu, a jungle and lagoon area and the last town in Sri Lanka still in the hands of the Liberation Tigers of Tamil Eelam (LTTE).
A naval blockade has been stepped up to prevent any rebel leaders from leaving the country.
On the ground, government forces claim they have the Mullaittivu area surrounded.
Reeling under the military's biggest ground, air and sea offensive in more than three decades of fighting, the Tigers have seen their territory rapidly shrink, with their political capital Kilinochchi falling earlier this month.
Sri Lanka's army chief, Lieutenant General Sarath Fonseka, said over the weekend that the LTTE's leadership, including elusive rebel supremo Velupillai Prabhakaran, 54, may have already fled by sea.
The guerrillas still control a 40-kilometre (25-mile) stretch of coastline in Mullaittivu around which the navy said it has set up four layers of barriers to restrict guerrilla boat move