Thilan Samaraweera was unbeaten on 10 and Michael Vandort on four not out at the break.
The hosts lead 1-0 in the three-Test series following their victory by an innings and 239 runs in the opening match in Colombo.
Sharma took two wickets and left-arm seamer Zaheer one in their incisive opening spells to put the hosts under pressure on a wearing track at the Galle International Stadium.
Sharma struck with his third ball which had opener Malinda Warnapura caught by Venkatsai Laxman at second slip, before dismissing in-form skipper Mahela Jayawardene in his next over.
Jayawardene, who scored a hundred in Colombo and 86 in the first innings here, was caught by Rahul Dravid at gully after contributing just five runs.
Zaheer's lone victim was Kumar Sangakkara, caught by Laxman in the slips after making only one.
India earlier failed to cope with Sri Lankan spinners Muttiah Muralitharan and Ajantha Mendis as they were bowled out for 269 after resuming at 200-4.They lost their last five wickets for 17 runs in a dramatic collapse.
Unorthodox spinner Mendis, who took six wickets in the first innings, added four more to his tally to complete a maiden 10-wicket haul in only his second Test.
Muralitharan took two of the six wickets that fell in the morning.
India suffered an early setback when overnight batsman Laxman (13) was trapped leg-before off Mendis.
Wicket-keeper Dinesh Karthik (20) counter-attacked, hoisting Muralitharan and Mendis for sixes early in his innings, but his flourish did not last long as he was caught by Sangakkara at deep mid-wicket.
Sourav Ganguly earned a reprieve when he was initially adjudged leg-before off Muralitharan by umpire Billy Doctrove of the West Indies, but asked the official to review his decision.
Doctrove consulted TV umpire Gamini Silva of Sri Lanka before changing his decision. The batsman was then on seven.
Ganguly failed to capitalise on the chance as he was caught behind off Muralitharan after contributing 16.
A new experimental rule allowing players to seek a second opinion on umpiring decisions is on trial in the ongoing series.
It allows a batsman or fielding captain to request a review of any decision by referring it to the third official monitoring television replays.
