Ambassador Kiyoshi Araki was one of the diplomats who had flown in two helicopters to the island's restive eastern Batticaloa district.
"Our ambassador was unhurt as he was on the second helicopter," a foreign ministry official said in Tokyo.
"The members of the group are now standing by at a safe place," he said.
Eleven people were wounded in the attack including the ambassadors of the United States and Italy, according to Sri Lankan officials.
Tiger rebels expressed "deep regret" for the artillery attack but accused the government of risking the diplomats' lives by taking them to a war zone.
They said the Sri Lankan government and the diplomats had failed to inform them that they would be flying in two military helicopters into an area where only the air force operated.
The shelling stopped after a UN official contacted the Tigers and said that diplomats were in the two helicopters, the Tigers said, blaming the government military for risking the lives of foreigners.