The pirates were among eight men whose boat overturned off the coast of Kenya in a storm as they left the Sirius Star following a two-month standoff that ended on Friday, a pirate and a relative of one of the dead men said.
"Six of our boys perished at the sea while coming from the released Saudi supertanker," Mohamed Said told the AFP.
"The small boat that was carrying those killed and eight who survived was overloaded and at high speed, we have been told by the survivors.
"They were afraid of a chase from outsiders (foreign naval forces) who invaded Somalia waters," he said.
The group's $300,000 share of the ransom was also lost, he said.
Three pirates reached shore after swimming for several hours, pirate Daud Nure said on Saturday.
"There has been human and monetary loss but what makes us feel sad is that we don't still have the dead bodies of our relatives. Four are still missing and one washed up on the shore," Abukar Haji, the uncle of one of the dead men, said.
The Sirius Star, owned by the shipping arm of oil giant Aramco, was carrying crude oil valued at $100m when it was hijacked on November 15.
Saudi Arabian oil minister Ali Naimi said on Saturday that the crew was safe and returning to the Persian Gulf coast city of Dammam.
An Iranian-chartered cargo ship was also released by pirates.
The ship Delight was carrying 36 tons of wheat when it was attacked in the Gulf of Aden on November 18 and seized by pirates.
All 25 crew are in good health and the vessel is sailing toward Iran.
The US Navy announced this week that it will head a new anti-piracy task force after more than 100 ships were attacked last year.
Other countries, including Iran, Russia and India, have sent naval ships to patrol waters around the Horn of Africa as acts of piracy increasingly threaten the safe passage of maritime cargo through the Gulf of Aden towards the Suez Canal.
Somali pirates are holding 17 ships and more than 250 crew members hostage, and have carried out more than 100 attacks in the last year.