"When we moved to that boat, the big boat fell into the water and my baby (of) two months and my 21-year-old wife, and all died in the middle of the ocean," Aymo told reporters at the offices of the Greek charity Praxis.
"Only 41 made it, we swam to save our lives to the small boat. And I saved two persons," he added.
The two said although they managed to get back on board the smaller boat, the smuggler refused to wait and help others struggling in the sea.
Isman, who said his two sisters and her baby died in the tragedy, said "We saw the dead people with our eyes."
When the small boat's engine broke down, the smuggler was picked up by a third boat and promised to return.
"We were three days in the ocean," the 25-year-old Aymo said, adding that they had nothing to eat or drink on board.
By chance, the survivors found a mobile phone on board with only one number saved in the memory. Aymo called it, and it was the Italian police.
The 41 survivors - 37 men, three women and a three-year-old baby - were picked up on Saturday by a passing cargo ship and transferred to Greece.
The United Nations refugee agency UNHCR said it had interviewed some of the survivors, and their testimonies matched those heard by reporters. Up to 500 people are thought to have drowned.
It remains unclear where the sinking might have taken place and neither the Italian, Greek, Libyan or Egyptian coastguards have confirmed the sinking. To date, there are no reports of bodies or debris washing ashore.
If confirmed, the shipwreck would be one of the deadliest migrant boat sinkings in the Mediterranean.
Migration rights groups have warned that with new restrictions placed on migrants entering Europe from Turkey, the continent is witnessing a large number of attempts to cross by boat from North Africa to Italy.