The boat sank in the early hours of Friday near Myanmar's Naikkhangdiya coast, Mohammad Rafiq, a worker from Shahparir Dwip, told bdnews24.com.
According to Rafiq, there were 180 passengers on board, all intending to cross illegally into Malaysia in search of work. An agent, Sharif Hossain, was with them.
The boat had headed off from Teknaf around 3.30am.
Near Naikkhangdiya, the boat capsized and only 17 were able to swim ashore, Rafiq said.
The Coast Guard, Border Guard Bangladesh and local police were contacted but could not confirm the incident.