Two people were recovered alive and Veerassamy said “a few” more survivors arrived at an IOM facility in the town of Obock on Tuesday evening.
One survivor estimated there were 130 people on his boat, but was not able to estimate the number of passengers on the other vessel.
The nationalities of the victims remained unclear, Veerassamy said.
Located across the Bab el-Mandeb strait from Yemen and next to Somalia and Ethiopia, Djibouti has in recent years become a transit point for migrants heading to find work on the Arabian Peninsula.
In 2017 some 2,900 people, mostly Somalis and Ethiopians, passed through from Yemen, a year that saw about 100,000 migrants head into the troubled country.
“The number of new arrivals arriving in Yemen has been continuously increasing since 2012, despite the deepened insecurity and violence following the war that erupted in March 2015,” the IOM said in its 2018-2020 regional migration response plan.
“Arrivals peaked in 2016, when over 117,000 arrived in Yemen,” it added.
This migration comes despite Yemen facing one of the world’s worst humanitarian crises.
UN officials say 80 percent of the population — 24 million people — are in need of aid and nearly 10 million are just one step away from famine.
IOM said that most of the journey migrants take to Yemen is by foot, walking across the scorching desert regions of eastern Ethiopia, Djibouti and northern Somalia.
Once they arrive in Yemen they face torture, blackmail, sexual abuse or forced labour.
The sea crossing itself has repeatedly proven perilous.
Last year, at least 30 migrants from Somalia and Ethiopia believed to be headed for Djibouti drowned when their boat capsized off Yemen amid reports of gunfire being used against those on board.
In August 2017, dozens of migrants from Somalia and Ethiopia died after human traffickers forced them off two Yemen-bound boats and into the sea.