The Joint Rescue Co-ordination Centre in Halifax issued a brief statement just after 6 p.m. saying the search would be handed over to the RCMP because of "frigid water and poor weather conditions."
Two coast guard vessels and five aircraft had covered more than 18,000 square kilometres of open ocean since the search started late Sunday shortly after 11 p.m.
Nova Scotia Fisheries Minister Sterling Belliveau is from Woods Harbour, the same village in southwestern Nova Scotia where most of the missing men are from.
"This is a very difficult time," he said on the Woods Harbour wharf as the sun went down on a still, flat-water night.
"These are professionals that makes these calls and we have to understand that."
Belliveau said he visited earlier in the day with George Hopkins, a childhood friend whose 27-year-old son, Joel, is among the missing crew. He said he knows all of the missing men through hockey and other sports.
Davis Nickerson said he also knew all of the "boys" on the boat, having watched them grow up and move into the fishery.
Nickerson, 65, said fishermen have tied up their boats at the Woods Harbour wharf out of respect for the men.
He said the boat's captain, Katlin Nickerson, was interested in technology and gearing up his boat with high-tech instruments.
"He just loved to go fishing," he said. "You couldn't hold him down, same as his crew. They loved it, bottom line, they loved it. It's what we do. It's in our blood."
A spokesman for the Mounties said they would release the names of the men Wednesday.
But at a small church on the road that winds through Woods Harbour, about two dozen people gathered for an emotional prayer service where the names of the five missing men were projected on a wall at the front of the room: Katlin Nickerson, Billy Jack Hatfield, Joel Hopkins, Cole Nickerson and Tyson Townsend.
News spread quickly through the tight-knit communities that dot the shoreline that the search had been called off, giving way to the grim resignation many seemed to be fighting off through the day.
"We do have to face reality that, short of a miracle, the boys will have been lost," an unidentified man said before five flickering candles at the front of the church.
Another man, who remembered his son's own close call at sea while fishing lobster, rose to say that "the miracle we were hoping for and praying for may not come."
The military said the 13-metre fishing boat Miss Ally had capsized about 120 kilometres southeast of Liverpool as it was tossed about by 10 metre waves whipped up by winds approaching hurricane force. The ocean temperature at the time was hovering between 2 C and 4 C, according to Environment Canada.
By early Monday, the crew aboard a U.S. Coast Guard jet reported seeing a life-raft in the dark, but they couldn't tell if anyone was inside.
Lt. Peter Ryan, a navy spokesman in Halifax, said the likelihood that the fishermen survived beyond 24 hours was remote, even if they were all wearing immersion suits.
"That 24 hours is what we use as a survivability rate, as a general rule of thumb," Ryan said earlier in the day.
Tina Hatfield of Cape Sable Island said her nephew, Billy Jack Hatfield, is one of the missing fishermen.
"Most of the young men in this community have fished since they were able to go aboard a boat," she said. "Most of the time, the young fellas go in the boat after their fathers and grandfathers."
Hatfield said her nephew, in his early 30s with two young children, is the oldest member of the crew, an outgoing man who grew up on Cape Sable Island and also worked as a carpenter.
The six-year-old moulded plastic fishing boat was on an extended trip to catch halibut when its emergency locator beacon transmitted a distress call via satellite Sunday night.
The beacon, known as Emergency Position Indicating Radio Beacon or EPIRB, is typically activated when it hits salt water. The navy says there were no distress calls broadcast from the vessel's crew before the EPIRB went off.
On the night the boat capsized, an offshore buoy — about 160 kilometres from Shelburne — recorded wind speeds at 92 km/h, and a weather station on Cape Sable Island registered southwest gusts at 124 km/h.
By Tuesday morning, wave heights had dropped to about six metres but the wind remained strong under clear skies.
A military Cormorant rescue helicopter, based in Greenwood, N.S., was joined Tuesday by a C-130 cargo plane from the same base. A small plane chartered by the Fisheries Department was also dispatched to the area.
The U.S. Coast Guard also sent two aircraft Tuesday to join the search — a Falcon and a twin-prop Ocean Sentry.
Two coast guard vessels — the light icebreakers Earl Grey and Sir William Alexander — also helped with search efforts.
The Transportation Safety Board has sent investigators to Clark's Harbour, N.S., to assess what happened.
— With files from Michael MacDonald in Halifax
Note to readers: This is a corrected story. A previous story said police would release names on Tuesday.Suggest a correction