Baga scored with a header just before halftime, which proved enough to secure the three points at the City of Coventry Stadium.
Belarus dominated the first half, before New Zealand came out stronger for the second but was unable to break through the Europeans' defence.
"Getting three points in such a tournament is really great and I am very satisfied with the play of our athletes today," said Belarus coach Georgy Kondratyev.
Sergei Korolenko had the first chance for Belarus in the opening minute, rifling a shot wide of New Zealand 'keeper Michael O'Keefe's goal. New Zealand striker Chris Wood forced a save from Belarus goalkeeper Aleksandr Gutor when he powered a header toward goal in the 15th minute.
O'Keefe then did well to keep out an attempt by Renana Bressan Bardini, who nearly scored from a corner. The goalkeeper was called into action again from the resulting corner, denying Baga from close range.
However, O'Keefe was partly at fault for the Belarus goal, as he failed to get to a cross from the left win, leaving Baga totally unmarked to head home.
New Zealand defender Ryan Nelson said he was "bitterly disappointed" with the result.
"We had the majority of the possession and created some really good chances," he said. "Generally most games are decided by mistakes and that was the case. ... Going a goal down was tough to take."
Belarus faces tournament favourite Brazil on Sunday in its second game of the group stage. New Zealand plays Egypt.