The purported message also claims the group is now demanding a prisoner exchange for the remaining hostage, Kenji Goto.
Japan's public broadcaster, NHK, says the Japanese government is investigating the authenticity of the video and quotes a top government official as saying the purported execution is "outrageous" and "unacceptable."
Earlier, the Japanese government promised on Saturday not to give up "until the very end" on efforts to free the two hostages from Japan.
The Islamic State group had threatened on Tuesday to behead the hostages within 72 hours unless it received a $200 million ransom.
Japan has scrambled for a way to secure the release of 47-year-old Goto, a journalist, and Yukawa, a 42-year-old adventurer and independent security contractor. Japanese diplomats had left Syria as the civil war there escalated, adding to the difficulty of contacting the militants holding the hostages.
ISIS had threatened to behead the hostages if demands for a $200 million ransom were not met.
Chief Cabinet Secretary Yoshihide Suga said late Saturday that cabinet ministers were holding an emergency meeting about the new video message.
More to come