The concert was scheduled for Thursday at one of the city's main concert venues.
A statement from Young's spokesperson said "It is with heavy hearts and deep sadness that we must cancel our one and only Israeli concert due to tensions which have rendered the event unsafe at this time.
"We'll miss the opportunity to play for our fans and look forward to playing in Israel and Palestine in peace."
Despite the band's statement, a police spokesman told Reuters that it was police who cancelled the show "in order not to put people in Gaza rocket range at unnecessary risk."
Despite disappointing fans in Israel, Young's rep said the veteran rocker will make donations to "two organizations that teach music to Palestinian and Israeli youth simultaneously by enabling them to play music together.”
Big name artists like Justin Timberlake, Rihanna and the Rolling Stones have all held shows in Israel in the past few months, despite calls from pro-Palestinian groups to boycott Israel over its policy toward Palestinians.
Palestinian officials say 175 people have been killed by Israeli air attacks in the past week. Israel has said its offensive will continue until the heavy rocket fire from Hamas militants in Gaza stops.