He was joined by his official delegation, which includes Conservative MPs and cabinet ministers.
According to the CBC's Terry Milewski, Harper's note in the memorial's guestbook reads: "They are remembered always, in our hearts, in our prayers and most importantly in our resolve. Never again."
Afterwards on Tuesday, the delegation made a trip to the Western Wall, where the prime minister took a moment to pray at the holy Jewish site.
The delegation was also scheduled to visit the Dome of the Rock, also located in Old Jerusalem. The site holds religious significance to Jews, Muslims and Christians. However, due to security concerns, Harper cancelled his trip there.
According to an email to CBC News from PMO spokesman Jason MacDonald, security organizations weren't satisfied with the planning and logistics. Specifically, Shin Bet (the Israeli security agency) would not guarantee that they would not enter the mosque.
"Out of respect we have decided not to visit the Dome of the Rock on this trip," MacDonald wrote.
Earlier in the day, Harper was greeted by Israeli President Shimon Peres on a red carpet.
The two held a photo-op in Peres's office and had a warm discussion comparing the two countries, with Harper noting that Canada is the "polar opposite of Israel" because it has "much geography but very little history."
Importantly, Harper and members of his cabinet also met with Israeli Prime Minister Benjamin Netanyahu to sign a wide-ranging set of bilateral goals, which includes growth in security and intelligence co-operation, enhanced defence and security relations, more business links and closer academic ties.
The two leaders also pledged to expand the existing Canada-Israel Free Trade Agreement.