The Italian coach, who had four years remaining on his contract, was ditched just as players from rival Manchester United were being cheered through the streets as they paraded the Premier League trophy that they re-captured from their neighbours.
City said its end-of-season review was brought forward "out of respect for Roberto" after it was reported that Malaga coach Manuel Pellegrini has been lined up to replace him. Mancini's final days in charge were marked by his attacks on City for failing to deny those reports.
For his part, Pellegrini is only denying that a deal is in place, leaving open the possibility that he could be headed to the Premier League.
Mancini has paid the price for City's failure to build on its position of strength, with the club set to end the season without a trophy after losing the FA Cup final on Saturday and exiting the group stage of the Champions League for the second straight campaign.
"Despite everyone's best efforts, the club has failed to achieve any of its stated targets this year, with the exception of qualification for next season's UEFA Champions League," City said in a statement.
"This, combined with an identified need to develop a holistic approach to all aspects of football at the club, has meant that the decision has been taken to find a new manager for the 2013/14 season and beyond."
United regained the Premier League trophy from City with four rounds to spare, and manager Alex Ferguson announced retirement plans last week after almost 27 years at Old Trafford.
Ferguson leaves having seen off his 14th permanent manager of City, which only starting posing a real threat and winning trophies after Mancini replaced Mark Hughes in 2009.
By City's standards, before being bought by Abu Dhabi's Sheikh Mansour in 2008, finishing second in the league this month would have been a commendable achievement. But a team that has benefited from about $1 billion of Abu Dhabi investment in five years appears to be going backward.
The FA Cup final, which Mancini won in 2011 to end the club's 35-year trophy drought, was lost this year to relegation-threatened Wigan on Saturday.
City only beat United to the Premier League title last May on goal difference, but Mancini was initially rewarded with a new five-year contract. That was terminated on Monday.
"Roberto's record speaks for itself and he has the respect and gratitude of Sheikh Mansour, myself and the board for all of his hard work and commitment over the last three and a half years," chairman Khaldoon Al Mubarak said. "He has clearly also secured the love and respect of our fans. He has done as he promised and delivered silverware and success, breaking the club's 35-year trophy drought and securing the title in 2012.
"I would like to personally and publicly thank him for his dedication to the progress that he has overseen and for his support and continued friendship."
Assistant Manager Brian Kidd will take charge of City for the last two games of the season, at Reading on Tuesday and at home against Norwich on Sunday, before leading a post-season tour to the United States.
Even before Mancini's departure, it was destined to a busy off-season for player moves at City — despite the restrictions imposed by UEFA's Financial Fair Play rules.
City's poor recruitment in the last off-season laid the seeds for the eventual demise of the former Inter Milan coach.
The club failed to sign its top target, with Robin van Persie opting to join United from Arsenal, and was forced to settle for low-profile recruits in Jack Rodwell, Scott Sinclair and Matija Nastasic for about 35 million pounds (now $53 million) in total.
The 48-year-old Mancini let the board know about his unhappiness about the club's dealings, regularly complaining about it with reporters.
Rodwell and Sinclair have hardly figured this season, and the promising Nastasic has been in and out of the team.
Many of Mancini's big-name players — Sergio Aguero, Yaya Toure and captain Vincent Kompany — have either failed to shine or spent crucial periods out unavailable, affecting City's ability to compete on multiple fronts. While his longtime indulgence of Mario Balotelli before the Italy striker joined AC Milan in January raised questions over his managerial approach.
Losses this season to Southampton and Everton will count among some of the worst displays in Mancini's reign and were symptomatic or the team's regression.
City's move to appoint former Barcelona officials Txiki Begiristain and Ferran Soriano as director of football and chief executive, respectively, also left Mancini's role under a cloud and hinted at the club heading in a new direction.
Former Barcelona coach Pep Guardiola seemed to be the perfect fit to eventually replace Mancini, but he decided in January to join Bayern Munich on a three-year deal starting next season.
Soriano and Begiristain know Pellegrini well from their time in Spain and will probably be the biggest supporters behind the Chilean's likely appointment.
Mancini has also fallen out with a number of players this year, including Kompany over his decision to play for Belgium soon after returning from a spell out injured, and Samir Nasri for his poor work rate and performances.
"The team looks without passion, desire and (it's) difficult to find an answer why everything went wrong," defender Pablo Zabaleta said after Saturday's 1-0 loss to Wigan in the FA Cup final.
It will be up to the new manager to restore unity and desire to the club.
AP Sports Writer Steve Douglas in Manchester, England, contributed to this report.