Investors applauded the move, pushing Abercrombie's shares up 8 per cent.
Jeffries, who has served as CEO since February 1992, is also retiring from the retailer's board of directors.
He reinvented the chain from an ailing retailer of hunting apparel to a seller of teen clothing that became a must-have brand for young consumers. But since the recession, it has lost its appeal amid stiffer competition and changing tastes.
Last week, Abercrombie & Fitch cut its annual profit outlook and reported an 11.5 per cent drop in sales in its third quarter.
"I believe now is the right time for new leadership to take the company forward in the next phase of its development," Jeffries said in a statement.
The company, whose other brands includes Abercrombie, Hollister Co. and Gilly Hicks, said it has begun a search for Jeffries's successor and has hired a search firm to evaluate both internal and external candidates.
Jeffries' departure was expected, but BMO Capital Markets analyst John Morris noted that the timing was "noteworthy" given his employment contract didn't expire until February.
Jeffries leaves under a cloud of controversy. Last year, activist hedge fund Engaged Capital sent a letter to the company's board pushing for Jeffries to be replaced. Engaged said at the time it believed the retailer's "perennial" underperformance was a result of a failure of leadership.
The outspoken Jeffries also stirred up controversy for statements about how Abercrombie & Fitch goes after attractive kids who can fit into its clothes, alienating customers who don't mirror the brand's image.
Like its competitors American Eagle Outfitters and Aeropostale, Abercrombie & Fitch has been looking to stock trendier clothing as its sales have weakened and teens have chosen to shop at lower-priced rivals like H&M and Forever 21.
Abercrombie has even stripped its once-prized Abercrombie logo off more products as teens are now seeking more individuality in their clothing. But the efforts haven't born fruit.
For the fall season, Abercrombie reduced its logoed merchandise by half and previously announced plans to try and have a minimal North American logo business in the spring.
Abercrombie & Fitch has also increased its emphasis on online shopping as it looks to reshape itself.
As part of the management changes announced Tuesday, the chain said that current Nonexecutive Chairman Arthur Martinez will become executive chairman. He has served as nonexecutive chairman since January, when Abercrombie separated its chairman and CEO roles and stripped Jeffries of the chairman title.
Abercrombie & Fitch Co. also said its board has created an Office of the Chairman that will be led by Martinez and will oversee Abercrombie & Fitch's strategic direction. It will also be responsible for managing the company's day-to-day operations until a new CEO is appointed.
Abercrombie & Fitch had 834 stores in the U.S. and 166 stores across Canada, Europe, Asia, Australia and the Middle East at the third quarter's end.
Shares of the New Albany, Ohio-based company rose $2.11 to close at $28.46.