Fujita, now with the Cleveland Browns, is undergoing treatment for an injured left knee which he hurt during the preseason and couldn't come to New York, the NFLPA said.
NFL spokesman Greg Aiello said the players left the NFL offices around 4 p.m. EDT, about three hours after they arrived. Smith and Hargrove did not make themselves available to reporters.
Fujita requested a video conference but the league rejected that idea. NFL officials wanted face-to-face meetings and plan to reschedule a session with the linebacker.
Fujita did not play in the season opener against Philadelphia because his suspension was only lifted two days before the game, and he wasn't able to practice on his knee, which he injured during an exhibition game against Detroit.
The 33-year-old made his season debut for Cleveland on Sunday. He started at strongside linebacker and was credited with four tackles in 36 plays during the Browns' 34-27 loss to the Bengals.
The players had to meet with Goodell after the suspensions he placed on them were lifted by an appeals panel. Fujita, Smith, Hargrove and Jonathan Vilma were penalized by Goodell for their roles in the New Orleans bounty program, which ran from 2009-11.
The NFL says coaches and players offered cash rewards for big hits during that time frame, though the players deny they were attempting to injure opponents.
The commissioner met Monday with Vilma, still a New Orleans linebacker. He and his attorney were given a sworn statement from former Saints defensive co-ordinator Gregg Williams saying the linebacker placed a $10,000 bounty on then-Vikings quarterback Brett Favre.
Ginsberg said they were given an affidavit at the meeting.
"What Gregg Williams said in his most recent affidavit is the same falsity he has previously provided," Vilma's attorney, Peter Ginsberg said on Monday night.
"I don't know what Gregg Williams' motives are, but I do know that any suggestion by Williams that Jonathan put up $10,000 as an incentive for his teammates to injure another player is absolutely false."
Vilma tweeted on Monday night that Williams was "bullied to sign the affidavit," saying Williams signed it on Friday.
AP Sports Writer Tom Withers in Cleveland contributed to this story.
Follow Rick Freeman at https://twitter.com/RWFreemanSuggest a correction