Coe, who headed the London Olympics, worked with Blatter as chairman of FIFA's ethics committee before taking a role with England's failed bid to host the 2018 World Cup.
Russia won the hosting rights after England's bid attracted only two votes following Blatter voicing concerns about English media intrusion into the FIFA executive committee.
In his new autobiography, Coe claims that Blatter has "always had a problem" with English football.
"From his perspective, it's not hard to understand," Coe writes in an extract published by The Times of London newspaper. "As president of an international federation, he sees the unwillingness of English clubs to release players for international duty. He sees the purchasing power of the English game — big-name clubs buying up players from all over the world.
"And he sees a national federation that, at the time of the bid, had no chairman or chief executive. And then he sees our press, permanently focusing on his organization."
Ahead of the FIFA vote, English Football Association chairman David Triesman was forced to quit after covert recordings of him making accusations about rival bidders were published by a newspaper. That came shortly after FA chief executive Ian Watmore quit following a power struggle.
"Your game is run by idiots. It's not run by bright people," Coe recalls Blatter once saying to him, without naming any individuals.
The power struggles in the English game frustrated Coe, who said "internecine warfare at home" hampered the World Cup bid.
"The fault, I believe, lies with the awful dysfunctionality of the English game, its personalities and its politics," Coe said. "First, you have the FA, which is the regulatory body. Then you have the Premier League, effectively the creation of Sky TV and (Rupert) Murdoch money, and we're talking billions here.
"Then you have the big clubs and the moguls, including (Roman) Abramovich at Chelsea and the Glazers at Manchester United, not forgetting the big-beast managers, such as Alex Ferguson (United) and Arsene Wenger (Arsenal). This has always been a very uncomfortable set of relationships, at best strained. The fact that they didn't trust each other and didn't much like each other was a problem."
Coe is now a vice-president of the International Association of Athletics Federations, and is due to become British Olympic Association chairman next month.