That didn't stop NBC from carving out a role for the announcer in its coverage of the Winter Olympics in Sochi, Russia.
Diffey is taking a break from the race track and making his Olympic debut as the play-by-play announcer for bobsled, skeleton and luge during NBC's coverage of the games, which get underway in earnest this weekend.
"Never in my wildest dreams did I imagine this is where my career would take me," Diffey told The Associated Press in a recent telephone interview. "Just working for NBC, that alone made my life. Now going to the Olympics for NBC? I just pinch myself as if to wonder is this really happening?"
An Australian who began his broadcasting career covering V8 Supercars in his native country and then the World Superbike Championship and WRC for the BBC, Diffey eventually moved to the United States and became a longtime member of the now-defunct Speed Channel.
He was a familiar announcer and host for just about anything with wheels ranging from NASCAR, Grand-AM, MotoGP, Formula One, CART and the 24 Hours of Le Mans.
When his contract was up and Speed was in the process of becoming Fox Sports 1, Diffey left for NBC Sports Group in 2013 to take the role as play-by-play announcer for Formula 1 and IndyCar coverage.
"Sam felt from the very beginning that, 'We can find a nice fit for you in the Winter and the Summer Games,' and I said, 'Sure,' but not really knowing what in the world he had in mind for me," Diffey recalled.
Flood found a role for Diffey right away at the Penn Relays track and field meet alongside sprint star Ato Boldon and at the World Cup Luge in Lake Placid, N.Y. After all, both events were essentially racing, which was right in Diffey's wheelhouse.
For the Olympics, Diffey has deep resources in the booth to assistant him. He's surrounded by analysts John Morgan, a former U.S. National Bobsled Team member working his ninth consecutive Winter Olympics, three-time Olympian Duncan Kennedy during the luge and 2010 Olympic bobsledder and former skeleton racer Bree Schaaf. Lewis Johnson, who will be working his eighth Olympics for NBC, will be the reporter for all three disciplines.
Even with that safety net and the deep NBC research department, Diffey has been studying endlessly to learn as much as he can. He's done it while maintaining his regular schedule: Leading up to his departure for Russia, Diffey called the Dakar Rally from a studio in the U.S., and inducted Mario Andretti into the National Motorsports Press Association Hall of Fame.
"I've been soaking it up, trying to be a sponge, preparing the same way you would for any sport," Diffey said. "It would be a little arrogant to say, 'It's just racing,' because it is a different form of racing. But at the end of the day, there is a beginning, a middle and an end. There's also a story to tell, and it's my job to tell the story the best I can."