When the brand that bears his name almost went bust four years after he had sold up and retired...
When the brand that bears his name almost went bust four years after he had sold up and retired, he dumped his golf clubs and repurchased the company, pumping in two million euros ($2.3 million) of his own money to put it back on its feet.
"I was 70, but I was bored stiff," Clergerie recalled.
He had toured the Greek islands, taken up gardening and golf, but he was "bored to death, particularly by the golf".
"It killed me to see the company go like that. I knew almost all the workers."
"The day I returned to the factory the women were in tears," he said, the emotion welling up in his voice.
"I found a second youth," he told AFP, and so did his brand, its trademark flat and wedged shoes and boots again becoming a must-have for fashionistas.
While other brands have been forced to go abroad for cheaper labour, Clergerie has bucked the trend.
Its factory is the last in Romans-sur-Isere, once the capital of French shoemaking.
Yet its founder, now 83, came very late to the business.
The son of a grocer, "my father brought me up the hard way, working in his shop until I was 24.
"I went to America but was called up to fight in Algeria (during its war of independence). Then I did lots of jobs, none of which made me happy," Clergerie admitted.
It was only at 37 that he discovered his passion after replying to a small ad in a newspaper: "Wanted -- someone strong enough to fill the shoes of manager of a factory in the Rhone valley, close to the mountains and the sea."
"It was the revelation of my life," Clergerie recalls in a new book, "Robert Clergerie, the Man Who Shod Women".
In 1978 he struck out on his own, taking over a factory in Romans and launching the first collection in his own name three years later.
His first three styles -- Paris, Paco and Palma -- were an instant hit, wowing the women's market with flat slightly boyish shoes which caught the spirit of the times.
"The 1980s were a golden age of disco, partying and real excitement," he said, with Bianca Jagger, Hollywood legend Lauren Bacall and Madonna beating a path to his Paris shop, which sold 11,000 pairs a year.
But the company grew too quickly, and needed more cash to keep up with demand. In 1996 Clergerie sold his controlling stake to investors, though he still held 10 percent of the shares and stayed on until 2001 as artistic director.
"Signing it over killed me," he said. "I went home and cried and vomited."
In a few short years his two main rivals in Romans went to the wall and by 2005 his own brand was also on its uppers.
That was when Clergerie returned to save the day. The brand prospered and seven years later he finally retired, passing the company and its 17 shops around the world on to French and Hong Kong investors, who had promised to keep making the shoes in Romans.
"It is now five years since I really retired," said Clergerie, and "I am starting to detach myself, though not totally," he added with a note of warning.
Because the Clergerie is "like a woman I loved madly -- and now there is someone else in her bed."