X is for the X factor

Our meeting had been scheduled to take place at The Ivy but just as I was setting off, one of 'Richard's people' rang to say he had pulled a muscle in his neck, so instead I was to head to his office off the Euston Road. It's always annoying to be turned away from The Ivy, but at least I get to see Richard Caring's art collection, or the modest portion of it that he keeps in his ballroom-sized office. There are a couple of Degas charcoals of dancers, a stunning Matisse, and a vast Henry Moore

The neck injury, it turns out, was caused by overexuberant running around the Wentworth course the previous weekend during the PGA tournament. 'It was an epic day,' says Caring. 'There was the world number one up against world number two, never to be repeated. It was amazing - I had to remind myself that I own it, that I own Wentworth.'

Despite dominating it, Caring remains in some ways an outsider in the London restaurant and club world. But he has always fitted in with golf on his own terms, for he was once a county-level scratch player. His game has suffered in recent years through lack of practice, though as he says, 'When you own a golf club you tend to set your own handicap.' Towards the end of 2004, a friend, knowing his passion for the game, called to tell him Wentworth was for sale, so Caring pounced with a bid reported to be £130 million, well over its then value. Caring was unrepentant, explaining that it is to golf what Wimbledon is to tennis, and that Wentworth was 'priceless'. It was a typical Caring manouevre, instinctive and incisive.

Caring is unusual-looking for a man of 63. He's whippet-thin, toned, tanned and immaculately groomed in his black Armani suit. He has extraordinarily white and impeccably spaced teeth, which he reveals when he laughs - which is quite often. He has lustrous hair, which may be tinted with grey and is swept back in a light, centre-parted bouffant. His accent is mid-Atlantic rather than North London, where he lives in what has been called the Versailles of Hampstead, which includes what he describes as either a small lake or a big pond, and the rest of his art collection (mostly 20th-century art, with a preponderance of Matisse; he likes the way he paints geometric patterns). He also likes to throw a lavish party, spending a reputed £9 million on flying 500 guests to a charity ball in St Petersburg six years ago. His son Ben's wedding in 2009 was held in the vast Hampstead home, with the dancefloor on a platform above the lake and individial swan ice sculptures filled with caviar as a starter. The wedding was attended, as Hello!'s breathless account put it, by a 'high-value guest list', including Philip Green, Kevin Spacey, Kate Moss, Elizabeth Hurley, Peter Stringfellow and David Walliams.

When not in Hampstead, Caring can be found at his country home in Somerset, often described as a shooting lodge, though apparently with all mod cons, or on his yacht, which he keeps in Italy. On the day we speak, Caring told me that his old friend Goldie Hawn was sailing on his boat.

Caring has been married for many years to Jackie, a former model, and has two grown-up sons, Jamie and Ben, who both work within their father's empire: Jamie as commerical director of the clubs and main London restaurants, and Ben, the younger, working for the catering and events arm of the business. Caring recently became a grandfather for the first time to Ben's son.

As we sit down to talk at his boardroom table, Caring removes his thin silver cufflinks and prods them with his finger as he talks. 'Robin Birley is great,' he asserts. 'Robin and I are friends. I have a lot of admiration and respect for Robin.' Caring famously fell out with Robin, son of Mark Birley, the man who founded Annabel's and Mark's Club, when Birley Sr, having disinherited his son, sold his clubs to Caring for over £90 million in 2007. Robin Birley was understandably resentful that his birthright had been sold from under his nose, and responded with plans to launch a club in Shepherd Market, a five-iron shot away from Annabel's. His original plan was to call it Birley's, but Caring objected and threatened to go to the High Court, saying he had bought the Birley name along with the clubs. The legal wrangle caused personal rancour. Jemima Khan, half-sister of Robin Birley, began a Twitter campaign against Caring, but the two men have since settled their differences over a glass of wine at Wiltons, which neither of them owns, and the club will be named Rupert's, after Robin's late brother. 'There was a situation with the branding of the club, but that is history. It was a misunderstanding. I have no problem with any of the Goldsmith-Birley group, they are all great people,' he says dogmatically.

Caring's frequent use of the word 'branding' explains both his success in the London club and restaurant scene, and why he is still viewed with suspicion and not a little snobbery by some of the established players. It is impossible to imagine Mark or Robin Birley talking about their clubs as brands.

Caring's mother Sylvia was Jewish, while his father was a GI of Italian descent. He says he got the best of a split Jewish-Catholic upbringing. 'My mother was very aware of being Jewish. I don't think of myself as being Jewish or not Jewish. I personally have never felt anti-Semitism. When I bought the Birley group, there were stories that old members were upset a Jew had bought the clubs. I have never experienced any of those feelings, no one said any of it to me.'

He made his first fortune in the rag trade, taking over the family business from his father, who was more interested in playing golf than expanding his company (a mistake Caring has not made). Caring spotted the potential for making money by producing clothes in China and selling them back in the West. He relocated his young family to Hong Kong in the late 1970s and supplied numerous high street chains, notably those owned by his friend Philip Green, to whom he still talks every day. He got into the restaurant business almost by chance, when he tried to improve the food at Wentworth and realised when talking to Le Caprice and The Ivy that they were covertly for sale, so he bought them from Luke Johnson, along with J Sheekey, in 2005 for £31.5 million (Johnson had himself purchased the restaurants from Chris Corbin and Jeremy King in 1998 for £13 million), and his appetite for the trade grew. He is still very involved in his garment business, spending two days a week analysing currency shifts and supply problems for International Clothing Designs, which provides clothes for Next, BHS and Topshop, as well as Bloomingdale's and Walmart in the US. Many associates believe that it is these long-honed international financial skills that have made his crossover into running a restaurant empire so seamless.

Despite the nice manners, it's easy to see the steel beneath that drove him towards a vast fortune. When I rudely ask him how much he is worth, he declines to say, though helpfully adds that the figure of £700 million - up £100 million on the year - in the latest Sunday Times Rich List is pretty accurate.

He is unabashed in his driving ambition to create the first global restaurant and club chain, and he is one of the very few Brits who has cracked America. The key to this ambitious dream is the expansion is the Soho House 'franchise', of which he owns 80 per cent, with the founder Nick Jones retaining the other fifth. Soho House is now in Hollywood, New York, Berlin - which is doubling in size to 100 rooms - and Miami, which has just opened, with plans for further pushes into Chicago, Mumbai and Istanbul.

'You can't rush in,' says Caring. 'These places take time to bed down. The reason why English restaurateurs have found America to be a graveyard is they don't understand the market, and don't take the time to learn it.' Those who know Caring say that he is exceptionally hands-on in all areas of his business, and has developed a genuine feel for what will work in different cities. Even his most grudging admirers are impressed that he will not be hurried into openings, even if delays are costing the company a lot of money. And Shanghai is the next part of the plan. 'I'm afraid I do believe that one day China will rule the world, I don't think there's any question of that. Look at the growth, the hunger, the work ethic; unbelievable.'