The gardener with balls

Diarmuid Gavin may be the bad boy of British gardening but old ladies love him. He can't walk more than five steps without one grabbing his arm, shouting "Diarmuid!" and pressing their body against him while a friend takes a photo.

He is filming for the BBC for much of the afternoon and I follow him, along with the hordes of ladies who jostle each other out of the way for the best view, maybe even the chance to touch him. Some take photos, another dares her friend to touch his shoes (he has taken them off while he walks around a garden). Even the BBC girl whose job it is to marshal him from garden to garden says she has never seen anything like it.

It's easy to see why he's so popular. With his Irish charm, mop of curls and sparkling blue eyes, he is "wonderfully sexy", as I was shocked to overhear one septuagenarian say.

Diarmuid (he's joined the ranks of celebrities known by one name), the 40-year-old TV presenter and celebrity gardener (a tag he hates), has created the most talked-about garden of the Chelsea Flower Show, with its "lollipops" - brightly coloured balls on sticks - plastic chairs and pop-art pod. But, thanks to a well-publicised feud with fellow competitor Bunny Guinness and controversy over whether he should have been accepted at Chelsea because of problems with his sponsorship, he is also its most talked-about gardener.

"I've been very low because of it all," he says. "I could have done without it." The spat with Guinness was about the height of the wall that separated their gardens. It began two weeks ago when Guinness, a six-times Chelsea gold medal winner and presenter of Gardeners' Question Time, said Diarmuid's wall was higher than it was meant to be. Diarmuid accused Guinness of "snobbery, elitism and rudeness". In return, Guinness said he was "a nasty piece of work".

It got worse when Diarmuid accused her of "setting him up" when he was interviewed on Gardeners' Question Time on Radio 4 (the interviewer asked him about the cost of his garden and whether he thought he could "buy" a gold medal). The two still do not speak.

"Chelsea always has the good, the bad and the ugly," he says, presumably referring to Guinness (he refuses to confirm which she is, but I assume he doesn't consider her "the good"). "That's the downside of Chelsea. I was very upset at the time and I think I will be for a long time. I find the rudeness hard to take. I can take criticism, but not rudeness and elitism. That drives me mental."

He has brought me into his garden, squeezing through a three-foot hole in the back, away from the crowds. It has taken nine months of designing and building and a reported £250,000. "I can do pretty gardens on the back of an envelope. You need to do something different. I don't expect people to stick multicoloured lollipops in their gardens but if it means someone will think about doing something different, that would be great. I'm not expecting everyone to like it."

He describes his garden as "Willy Wonka's chocolate factory and an experiment of colour with green. It's who I am. I've noticed over the past few years that there hasn't been a lot of fun at Chelsea and I wanted to introduce some of that."

There are regular shouts of " Diarmuid!" and his team bring brochures and books to sign. "If I go out there myself, it's madness," he says. "The attention can be quite draining. I do very few shows, probably for this reason."

He shrugs off the controversy over his sponsorship. Gardeners must finalise funding in order to qualify as one of the show's 21 big gardens. Diarmuid admitted that he didn't have a sponsor on his own BBC show, and the Royal Horticultural Society (RHS) received a flood of complaints from those turned down.

"I was always honest about sponsorship. They said: 'We just want to know you're going to be able to produce the garden.' We did it, there's no issue with it. Nobody should be complaining about not getting a plot at Chelsea. They can go and get stuffed. We've built a damn good garden. We deserve to be here as much as everyone else."

Diarmuid grew up in surburban Dublin. His father was a personnel manager, his mother looked after the five children. When Diarmuid was six, his younger brother Connor was knocked down by a car when they were walking to school and was killed. He is reluctant to talk about it, except to say that it was "devastating for the family".

He says he was a "disaster academically" but always knew he wanted to be a gardener. After agricultural college, he started a business. He was 29 when he met his future wife, Justine, the daughter of a prominent Irish journalist and Ireland's Lord Chief Justice, while he was working on their garden. He jokes that she ran off with the gardener (they have been married for nine years).

But his style wasn't to everyone's taste. After selling his van and being evicted from various flats for not paying rent, Diarmuid turned up at the RHS 10 years ago and asked to submit a garden. After being turned away by an official saying he had to apply in writing, Diarmuid and two friends scribbled a proposal in a café. Somehow, it was accepted and the Irish lads built their first garden with £300 and some plants they'd dug up in Kerry. They won a bronze medal.

Diarmuid was back the following year. He didn't win a medal, but he was interviewed by Alan Titchmarsh and was so good on camera that he was inundated with offers. His TV career blossomed. Diarmuid and his wife now divide their time between their house in Harlesden, north-west London, and a Dublin flat.

Is his own garden weird and wonderful? "No, it's very simple - a patch of lawn, a small pavilion at the back, herbaceous borders along the side and a bit of decking." Isn't decking horribly passé, I ask? "I don't care about trends. The thing I hate most about British gardening is snobbery and people telling you what gardening ought to be."

Is he disappointed that he only won silver gilt this year? "Medals aren't important. I don't think we deserved gold, some of our planting was sloppy. Christopher Bradley-Hole's prize-winning garden was my favourite." He refuses to say what he thinks of Bunny Guinness's garden.

A friend calls to say he's in London with his new baby and could Diarmuid spare half an hour ... He can't, of course, but he grabs me, makes a dash through the crowd, scribbling his name on a few brochures as he goes, and runs out of the front gate. "It's illegal for me to leave the site," he says with a glint in his eye as he turns his phone off. "But feck it."

"That's Diarmuid Gavin," I hear one woman whisper to her friend. "He's the one with the balls."