have to walk past Daniel Libeskind's latest, screaming, look-at-me abstract extravaganza yards away on the Holloway Road - one of a flock of projects by the architect selected to rebuild Ground Zero - is a rude reminder of another world of architecture to which the single-minded Wilson could never belong.
But this is to deviate. The prime purpose of our encounter is to discuss another matter: his remarkable art collection.
It consists of some 600 works of modern British art which he has announced he will leave to the nation (via the National Art Collection Fund), to be housed at Pallant House, an elegant 1712 house in Chichester, where he and MJ, as his wife is known, are working on a purpose-built space for it.
"Collecting is an obsession. When I left the Navy in 1946, they gave me £35. I immediately spent it on a portrait by Spencer Gore. That was it. I was hooked. At a certain point it's not you who runs the collection. It runs you. It demands its own rights, like how to keep it together. I reached that point long ago."
Wilson considers himself a painter "manqué". Having abandoned his own career as an artist in favour of architecture, he regards the pictures he has purchased as recompense.
Most are by his friends. All are names representative of post-war British art: David Bomberg, Frank Auerbach, Lucian Freud, David Hockney, RB Kitaj, Eduardo Paolozzi among others.
Collectively they present 60 years of art but the strong emphasis is on that new generation associated with the Swinging Sixties, from a Peter Blake image of the clean-cut 1962 Beatles - "Which they disliked because Peter took so long to finish it that they'd moved into a hairier, Sergeant Pepper phase.
The white gaps in each corner are where they refused to sign it" - to Richard Hamilton's iconic Swinging London which recasts a newspaper image of Mick Jagger's 1967 drug-bust in West Wittering.
"We used to meet, either at the ICA - still in Dover Street in those days - or at one of two famous Soho drinking holes where the art world congregated: the French House and the Colony Room [represented in the collection by Michael Andrews's group portrait of key members, painted in 1962].
We were mostly bachelors, there on Saturday mornings-with our little bags of pasta from the Italian delicatessen, taking it home to our bedsits in Primrose Hill. You'd bump into Eduardo, or Hockney or Hamilton - or Francis Bacon, whom I only knew slightly."
Did he manage to buy a Bacon, or were prices already too high? "I've always had to sell in order to buy - and I have what I call a 'ghost collection' stored in my memory of 50 or so pictures I've had to sacrifice. But yes, I did manage to buy one Bacon, a tiny head, exquisite ..."
He breaks off, clearly troubled at the memory. Surely he didn't let that one go? "No. Its fate was far worse. In the mid-Seventies 12 of my pictures were stolen. I gave the police all the details but heard nothing.
Then one day, weeks later, I was at home one lunchtime feeding my young son. The news was on. Suddenly they flashed up an early self-portrait by Lucian Freud and said if anyone knows who this belongs to, please contact the police." The story grows more disturbing.
Another of Wilson's Freuds turned up in a Christie's catalogue in New York. "I had to pay $1,000 to lawyers to prove it belonged to me. Strangest of all was when the police received a mystery phone call instructing them to go to a pillar box in N17. There they found more of my pictures, rolled up.
Eventually I recovered all but three: a Peter Blake, a Tom Phillips and the Bacon. Some of my artist friends were rather put out that their works hadn't been selected
... I would still dearly love to get those missing three back."
Since that theft, key works in the collection have been on loan in the safer conditions of public galleries. At Pallant House they will be reunited for the first time.
"They'll all be on show, rather like a Royal Academy exhibition in the 18th century, walls absolutely covered floor to ceiling, top-lit, with plenty of natural light.
I find very few galleries designed by architects to be of any use to the art they contain." Meaning? Colin St John Wilson looks mischievous. "No names, but one of my acquaintances recently designed a big gallery. At the opening he insisted that there should be no art on the walls! Can you believe it?"
