Notting Hill set whispers spell trouble for Tories