Stick with Howard or risk split, says Letwin