The Transport Secretary also dismissed people booing the Prime Minister, saying ‘politicians don’t expect to be popular all the time’