PM rapped over power of advisers