Travellers start leaving Prince Charles village