Backlash grows over asylum plans