This has much to do with the economic debate although there are aspects of social prejudice too. It is another key area that will decide the June 2016 referendum.
Ideally people would travel and work wherever they wished in the world but there are practical constraints. These include pressures on schools, hospitals, houses, green spaces, etc. as the UK population grows. Then there are fears of imported terrorism and cultural integration. Many people worry that their jobs and incomes are threatened by immigrants from poorer countries and this has increased since the EU expanded into Eastern Europe. The Dutch people have shown their concern in a referendum on Ukraine’s prospects of joining – though they may be ignored again, as they were over the EU constitution (now the Lisbon Treaty).
The free movement of labour is a sacrosanct component of the Single Market but it is hard to see why EU citizens should be preferred over others with skills we need, or with family ties or with close cultural affinity, unless it is to further the single state. We see no other good reason why a nation cannot choose an immigration strategy to suit its own circumstances. Opinion polls show that large majorities in France, Italy and Germany feel the same.