Financial and professional services, which rely heavily on access to Europe's single market, employ 2.2 million people.
In Manchester's buzzing finance district of Spinningfields, a worker at an insurance company said he thought it would have "a huge impact".
"Business are going to pull out, there will be a lot of job losses," Mohammed, who declined to give his surname because of the sensitivity of the issue, told AFP.
Financial and professional services, which rely heavily on access to Europe's single market, employ 2.2 million people, or seven percent of Britain's working population, according to TheCityUK, a lobbying group.
Two-thirds of those workers, or 1.5 million, are based outside London.
The northwestern English city of Manchester, for example, is one of Britain's biggest finance hubs.
Fraser Morris, client director at Brown Shipley, a private bank in the city, said the loss of banking "passports" which allow firms to operate freely across the single market could spell trouble.
"If we lose the passporting, it might have a big effect because finance services are such a big area for the whole country," he said.
"We are obviously a little bit nervous about the outcome," he said, adding that he believed negotiations would "last a few years".
Prime Minister Theresa May is set to kickstart the process on Wednesday with a formal notification to the EU of Britain's intention to leave after last year's historic Brexit referendum.
"It's difficult to get a full handle, purely because we do not know the outcome of the negotiations," said Keith Pilbeam, a professor at the Department of Economics at City, University of London.
"If they go well -- which is a big if -- there should not be too many job losses," he said.
But if things go badly, he warned that up to 240,000 jobs could be cut, or around 10 percent of the financial sector workforce.
Back-office jobs, the support roles that tend to be based outside London, were cut drastically after the 2008 global financial crisis -- and also in the wake of rapid technological advances.
More tough times could now be ahead.
"Companies are engaged in contingency planning -- and Brexit is an issue which obviously they need to think about," said Gary Campkin, director of policy and strategy at TheCityUK.
"Whether that actually means that ultimately jobs will go is unclear," he said.
Britain's much-envied status as a centre of global finance will remain unchanged, Campkin said, but he cautioned against too much attention on London.
"Inevitably there's going to be a focus on the jewel in the crown, which is London," Campkin said.
"It's what the country delivers as a whole for the industry that really makes it the world-beating financial centre that it is. We believe that this position will be maintained in the future."
Other key finance hubs include Birmingham in central England; Bristol in the southwest; Leeds, Liverpool and Sheffield in the north; Edinburgh and Glasgow in Scotland; Cardiff in Wales; and Belfast in Northern Ireland.
Back at Spinningfields, demand is still high for office space.
"Only time will tell what impact on growth Brexit will have," said Howard Bernstein, chief executive of Manchester City Council.
But he added that since the referendum, "the market is not showing any lack of confidence".