Insiders close to Mr Staley said he was surprised by City regulators' findings.
They have been investigating if Mr Staley's links with the dead financier were closer than first thought.
Barclays said it had been made aware of the conclusions of the probe and "Mr Staley's intention to contest them".
At the heart of Mr Staley's departure is apparently a perceived inconsistency between his account to his own board of his relationship with Epstein and evidence seen by the regulators.
Mr Staley insists that while Epstein was an important client of JP Morgan, where Mr Staley worked for a number of years and as such they were in contact regularly, their dealings were well within the grounds that could be described as professional.
The BBC understands the regulator took the view that the volume and tone of the emails between the two suggested a closer relationship than the purely professional. But Barclays has stated that the report makes no findings that Mr Staley saw, or was aware of, any of Epstein's crimes.
In fairness to Mr Staley, he is not the only boss of Barclays to have fallen from grace in recent times.
In fact, he is the bank's fifth chief executive in seven years - following in the footsteps of men such as Bob Diamond, who resigned over an interest rate-rigging scandal, and Antony Jenkins, who was sacked after falling out with the board over the bank's cost-cutting and profitability.
During the pandemic, he spoke out against the culture of working from home, saying it was "not sustainable".
At a virtual meeting of the World Economic Forum, he said: "It will increasingly be a challenge to maintain the culture and collaboration that these large financial institutions seek to have and should have."
Now Barclays faces the task of maintaining its culture without him - and he may prove a hard act to follow.
Regulators began investigating the Barclays boss after getting a cache of emails between the men from Mr Staley's former employer.
Before joining Barclays the married father-of-two was an executive at US bank JP Morgan, where Epstein, who took his own life in 2019 while in prison, was already a client.
Over many years, Epstein, who was convicted of trafficking a minor for prostitution in 2008 and served 13 months in custody, cultivated contact with the rich and powerful, including former US presidents Bill Clinton and Donald Trump.
Microsoft billionaire Bill Gates has spoken of his "huge mistake" in spending time with Epstein, while the equally super-rich Leon Black stood down from US private equity firm Apollo Global Management over his links.
Although Mr Staley, 63, has characterised his relationship with Epstein as professional, with contact starting to "taper off" from about 2013, regulators wanted to know whether the emails pointed to a friendlier connection.
Mr Staley had already admitted he maintained contact with Epstein for about seven years after his 2008 conviction. And it is also known Mr Staley visited Little St James, a retreat owned by Epstein in the US Virgin Islands in 2015, months before taking the top job at Barclays.
The relationship between a US high net worth individual and his private banker is not like your average relationship with the bank manager. In fact, it would be quite odd if Mr Staley had not visited Epstein at home or indeed vice versa.
Nevertheless, the board of Barclays clearly felt that the perceived disconnect between the accounts of the relationship would mean an extended and distracting wrangle with the regulator.
Having accepted Mr Staley's promise that he had been totally transparent about his relationship with Jeffrey Epstein, the board described itself as "disappointed", but accepted that it would not be possible for Mr Staley to run the bank while being in open dispute with the regulator.
This is not the first time the board of Barclays has been blindsided by Mr Staley.
In 2018, he was fined £642,000 and Barclays £15m after Mr Staley took it upon himself to instruct the bank's own security unit to track down a whistleblower who had raised concerns about the suitability of the appointment of a former colleague of the chief executive. That could have been a straight red card but he escaped with a yellow.
The board will also be disappointed because Mr Staley's strategy for the bank seemed to be paying off. He saw off an activist shareholder challenge to split the investment and retail banks, the investment bank had started delivering good returns and the profits were beating analyst estimates.
But ultimately, boards don't like surprises - particularly ones that are perceived to tarnish a reputation and culture they have spent many years trying to rehabilitate.
Mr Staley said last year: "Obviously I thought I knew him well and I didn't. For sure, with hindsight with what we know now, I deeply regret having any relationship with Jeffrey."
He has yet to comment publicly on Monday's news, but Reuters reported it had seen an internal memo from him indicating he did not want plans to contest the investigation's findings to be a distraction.
"Although I will not be with you for the next chapter of Barclays' story, know that I will be cheering your success from the sidelines," the memo reportedly said.
Barclays said in a announcement on Monday that it was "made aware on Friday evening of the preliminary conclusions from the FCA and the PRA of their investigation into Mr Staley's characterisation to Barclays of his relationship with the late Mr Jeffrey Epstein and the subsequent description of that relationship in Barclays' response to the FCA".
"In view of those conclusions, and Mr Staley's intention to contest them, the board and Mr Staley have agreed that he will step down.
"It should be noted that the investigation makes no findings that Mr Staley saw, or was aware of, any of Epstein's alleged crimes, which was the central question underpinning Barclays' support for Mr Staley following the arrest of Mr Epstein in the summer of 2019."
Barclays praised Mr Staley's success at the bank, but added that "the regulatory process had still to run its full course and it was not appropriate for Barclays to comment further on the preliminary conclusions".
The Financial Conduct Authority (FCA) and the Bank of England's Prudential Regulation Authority (PRA) announced an investigation last year into Mr Staley's "characterisation to the company of his relationship" with Epstein.
Barclays said last year that following Epstein's death in 2019, Mr Staley had "volunteered and gave to certain executives, and the chairman, an explanation of his relationship with Mr Epstein".
The probe's conclusions are yet to be made public, but Hargreaves Lansdown's senior investment analyst Susannah Streeter said it seems clear they are critical of Mr Staley.
"Although detail is limited, it appears regulators believe there was a distinct lack of transparency over this relationship," she said. "It's understood Mr Staley will contest the conclusions, and clearly the board want to distance Barclays from what could be a long drawn out process."
Mr Staley has faced regulatory scrutiny before during his six years at Barclays.
In 2018, he was fined £642,430 by the FCA and PRA for trying to unmask a whistleblower who had raised concerns about a senior executive appointment at the bank's US operation. Many commentators thought the chief executive was lucky to stay in his job.
Under the terms of his contract with Barclays, Mr Staley will receive 12 months' pay, totalling £2.4m, as well as his pension allowance of £120,000 for the year, and any other benefits.
He will be replaced as chief executive by Mr C.S. Venkatakrishnan, known as Venkat, subject to regulatory approval.
The board said Venkat had been identified as its preferred candidate for the role over a year ago.
Reuters said it had seen another internal memo, from Venkat, telling staff Mr Staley's strategy at Barclays was the "right one", although he added he would announce changes to the organisation in the coming days.
In a statement, the FCA and PRA said they "do not comment on ongoing investigations or regulatory proceedings beyond confirming the regulatory actions as detailed in the firm's announcement".