Universities minister Sam Gyimah is the seventh minister to quit since Theresa May presented her Brexit agreement two weeks ago.
The East Surrey MP, 42, resignation is another blow to the prime minister hopes of getting the deal through the House of Commons.
It follows the resignation of Brexit minister Dominic Raab earlier this month, after his predecessor David Davis left to rally against the Chequers agreement.
On Facebook, Mr Gyimah said he could not support an agreement which would hand sovereignty to Brussels, leaving Britain poorer, less secure and weaker in the pursuit of our national interests.
Mr Gyimah said that if MPs were to support the agreement it would set ourselves up for failure by surrendering our voice, our vote and our veto.
With a number of Tory MPs now publicly opposed to the deal, his departure highlighted the scale of the task the May faces if she is to win the crunch vote in the Commons on December 11.
Mr Gyimah backed remain during the referendum.
He added in his article: Britain will end up worse off, transformed from rule makers into rule takers. It is a democratic deficit and a loss of sovereignty the public will rightly never accept.
It has become increasingly clear to me that the proposed deal is not in the British national interest, and that to vote for this deal is to set ourselves up for failure. We will be losing, not taking control of our national destiny.
Downing Street confirmed that his letter had been received and said there would be a response in due course.
Mr Gyimahs announcement came as Mrs May disclosed she had given up efforts to gain access to the EUs Galileo satellite navigation system for defence and critical national infrastructure purposes, after being frozen out by Brussels because of Brexit.
She confirmed the UK would instead aim to build its own Global Navigation Satellite System, at a cost estimated by independent experts at £3-5 billion.
She said: Given the Commissions decision to bar the UK from being fully involved in developing all aspects of Galileo, it is only right that we find alternatives.
Mr Gyimah was added to the Conservative party A-List and selected as the prospective parliamentary candidate for East Surrey in 2010 following the retirement of Peter Ainsworth.
He took the seat with a 17,000 vote majority and had only been in Westminster for two years when he was made parliamentary private secretary to then Prime Minister David Cameron.
But he pushed for a bigger role in Government, becoming a whip in 2013 and childcare minister in 2014.
Further promotions to prisons minister and his most-recent role as universities minister followed.