Carson said he has no experience in running a federal bureaucracy, but would remain a close adviser of Trump and a friend.
Carson, a popular writer and speaker in conservative circles, has been a close adviser to Trump and is a vice chairman of Trump's transition team.
He has been mentioned as a possible secretary of health and human services or education.
Carson's business manager, Armstrong Williams, said Carson has made clear he has no experience in running a federal bureaucracy.
"Dr. Carson doesn't feel like that's the best way for him to serve the president-elect," Williams told Reuters.
He said Carson would remain a close adviser of Trump and a friend. "His life has not prepared him to be a Cabinet secretary," Williams said.