She has accused him and his allies of acting unconstitutionally in their standoff against the opposition in recent months of deadly anti-government protests.
Last week, she filed a challenge against his effort to rewrite the constitution, branding it undemocratic.
The court dismissed the appeal on Monday.
Ortega said members of her family had received threatening telephone calls, and had been harassed and pursued.
"I hold the executive responsible for any injury or attack that my family might suffer," she said in an interview with Union Radio.
"This is a matter that must be resolved with me, not with my family," she said.
"They are being pursued by patrols that appear to be from SEBIN," the state intelligence service, she added.
"They are sending them messages directly from SEBIN, which answers to the government."
Ortega, 59, said she herself had not received threats -- but some government officials have said on television that she should be imprisoned.
Maduro is accused of controlling the Supreme Court, which has fended off numerous legal and legislative moves against him over the past year and a half.
Clashes at daily protests by demonstrators calling for Maduro to quit have left 66 people dead since April 1, prosecutors say.
Protesters blame Maduro for an economic crisis that has caused desperate shortages of food and medicine in the oil-rich country.
Maduro says the crisis is a US-backed conspiracy.
He has launched moves to set up an elected assembly to reform the constitution in response to the protests, but his opponents say that is a ploy to cling to power.
A survey by pollster Datanalisis indicated that 85 percent of Venezuelans opposed that plan.
The president retains the public backing of the military.
Analysts said last week that Ortega's suit could build bridges between the opposition and disgruntled officials and widen divisions in Maduro's camp, making it harder for him to stay in power.
But the court on Monday rejected her appeal as "incompetent."
That ruling "removes any doubt about the absence of judicial remedies" for the political crisis, said constitutional law expert Jose Ignacio Hernandez.
"It is a clear attempt to discredit the attorney general."
Ortega responded to the ruling by upping the ante -- and the political tension.
She presented a further legal challenge aiming to fire 13 of the court's judges, who she argued were named without her approval.
Her motion challenges a controversial decision in 2015 to name the judges, whom the opposition says are biased in favor of Maduro.
The president is resisting calls for elections to replace him, vowing to continue the "socialist revolution" of his late predecessor Hugo Chavez.
Opponents of Maduro had gone to the court earlier to try to add their names to the list of plaintiffs in Ortega's lawsuit, but were kept away by military police.
Anti- and pro-government activists exchanged blows outside the court in the latest in more than two months of street unrest.
"They do not want the people to demonstrate against the constitutional assembly. Look at how many people reject it," said one young demonstrator, Maria Rodriguez.
"Get away, the streets belong to the people, not to the bourgeoisie," yelled a rival supporter dressed in the traditional red of Chavez supporters and holding a copy of the constitution in his hand.
"What there is here is revolution."