Attorney General Eric Holder testified under oath two weeks ago and said he was not aware of any potential prosecution, although he did know about an investigation that targeted Fox News reporter James Rosen.

Yesterday Top Republicans on the House Judiciary Committee openly challenged him about his testimony. In a letter to Holder, they asked questions about the department's dealings with the press and say the Fox News case contradicts his testimony.

"It is imperative that the committee, the Congress, and the American people be provided a full and accurate account of your involvement," Fox News has reported.

On May 15, Holder said the potential prosecution of the press for the disclosure of material is not something he was involved in or knew about. Then, the Justice Department obtained access to the emails of the Fox News.

In their letter to Holder, Committee Chairman Rep. Bob Goodlatte (R-Va.) and Rep. James Sensenbrenner, Jr. (R-Wis.) did not accuse Holder of committing perjury, but reminded him he was "under oath."

Also in the letter, they wrote: "The media reports and statements issued by the Department regarding the search warrants for Mr. Rosen's emails appear to be at odds with your sworn testimony before the Committee."

They asked Holder how he could claim to have never heard of the potential prosecution of the press and then personally approve the search warrant request.

The top Democrat on the Judiciary Committee, Michigan Rep. John Conyers, said Tuesday he thinks Holder "was forthright and did not mislead the Committee."

As more is uncovered in the scandal, Democrats along with Republicans are joining together in questioning if Attorney General Holder is the right man to lead the Department of Justice.

Jonathan Turley, an attorney and law professor at George Washington University, said in his USA Today column:

In the end, Holder was the best witness against his continuing in office. His insistence that he did nothing was a telling moment. The attorney general has done little in his tenure to protect civil liberties or the free press. Rather, Holder has supervised a comprehensive erosion of privacy rights, press freedom and due process.

But in the wake of the reporter records scandal, Democrats are starting to join with Republicans in questioning whether Holder continues to be the right man to lead the Department of Justice in President Obama's second term.

Turley referenced a recent call by the Republican National Committee chairman for Holder's resignation. "Unlike the head of the RNC, I am neither a Republican nor conservative, and I believe Holder should be fired."

A Justice Department official said today that Holder will hold meetings with several Washington bureau chiefs of national news organizations over the next two days.

Since Holder was involved in the surveillance involving Fox News in the first place, Turley wrote, "Such an inquiry offers no reason to trust its conclusions."

Watch Eric Holder's Reaction After Recorded Phone Call Is Played