Manhattan U.S. Attorney Preet Bharara said investment firms that repeatedly violate the law should be concerned that they will eventually be held accountable.

"People should be afraid that bad actions they have committed in the past" will catch up with them, Bharara said at the CNBC Institutional Investor Delivering Alpha Conference.

But Bharara refused to bite when asked several times by the CNBC host, Jim Cramer, about the insider trading investigation into hedge fund SAC Capital Advisors and its founder, Steven A. Cohen. Both have been the subjects of federal investigations of insider trading for years.

Even as firms in the $2.25 trillion hedge fund industry put in compliance programs aimed at rooting out malfeasance, Bharara said some companies' compliance programs are clearly just offering lip service. He said his office is as committed as ever to crack down on how some hedge funds may use illegally obtained information to get an edge.

He said that at some point, institutions that have had multiple violations must be held accountable.

The long-running investigation of insider trading at hedge funds and specifically at SAC Capital has been one of the high priorities of Bharara's tenure as U.S. attorney.

Reuters and other news organizations have reported that prosecutors are considering bringing a racketeering charge against SAC and Cohen in light of a number of guilty pleas some former SAC employees have made to using illegally obtained information.

A spokesman for SAC declined to comment.

Bharara, who joked at last year's conference that he has enough subpoenas for everyone in the audience, said on Wednesday that his office is weighing whether to bring actions against firms for violations that have not been made public.

While he said the press can serve an important role, he warned that "armchair prosecutors," including some in the media, are often wrong.

As the investigations move ahead, Bharara also said his office could file charges against a company. Although it would be a "rare use of power," he acknowledged that this is something his office could do.

He made it clear that his office was not responsible for leaks to the media about investigations. If he found one of his prosecutors disclosing information to journalists, he said that would be punished appropriately.