Nasdaq OMX on Wednesday agreed to pay $10 million, the largest penalty ever levied against a stock exchange, to settle civil charges stemming from mistakes made during Facebook's initial public offering last year, the U.S. Securities and Exchange Commission said.

The SEC said that Nasdaq's series of "ill-fated decisions" on the day of the IPO led to regulatory violations.

As a result, more than 30,000 Facebook orders remained stuck in Nasdaq's system for more than two hours when they should have been either executed or canceled, leaving investors in the lurch and causing market makers to lose an estimated $500 million.

Editor’s Note: Put the World’s Top Financial Minds to Work for You

"This action against NASDAQ tells the tale of how poorly designed systems and hasty decision-making not only disrupted one of the largest IPOs in history, but produced serious and pervasive violations of fundamental rules governing our markets," said George Canellos, co-director of the SEC's enforcement division.

The exchange operator agreed to settle the charges without admitting or denying the allegations.

The exchange has also separately agreed to pay as much as $62 million to compensate market makers for losses in a plan approved by the SEC earlier this year.

In an open letter issued on Wednesday, Nasdaq CEO Robert Greifeld said the challenges that the exchange faced when the Facebook stock debuted "were unprecedented."

"In the last year, we have carefully reviewed these events," Greifeld wrote. "As market leaders, we view our experiences as opportunities to learn and improve."

Facebook's initial public offering, the largest ever in terms of volume, was a much anticipated event last year.

But on May 18, 2012, the hype surrounding the IPO was soon replaced with panic after a software error at Nasdaq led to a 30-minute delay.

The SEC said that Nasdaq's senior management thought they had fixed the systems problem after removing a few lines of computer code and decided not to delay the start of secondary market trading.

However, the exchange did not truly understand the extent of the problems, the SEC said.

As a result, more than 30,000 orders became stuck in the system, with many brokers waiting to hear about the status of their orders for over two hours.

In addition to charges stemming from poor decision-making, the SEC also said it was charging Nasdaq with a series of technical rule violations.

Nasdaq assumed a short position of more than 3 million shares of Facebook in an unauthorized error account and covered that short position for a $10.8 million profit, two violations of exchange rules, the SEC said.

Moreover, the SEC said it found other problems unrelated to Facebook, based on a failure by the exchange to monitor and enforce compliance with price-test restrictions in October 2011 and August 2012.

Editor’s Note: Put the World’s Top Financial Minds to Work for You