The exchange had earlier agreed with WhiteHorse and its underwriter to push the IPO out until later in the day. But instead of extending the IPO, a person on the company's market surveillance team canceled it, said the source, who was not cleared to speak publicly about the matter.

"This was a human error, this wasn't a trading system or a computerized glitch," the source said.

Nasdaq spokesman Joe Christinat said the company had no comment on the IPO beyond the emails sent out, one of which said, "All open and pending orders will be canceled back to customers."

The offering consisted of 6.67 million shares and had priced at $15 per share, WhiteHorse said earlier. Whitehorse is a business development company that originates and invests in loans to privately held small-cap companies.

The error brought back memories of the botched Facebook IPO back in May, when massive demand for the social networking giant's offering led to a 45-minute delay in the start of trading in the stock.

Market makers claimed to have lost hundreds of millions of dollars due to the IPO's problems that day.

Nasdaq OMX shares were down 1.2 percent at $23.87, underperforming a 0.4 percent decline on the Nasdaq Composite.