The Dodd-Frank rewrite of Wall Street rules requires lenders and mortgage bond issuers to keep a 5 percent share of the loans they securitize, unless the mortgages meet a quality threshold that regulators are in the process of defining.

The idea is to curb shoddy lending practices by ensuring mortgage market participants face losses if a loan goes bad.

However, an overly tough standard for so-called qualified residential mortgages, or QRMs, that would be exempt from this risk-retention rule could frustrate efforts to stabilize the fragile U.S. housing market by making it difficult for otherwise sound borrowers to obtain loans.

"This policy will have a long-lasting effect on who gets homes and who gets homes at the best rates," said David Stevens, president and CEO of the Mortgage Bankers Association, which represents the lending industry.

Six regulatory agencies, including the Federal Deposit Insurance Corp and the Securities and Exchange Commission, issued a draft proposal for QRMs in March that would require borrowers to make down payments of at least 20 percent.

Opponents of the proposed definition say it would price credit-worthy first-time buyers out of the market.

The sources said regulators are considering offering an option that would only require a 10 percent down payment if the loan is backed by private mortgage insurance to protect lenders from default.

The critics, who include small banks, consumer groups and real estate brokers, contend the original proposal would keep private capital out of a mortgage market dominated by government-controlled Fannie Mae and Freddie Mac and the Federal Housing Administration, which have more flexible terms. Those three entities own or guarantee about 90 percent of new U.S. home loans.

These groups argue that exempting only loans with high down payments would raise borrower costs and effectively shut out smaller originators. Big banks would be in a better position as they are more likely to be able to afford to keep 5 percent of the loans they originate on their books, letting them issue loans that do not meet the QRM standard.

Stevens said the rule, as currently proposed, would mean a larger government role in mortgage securitization than otherwise. He said private issuers are currently sitting on cash as they wait to see what standard is adopted.

"Without question, there is capital sitting on the sidelines of the housing finance system that could otherwise be deployed to help consumers," he said. "Uncertainty about QRM is clearly a big driver; it won't be helped if the proposed rule goes through."

The Mortgage Bankers Association, the Center for Responsible Lending and the National Community Reinvestment Coalition are among an unlikely block of consumer advocates and minority, housing and real estate industry groups that have petitioned for regulators to find a way to widen the net to include more borrowers.

The Center for Responsible Lending says that based on average home prices, it would take 14 years for the typical American family to save enough money for a 20 percent down payment, and even longer in high-cost areas like California or New York.

But even if the QRM down payment was lowered to 10 percent, one in four home buyers could still be priced out of the market, based on the nearly 25 percent of home buyers in 2010 who put less than 10 percent down, according to CoreLogic.

Many existing home owners could be among those hard hit by a tough QRM standard. Almost 25 million home owners would be unable to refinance into QRM loans because they do not have ample equity in their property.

A final decision is likely to get pushed into next year as the group of regulators defining the standard awaits a decision from the new Consumer Financial Protection Bureau on a rule to curb predatory lending.

"As long as there is no action on the proposal, the market is frozen, with the uncertainties being so profound," said Karen Shaw Petrou, managing partner at Federal Financial Analytics, a Washington regulatory research firm.

"It's already a tough climb with any risk retention, and coming up with the structure will be a determining feature of whether or not you want to get into the business."