For more than a decade, banks and insurance companies convinced governments and nonprofits that financial engineering would lower interest rates on bonds sold for public projects such as roads, bridges and schools. That failed promise has cost more than $4 billion, according to data compiled by Bloomberg, as hundreds of borrowers from the Bay Area Toll Authority in Oakland, California, to Cornell University in Ithaca, New York, quietly paid Wall Street to end agreements since 2008.

California’s water resources department this year spent $305 million unwinding interest-rate bets that backfired, handing over the money to banks led by New York-based Morgan Stanley. North Carolina paid $59.8 million in August, enough to cover the annual salaries of about 1,400 full-time state employees. Reading, Pennsylvania, which sought protection in the state’s fiscally distressed communities program, got caught on the wrong end of the deals, costing it $21 million, equal to more than a year’s worth of real-estate taxes.

“It was brilliant, and it all blew up on me,” said Brian Mayhew, chief financial officer of the Bay Area Toll Authority, the state agency that gave Ambac Financial Group Inc., the New York-based bond insurer that filed for bankruptcy this week, $105 million to end $1.1 billion of interest-rate agreements. The payments equal more than two months of revenue on seven bridges the authority oversees around San Francisco.

Budget Deficits

The termination payments to Wall Street firms come at the worst possible time. The longest recession since the Great Depression left states facing budget gaps of $72 billion next fiscal year, according to the National Conference of State Legislatures. U.S. cities saw their general fund revenue fall the most since at least 1986 in the budget year that ended June 30, according to the National League of Cities.

Wall Street banks and insurers peddled financial derivatives known as interest-rate swaps to governments and nonprofits that bet they could lower the cost of borrowing. There were as much as $500 billion of the deals done in the $2.8 trillion municipal bond market before the credit crisis, according to a report by Randall Dodd, a senior researcher on the Financial Crisis Inquiry Commission, published by the International Monetary Fund in June.

$4 Billion

Borrowers from New York to California are now paying to get out of agreements. Altogether, they have made more than $4 billion of termination payments to firms including New York- based Citigroup Inc., New York-based JPMorgan Chase & Co. and Charlotte, North Carolina-based Bank of America Corp. since the beginning of 2008, according to a review of hundreds of bond documents and credit-rating reports by Bloomberg News.

In contrast to the subprime crisis, few taxpayers know anything about the cost of untangling municipal swaps. The only disclosure of payments to Wall Street often is buried in documents borrowers have to give investors when they sell bonds.

In many cases, firms getting payments aren’t explicitly identified and government officials often don’t call attention to payments made to cancel contracts. Many of the telephone calls and e-mails from Bloomberg News to dozens of government and nonprofit officials over the last eight months seeking comment on derivative transactions went unanswered.

‘No Reason’

“Money that should be invested in students, classrooms and fixing infrastructure in Pennsylvania is instead lining the pockets of Wall Street,” Jack Wagner, the state’s auditor general, said in a statement in April after calling on lawmakers to ban swaps. “State and local governments must stop gambling with public money,” he said.

In an interest-rate swap, two parties exchange payments on an agreed-upon amount of principal. Most of the swaps Wall Street sold in the municipal market required borrowers to issue long-term securities with interest rates that changed every week or month. The borrowers would then exchange payments, leaving them paying a fixed-rate to a bank or insurance company and receiving a variable rate in return. Sometimes borrowers got lump sums for entering agreements.

The swaps were popular because governments and nonprofits could pay a rate that was lower than what they would otherwise face had they sold conventional fixed-rate securities. The agreements backfired after the credit crisis broke out. While borrowers had to continue selling adjustable-rate securities under the deals, the payments made by Wall Street plunged and no longer were enough to cover the municipalities’ own debt costs.

1990s Design

Banks and insurance companies such as New York-based American International Group Inc. started designing municipal swaps in the 1990s as derivatives trading on Wall Street soared. Derivatives are agreements whose value is derived from stocks, bonds, loans, currencies and commodities, or linked to specific events such as changes in interest rates or the weather. They were blamed in part for causing the global financial panic.

The financial manipulation was a boon for Wall Street. While banks got paid to underwrite municipal bonds for public projects, they were able to generate additional fees if the borrower used a swap with the transactions. Because the contracts were unregulated and privately negotiated, the profits that Wall Street booked were never disclosed.

“The basic idea from the bank’s perspective is just to do a swap because that’s where the money is,” said Andrew Kalotay, head of the debt-management advisory firm Andrew Kalotay Associates Inc. in New York. “Look at all the fees they get.”

Jefferson County

In Alabama, $5.8 billion of swaps Jefferson County used in a sewer-system financing in 2002 and 2003 produced $120.2 million in fees for banks, as much as $100 million more than it should have based on prevailing rates, according to James White, an adviser hired by the U.S. Securities and Exchange Commission. The derivatives, which pushed the home of the city of Birmingham to the brink of bankruptcy, led to a $722 million settlement with JPMorgan in November 2009 after an SEC probe and the conviction of a county commissioner who steered business to bankers in exchange for bribes.

The New Jersey agency that makes college-student loans and grants paid tens of millions of dollars when it canceled derivative agreements with banks led by UBS AG and Citigroup in January.

The deals by the Higher Education Student Assistance Authority date back to April 2001, when the agency was getting ready to sell $190 million of fixed-rate bonds. Paul Wozniak, a UBS investment banker, told a meeting of the authority’s board in Trenton it could borrow more cheaply by using swaps rather than selling conventional tax-exempt bonds, according to minutes and a copy of his presentation obtained by Bloomberg News after a request to state officials.

Cost Covered

All the authority had to do to get the deal from UBS was to sell auction-rate securities, he said. The Zurich-based bank would help cover the cost of that adjustable-rate debt in exchange for annual fixed-rate payments from the authority, he said. The fixed rate was 4.65 percent, about a half percentage point less than the 5.18 percent the state would pay if it sold conventional bonds, he said.

The disadvantages were few, Wozniak told board members. The swap was a contract, so it would have to be footnoted in the authority’s financial statements, he said. The state would have to count on getting periodic payments from UBS over the deal’s life, he said.

“They found the swap agreements extremely complicated,” New Jersey’s former Inspector General Mary Jane Cooper said in a report in May after auditing the authority and interviewing board members who listened to Wozniak’s pitch.

No Help

“The explanations were not particularly helpful,” she said. In the end, they relied on recommendations made to them by management, according to the report.

The 18-member board, which consisted of college administrators and New Jersey government officials and students, approved about $1 billion of the deals over the next five years. The authority started exiting the contracts in January, making $49 million in termination payments, including $23 million to UBS and $17 million to Citigroup.

“Government operates with a very short-term mentality,” said Matt Fabian, a senior analyst at Municipal Market Advisors in Westport, Connecticut. “There isn’t much upside to look long term. They are looking for near-term savings on things.”

AnnMarie Bouse, a spokeswoman for the authority, referred to Cooper’s report, which included written responses from management.

Board-Member Action

“A board member’s decision to rely on the recommendation of management where the underlying transaction remains unclear is a reflection on that particular board member, not necessarily an authority deficiency,” Michael Angulo, the authority’s executive director, wrote in the report.

Wozniak, who is chairman of Las Vegas-based education lender College Loan Corp. and left UBS in 2008, said he doesn’t recall the meeting. “You wouldn’t have done it if you wouldn’t have thought it would save you money,” he said in a telephone interview.

Douglas Morris, a UBS spokesman in New York, wouldn’t comment, nor would Alexander Samuelson, a Citigroup spokesman.

New York Governor George Pataki was seeking ways to close an $11 billion budget deficit in 2003 when he embraced Wall Street’s alchemy. The former governor included a provision in his spending plan that authorized all state agencies to use swaps, resulting in a total of $5.9 billion of the deals with firms such as Goldman Sachs Group Inc., based in New York.

Evaporated Savings

The state sold floating-rate securities to refinance existing fixed-rate bonds and then locked in lower fixed rates on the new debt using swaps. Before the credit crisis, officials said they had generated $203 million of savings. Since the crisis, unwinding the swap contracts has cost $247 million, according to the state budget office.

Pataki didn’t return telephone calls and e-mails to his office at the New York-based law firm Chadbourne & Parke LLP seeking comment. Erik Kriss, a spokesman for the state’s budget department, said the “swap portfolio will continue to show modest savings,” in part because state officials are refinancing existing debt with lower fixed rates.

New York was among about 40 states that passed laws, often at Wall Street’s urging, permitting municipal derivatives before the credit crisis, according to Dodd’s research for the IMF. Tennessee passed rules in 2001 that required borrowers to attend a swap school.

Morgan Keegan Classes

Memphis-based Morgan Keegan Inc., a division of Birmingham, Alabama-based Regions Financial Corp., was selected to teach the classes. The firm sold many of the $12.7 billion of the deals subsequently done by more than 40 counties, municipalities, districts and authorities, according to Justin Wilson, the state comptroller.

“There’s just no reason these entities should be playing with this stuff,” said Christopher Whalen, managing director at the Torrance, California-based research firm Institutional Risk Analytics. “They don’t have the capacity to evaluate these instruments. They are totally lost.”

Just as banks loosened mortgage underwriting standards as part of the effort to create more subprime-linked securities, Wall Street targeted some of the riskiest credits in the municipal market with its swaps pitch. Nonprofit and government- run health-care providers, which pay higher tax-exempt interest rates because they have among the lowest bond ratings, accounted for 40 percent of the derivative deals, Standard & Poor’s found in a study in 2007.

Lucrative Business

The business was so lucrative that banks and insurers were able to write teaser checks to lure borrowers into swaps. The arrangements were akin to Goldman Sachs giving Greece $1 billion in off-balance-sheet funding in 2002 through a currency swap, helping the nation mask budget gaps to meet a European Union debt target.

“Tinkering with debt was something that you could hide behind,” said Jeffrey Waltman, a city councilor in Reading. The city got upfront payments totaling $7.6 million from Wachovia Corp. in 2005 and 2006 for contracts it later terminated.

“Maybe it didn’t mean so much of a tax increase, or maybe it didn’t mean laying off people,” said Waltman. “It was what appeared at the moment to be a painless effort.”

Ferris Morrison, a spokeswoman with San Francisco-based Wells Fargo & Co., which acquired Wachovia in December 2008, didn’t respond to a request for comment.

Other Victims

Reading taxpayers weren’t Pennsylvania’s only swap victims. The school district in Butler, 32 miles (51 kilometers) north of Pittsburgh, got a $730,000 check in 2003 from JPMorgan. It cost officials $5.3 million two years ago to exit the contract, enough to hire 100 new teachers for a school year. In a lawsuit it filed against its adviser and JPMorgan, the district said the bank booked an $890,000 fee from the transaction, which it called excessive.

A New York court last year dismissed the complaint and others alleging securities fraud, ruling that interest-rate swaps were privately negotiated contracts and not securities.

Borrowers in the municipal market primarily sold two types of adjustable-rate debt to do swaps. Auction-rate securities were bonds maturing typically in about 40 years that paid investors a rate that changed every 7, 28 or 35 days at bidding run by banks. Variable-rate demand bonds were similar except they were also often secured by an agreement from a bank to buy the debt if no investors did when rates were periodically reset.

Market Collapse

The $330 billion auction-rate securities market, which dates back to the 1980s, collapsed in February 2008. Investors stopped buying the bonds because much of the debt was backed by bond insurers that were about to lose their AAA ratings after expanding into mortgage-related derivatives. When banks that ran the bidding started permitting auctions to fail, rates paid by borrowers to bondholders were reset in some cases as high as 20 percent.

While auction rates soared, the periodic payments that banks made to borrowers as part of the swaps plunged because they were linked to benchmarks such as U.S. Federal Reserve lending rates, which were slashed to almost zero percent to combat the financial panic.

“That’s the black swan,” said Robert Fuller, a municipal financial adviser at Capital Markets Management LLC in Hopewell, New Jersey. “The things you can’t imagine kill you.”

Hospital Debt

The University of California had to unwind derivatives it used with debt sold for its medical centers, which form the third-biggest U.S. public hospital system. In April 2008, it sold $322 million of fixed-rate bonds to refinance auction-rate securities and pay $6.8 million to JPMorgan, Goldman Sachs and Merrill Lynch & Co., later acquired by Bank of America, to terminate swaps, according to bond documents. The exit fee was enough to cover the annual tuition of 200 students in its public-health program.

The pace of swap cancellations in the municipal market accelerated after Lehman Brothers Holdings Inc. filed for bankruptcy in September 2008. The filing triggered the termination of all the New York-based bank’s derivative contracts, including hundreds with tax-exempt borrowers.

While the market for variable-rate demand bonds didn’t collapse, the cost of the debt increased as banks lifted the fees they charge to serve as buyers of last resort. California’s State Department of Water Resources refinanced almost $4 billion of the securities this year and terminated swaps as its so- called liquidity agreements with banks expired. The agency began borrowing the money in 2002 to buy electricity to help alleviate the state’s energy shortage.

‘Something More Stable’

“They wanted to get out of this variable rate,” said Joe DeAnda, a spokesman for state Treasurer Bill Lockyer, whose office oversaw the water resource department’s refinancing. “They wanted to move into something more stable.”

Municipal borrowers have refinanced or retired about $135 billion of $525 billion of variable-rate demand bonds since 2008, according to a report in September from Christopher Mauro, head of municipal-market strategy at RBC Capital Markets in New York. There is another $101 billion of the securities backed by banks under contracts that expire next year, he said.

In addition to getting termination payments, Wall Street is finding a way to profit from the meltdown by underwriting bonds that borrowers sell as they unravel their swaps. Morgan Stanley, JPMorgan and Bank of America were among firms that got termination money from California’s water resources department this year at the same time they were paid to help the agency sell bonds, according to offering documents.

Halt to Sales

Mary Claire Delaney, a Morgan Stanley spokeswoman, declined to comment, as did Danielle Robinson from Bank of America and JPMorgan’s Justin Perras. JPMorgan in September 2008 said it would stop selling interest-rate swaps to government borrowers.

Even Ivy League universities were caught in the market’s demise. Harvard University paid $497.6 million in December 2008 to end $1.1 billion of interest-rate swaps with JPMorgan and Goldman Sachs, and separately agreed to end another $764 million of the agreements at a cost of $425 million. JPMorgan was the lead banker when the university in Cambridge, Massachusetts, sold bonds whose proceeds were used to make the termination payments.

Future Flexibility

Cornell, one of the eight private colleges and universities in the Ivy League, paid $22.8 million in May to get out of deals with Wall Street firms. The exit fee would cover the annual tuition for 500 students at the university. Unwinding the derivatives gave the university “greater future flexibility” because it was able to replace 50 percent of its variable-rate debt with fixed rates, Joanne DeStefano, chief financial officer, said in an e-mail.

Many borrowers are unwinding swaps because they want to refinance variable-rate debt with municipal fixed rates at historic lows. The savings can offset the cost of termination fees, said Peter Shapiro, managing director of Swap Financial Group in South Orange, New Jersey. The financial engineering also generated savings before the crisis, he said.

Shapiro, 58, the former head of Essex County, New Jersey, who ran for governor as the Democratic nominee in 1985, formed his municipal-swap company in 1997 and may be the biggest industry adviser, with more than 100 government and nonprofit clients, according to his website. There are no formal rankings because the business is all privately negotiated.

Orderly Market

“The swap relied upon an orderly functioning variable-rate market,” said Shapiro, who has advised borrowers such as the California Housing Finance Agency, which has more than $4 billion of the derivatives. “There hasn’t been an orderly functioning variable-rate market for two-and-a-half years.”

Some public officials are trying to prevent a repeat of the swaps meltdown. Tennessee’s comptroller last year tried to ban municipal derivatives outright before pushing through rules that place limits on who can use them. In Pennsylvania, Wagner, the state’s auditor general, last year asked lawmakers to adopt rules to outlaw financial fiddling after investigating school- district deals.

The board of the Delaware River Port Authority voted to ban using swaps last December after losing more than $60 million on the contracts. Pennsylvania’s auditor general is on the board of the authority, which operates four toll bridges and a commuter rail line between Philadelphia and southern New Jersey.

Houston, which still has two swaps linked to about $900 million of its bonds, says it’s done with the derivatives after the promised savings disappeared.

“If you have to create a flow chart to explain how a transaction works,” Annise Parker, the Texas city’s mayor, said in a September interview at Bloomberg’s New York headquarters, “that’s a problem even for a city the size of Houston.”