Holiday spending started off with a bang last weekend, but some economists are worried that consumer fears about the fiscal cliff will lead them to tighten their holiday purse strings.

White House economists wrote in a report, “As we approach the holiday season, which accounts for close to one-fifth of industry sales, retailers can't afford the threat of tax increases on middle-class families."

Consumer spending totaled $59.1 billion over the weekend, a 13 percent increase from last year, the National Retail Federation estimates. The NRF forecasts holiday sales will gain 4.1 percent overall this year, CNNMoney reports.

Editor's Note: Startling Proof of the End of America’s Middle Class. Details in the Video

The Thomson Reuters/University of Michigan Surveys of Consumers show concern about the fiscal cliff is increasing.

"While consumers had anticipated a last-minute settlement, some consumers are beginning to doubt whether that will happen before higher tax rates take effect at the start of next year," Richard Curtin, director of the survey tells CNNMoney.

"While a successful resolution before year-end could reverse any declines, it would nonetheless diminish holiday spending."

Retailers, such as Wal-Mart, have expressed worries about the fiscal cliff’s impact too.

Fiscal cliff or not, the strong start to the holiday shopping season doesn’t guarantee a strong showing for the entire period.

“I’ve seen it go both ways,” Beryl Raff, CEO of jewelry-store chain Helzberg Diamonds, tells The Wall Street Journal.

Editor's Note: Startling Proof of the End of America’s Middle Class. Details in the Video