LONDON (AP) — The old adage says an Englishman's home is his castle. But a court says Robert Fidler's castle can't be his home.

A judge ruled Monday that the 66-year-old farmer will go to prison if he doesn't demolish a mock-Tudor castle he built without planning permission.

Fidler built the four-bedroom structure — complete with two turrets — on his farm about 20 miles (32 kilometers) south of London in 2000, keeping it hidden behind piles of straw bales and tarpaulin.

Local authorities ordered him to tear it down in 2007, saying it breached planning rules designed to protect the countryside, and Fidler lost several legal challenges against the order.

High Court judge Ian Dove said Fidler would be jailed for three months for "intentional defiance" unless he complied by June 6, 2016.

"This process has gone on now for many years," the judge said. "It is time now for those breaches of planning control to be remedied and that is what will happen. Otherwise there will be serious consequences."

Fidler, who represented himself in court, claimed the local council was out to destroy his life and argued that he couldn't demolish the building because he had discovered endangered bats and newts inside. He left court without speaking after the ruling.