A third building is to be demolished as part of Dewsbury Regeneration Board’s drive to smarten up the town.
Asco Place, on the junction of High Road and Wakefield Road, was one of 11 sites on a hitlist drawn up in 2010.
Its owner had been in legal wranglings with the council for more than a year after a demolition notice was issued.
But Bradford Crown Court dismissed his appeal on September 13.
It means the mill building must be knocked down and the site cleared.
Regeneration board chairman Coun Paul Kane (Lab, Dews East) said: “I’m hoping there’s some light at the end of the tunnel and we can get this building removed. It’s not nice for the man that’s in there, but we’ve given him every opportunity.
“It’s just an eyesore and it’s synonymous with what’s been happening on routes into Dewsbury. I’m happy for it to be removed.”
A Kirklees Council spokeswoman said the owner had been asked to confirm his plans and give a timetable, but no details had been received so far.
She said: “If no agreement is reached the council could undertake the demolition itself after obtaining all the necessary permissions including planning permission and a wildlife survey.
“The cost of the demolition would be recovered through a land charge on the property when it is sold.”
The council is also taking separate action to recover its legal costs from the landowner.
Coun Kane said the demolition of Asco Place was a significant success for the officers working on the hitlist buildings.
“I think given the circumstances in which these [buildings] have been left to rack and ruin, to do something with nearly 50 per cent of them in four years is quite good. It’s a credit to officers and the time we’ve set aside,” he said.
“I have some doubts as to how far we can get with the others, but we’re not going to take our eyes off the ball.”