Town councillors are lodging an objection to plans to develop a pub site on the Mirfield-Ravensthorpe border.
Shahid Patel wants to redevelop the Swan after failing to attract a retailer to take on the existing building.
The pub in Huddersfield Road would be demolished to make way for a convenience store and two smaller units.
But Mirfield Town Council heard on Tuesday that a number of residents had voiced concern.
Mayor of Mirfield Coun Vivien Lees-Hamilton said: “The Swan is an eyesore, but the proposal for its redevelopment would also be an eyesore and it would be an inappropriate building for that historical end of Mirfield.”
She said a building of concrete, glass and corrugated steel would be out of keeping.
Coun David Pinder agreed, but said the neighbouring garage could not be said to hold any architectural merit whatsoever.
Worries about the impact on congestion in the area were raised by Coun Thomas Hirst, who added that there were already plenty of takeaways in Ravensthorpe.
Coun Alan Mapplebeck said he knew the site had caused problems with parking and traffic before the pub closed in 2010.
An application submitted to Kirklees Council said Mr Patel, the son of Coun Salim Patel (Con, Dews South), won permission to change the site’s use in February 2012.
But two years on, there had been little interest in the existing building from potential tenants.
The town council agreed to object on the grounds that it was out of keeping, detrimental to living and visual amenity, would cause highway safety issues and could lead to littering.
Public consultation ends on March 10.