Last week the NSW state government announced plans to demolish the iconic Sirius public-housing building in The Rocks, and replace it with luxury apartments.

Most people in Sydney can instantly picture the Sirius building, even if they don’t know it by name. It’s that wonky set of giant concrete steps you pass as you drive over the Harbour Bridge. For years its most famous feature was a sign in one of the windows reading, “ONE WAY! JESUS”.

Sirius has always attracted attention and controversy. Public opinion has been divided between those who dismiss it as an eyesore (some senior members of the state government who plan to knock it down among them) and those who regard its blocky exterior and strong resemblance to a pile of cement-covered Lego pieces with affection.

Former Archibald Prize winner Del Kathryn Barton called the decision a “cultural tragedy”. Sydney Lord Mayor Clover Moore has for the past several years used her limited power in local government to try to at least help the building’s residents. She has said Sirius was “exactly the kind of building our heritage protections are for”.

But the people who lived in Sirius’s weird, shoebox-like units are just as worthy of attention as the building that houses them – reminders of a Sydney most of us couldn’t imagine.

A couple of years ago, a long-time Sirius resident named Jack taught me a trick from when The Rocks was a raucous, working-class suburb full of sailors, dockworkers and roving street gangs – how to turn a newspaper into a semi-deadly weapon. Open it up to the centrefold, roll it lengthwise, bend the rolled-up tube in half and voilà: a makeshift club that can do serious damage. Jack also shared some dubious advice on the throat-slitting abilities of your standard two-dollar hair comb.

If many Sydneysiders don’t get why a big lump of grey concrete blocks is worth preserving, let alone in such a prized part of the city, it’s worth looking at how old buggers like Jack wound up living in Sirius in the first place.

The beating heart of the building has always been its public-housing tenants, mostly pensioners who grew up in the area. These erstwhile Rocks residents were at the centre of the famous Green Bans movement, a years-long battle in the 1970s between property developers who wanted to build skyscrapers along Circular Quay’s western edge and the local community who opposed them. The historic buildings in The Rocks that were slated for demolition, including precious relics of Sydney's convict beginnings and Cadmans Cottage, Sydney's oldest-surviving building, were preserved instead, and are still there today. Had the Green Bans movement never existed, The Rocks would likely resemble other forgettable, overdeveloped parts of the city such as the CBD or Darling Harbour. The largely working-class locals were moved into Sirius, which has 79 public-housing units, in 1979.

Sirius is a complex designed specifically to enrich the lives of the people who live in it. Unlike many public-housing projects, the design includes courtyard gardens, artworks and common areas to foster a sense of community. Its crooked rows of apartments are deliberately staggered in height and alignment to reduce traffic noise from the bridge, and its layout was designed to cater to the needs of families and elderly residents.

In 2014, as the state government prepared to demolish Sirius, those residents – some of whom had lived in the building since it first opened – were given notice that they were soon to be evicted. The locals resisted the government’s efforts to repossess their homes, forming the Save Our Sirius Foundation and teaming up with another group of public-housing residents in nearby Millers Point. For a while the letters “SOS” were emblazoned on one of the windows alongside the iconic Jesus message.

But according to Save Our Sirius Foundation chair Shaun Carter, nearly all of Sirius’s residents were relocated to public-housing estates across the city last year. Designed to house up to 200 people, the building is now almost empty.

“The government’s tried to offer them accommodation within a reasonable distance, but it’s difficult to find housing that meets their specific needs. There are still six units occupied – those residents have refused to be moved on,” Carter says.

Carter first got involved with the Save Our Sirius Foundation in his professional capacity as the president of the NSW chapter of the Australian Institute of Architecture. But for him, the fight to save Sirius has a personal side, too.

“As a young child going over the Harbour Bridge, I used to look at that building every day, and it’s probably the sole reason I became an architect. It let me imagine that I could build things that made people’s lives better,” Carter says.

It was built during Sydney’s love affair with Brutalist architecture, a style that emphasises utility, accessibility and ethical design over aesthetics. Besides the UTS Tower in Ultimo, Sirius is arguably the most recognisable example of Brutalism in Sydney.

It’s this architectural rarity that led the Heritage Council of NSW, an advisory body to the state Department of the Environment, to unanimously recommend in February that Sirius be heritage listed. Last week, Heritage Council chair Stephen Davies told the Sydney Morning Herald the decision to demolish Sirius is “disappointing”, because the building’s “aesthetic significance” warranted preservation.

That partly explains why it’s not only Australians who are sad Sirius is headed for the chop. SOSBrutalism, a campaign run by Germany’s Deutsches Architekturmuseum (DAM), which is devoted to preserving Brutalist architecture, said in a statement that it is, “very sorry to hear that the fight to save Sirius seems to have come to an end”.

Sirius will probably be replaced by up to 250 luxury apartments – a similar scenario to what’s happened in Millers Point. There, hundreds of public-housing terraces have been sold off by the state government over the past few years. Now the median house price sits at a whopping $2,470,550. Coincidentally, the suburb was named “Sydney’s newest haven of liveability” by Domain earlier this week.

Sirius and the Millers Point terraces were some of the last affordable-housing options in innermost Sydney, and while the remaining residents are determined to fight on, the likelihood of their being allowed to stay in their homes is low. In an age of rapid gentrification, increasingly unaffordable housing and rising inequality, the right of low-income people to live in their own city is more precarious than ever. When people travelling over the bridge are greeted by some bland, glass-and-steel luxury apartment complex in a few years time, those who dislike the building might wish for those ugly, old Lego blocks back.