Retailers are facing mounting pressure to increase safety standards at overseas factories after a garment manufacturing facility in Bangladesh collapsed last week, killing more than 400.

That building collapse is just the latest to illustrate the working conditions for factory employees in Bangladesh, where retailers have flocked, attracted to the cheap labor.

At least four other incidents since 2010 have each killed dozens, if not hundreds, of workers due to factory fires; other incidents have injured hundreds of others, according to Worker Rights Consortium, an independent group that monitors and investigates labor rights and working conditions.

Many of the retailers that rely on these factories are known for their inexpensive apparel, including Walmart, Gap, Dress Barn and H&M. As retailers continually attempt to stay competitive by offering shoppers the lowest prices, they’re turning to countries with little regulation of safety and health standards.

In turn, to compete with each other to offer retailers the cheapest prices, garment manufacturers are ignoring standards that may be in place, and the government doesn’t enforce them, says Scott Nova, executive director of Worker Rights Consortium.

“With no local government regulation, (factory owners) know they can reduce their costs by ignoring safety standards or labor law,” Nova says. “The country is giving Western retailers what they crave, which is low prices. But that comes at a huge cost to workers.”

Several retailers tout their auditing programs or have recently amended them. Last month, Walmart announced a new plan to promote fire safety in Bangladesh by helping fund a fire, health and safety academy there to promote training on workplace safety; it also announced it will conduct fire, building and electrical assessments of its factories.

“We know that continued engagement is critical to ensure that reliable, proactive measures are in place to reduce the chance of factory fires,” says Walmart spokesman Kevin Gardner.

Nova says the announcements come a little late. “It is only in the face of massive public criticism that they’ve chosen to act,” he says. In November, a fire at a factory manufacturing Walmart goods, among other retailers, killed 112 people.

It can be difficult for retailers to take on the responsibility of ensuring worker safety at all the factories they work with, says Peter Burrows, executive director of Fair Factories Clearinghouse, a non-profit that helps companies with supply-chain safety inspections. Apparel retailers don’t necessarily have long-term relationships with the same factories, as they change depending on what needs to be produced, he says. And sometimes companies may be unaware of a factory with safety issues because they use third-party agents to find manufacturers.

“The average supply chain can include dozens of countries, hundreds of primary factories,” Burrows says. “It’s almost an impossible task for any retailer to do.” Retailers often have to decide which countries to focus on based on risk, he says. Fair Factories helps retailers that use the same factories to collaborate on ensuring safety standards.

The consortium and other workers rights groups are advocating for retailers to sign the Bangladesh Fire and Building Safety Agreement, first proposed two years ago by a group of workers rights advocates and Bangladeshi unions. It would require companies to conduct fire and building-safety inspections and issue public reports on the findings, as well as pay for any needed building repairs.

So far only two companies have signed on: PVH, which owns Calvin Klein and Tommy Hilfiger, and German retailer Tchibo. After attempting to negotiate with Gap, the retailer decided not to sign the agreement, Nova says, adding that he hopes last week’s incident will compel more retailers to sign.

Gap instituted its own fire-safety plan in October for its factories in Bangladesh, which includes fire-safety teams in Bangladesh, South Asia and the U.S., according to its website. Gap was unable to comment before this story’s deadline.

The Department of Labor is also drafting a plan for a grant program to help the Bangladesh government improve its enforcement of safety laws and workers rights. The department says advocating for change has to be a combined effort on the part of both retailers and government and other organizations in Bangladesh.

Nova guesses consumers would be willing to pay slightly more for garments manufactured in safe conditions, and that most of them don’t realize the cost to human life for being able to buy cheap clothes.

Kalpona Akter, a former garment factory worker who now serves as executive director of the Bangladesh Center for Worker Solidarity in Dhaka, wants consumers to keep buying clothes made in Bangladesh but also to encourage retailers to take on more responsibility.

“Please buy the clothes made in Bangladesh,” she says, “but at the same time put pressure on all the retailers to pay a fair amount of money for these clothes so our workers can get a decent wage, put pressure onto them to sign the agreement so our workers can get a safe working place. We don’t want to die in these factories.”