Volkswagen’s rigging of emissions tests for diesel cars comes after nearly 20 years of the technology being incentivised in Europe in the knowledge that its adoption would reduce global warming emissions but lead to thousands of extra deaths from increased levels of toxic gases.

Diesel was a niche market in Europe until the mid-1990s, making up less than 10% of the car fleet. Diesels produce 15% less CO2 than petrol, but emit four times more nitrogen dioxide pollution (NO2) and 22 times more particulates - the tiny particles that penetrate the lungs, brain and heart.

Following the signing of the Kyoto protocol climate change agreement in 1997, most rich countries were legally obliged to reduce CO2 emissions by an average of 8% over 15 years.

Japanese and American car makers backed research into hybrid and electric cars, but the European commission was lobbied strongly by big German car makers BMW, Volkswagen and Daimler, to incentivise diesel. A switch to diesel was said by the industry to be a cheap and fast way to reduce the carbon emissions that drive climate change.

The subsequent EC 1998 Acea agreement with all European car makers was backed by then EU transport commissioner Neil Kinnock and UK environment secretary John Prescott. It committed passenger car-makers to reduce CO2 emissions by 25% over 10 years.

“It was practically an order to switch to diesel. The European car fleet was transformed from being almost entirely petrol to predominantly diesel. Britain, along with Germany, France and Italy, offered subsidies and sweeteners to persuade car makers and the public to buy diesel,” said Simon Birkett, director of the Clean Air London group.

The European auto industry ramped up diesel engine production. Under EU pressure, governments kept the diesel price below that of petrol. In the UK, the amount motorists paid in vehicle excise duty was linked to cars’ CO2 emissions, effectively incentivising people to buy diesels.

“Diesel cars should attract less vehicle tax than their petrol equivalents because of their better CO2 performance,” said then chancellor Gordon Brown in 1998.

The results were dramatic, says the Society of motor manufacturers and traders (SMMT). From being a quirky choice, diesel went mainstream in Europe. Its market share in the UK rose from under 10% in 1995 to over 50% in 2012. Britain now has 11.8m diesel cars in use, making up one of the greatest diesel car fleets in the world.

But the trade-off between reducing climate emissions and increasing health problems was not widely debated, say civil servants and politicians. In addition, they say, carmakers found it easy to cheat the system.

“Diesel was seen as a good thing because it produces less CO2, so we gave people incentives to buy diesel cars,” said Martin Williams, professor of air quality research at King’s College London since 2010, and former head of the government’s air quality science unit.

“The [emission] tests were simply not stringent enough. They were devised by a UN committee based in Geneva called the World Forum for Harmonisation of Vehicle Regulations, which was dominated by people from the car industry.

“What’s more, it was easy for some manufacturers to calibrate cars’ computers to spot when the car was being tested and reduce emissions until the test was over,” he told the Sunday Times in July.

David Fisk, chief scientist and policy director in the department of environment and transport in the 1990s, told the Guardian that there had been “concern” in government when it was proposed that diesel be backed over petrol.

“The motor manufacturers made a démarche to the department of the environment, showing that a major switch to diesels would lead to a substantial drop in CO2 emissions,” Fisk said. He added that the air quality division in the then-Department of the Environment, Transport and the Regions (Detr) saw there was a problem he said – that local air pollution would increase as a result – so a national system of air quality standards was created.

Former Greenpeace UK chief Stephen Tindale, who was working on air pollution in the department of the environment at the time, remembers a battle between environment and air pollution divisions.

“Air quality was just not on the political agenda in the late 1990s. In 1997, Labour’s policy was for a 20% cut in CO2 emissions by 2020. Tony Blair wasn’t obsessed with climate but he saw the political and soundbite advantage. The department of health was dozy, with Frank Dobson and then David Blunkett in charge.”

A senior civil servant, now retired, who worked in the department for transport but asked not to be named, said that cost-benefit studies of a switch to diesel were done by government but climate change was “the new kid on the block” and long-term projections of comparative technologies were not perfect.

“We did not sleepwalk into this. To be totally reductionist, you are talking about killing people today rather than saving lives tomorrow. Occasionally, we had to say we were living in a different political world and everyone had to swallow hard,” he said.

But Michael Meacher, environment minister 1997-2003, said he could not recall ever being consulted on the UK’s switch to diesel. “If it had been a question of a trade-off between carbon emissions and health, much as I would have wanted to reduce climate emissions, the medical effects would have trumped it. I would have been keen to see a major reduction in [carbon] emissions, but I would not have wanted that to be at the price of lives.”

As new research shows that diesel fumes are worse than expected for health, triggering cancers, heart attacks and the stunting of children’s growth, many politicians have admitted a major environmental mistake.

Shadow environment minister Barry Gardiner told C4 last year that that it was the “wrong decision” to incentivise diesel. “Hands up — there’s absolutely no question that the decision we took was the wrong decision. But at that time we didn’t have the evidence that subsequently we did have.”