The kid, as Randy Thompson calls the land agent from TransCanada Corp., wanted to talk. He was persistent. So Mr. Thompson arranged a meeting at the family’s Nebraska ranch house.

“He wanted to know if they could do some surveying,” Mr. Thompson recalls. “We told the kid – if you want to waste your time, go ahead and survey it. But I can tell you, we don’t want the damn pipeline.”

TransCanada wanted to build its Keystone XL project through the middle of the Thompsons’ corn field. The family was worried that it would disrupt the farm’s irrigation system. But there was a solution. If TransCanada would move the pipeline an eighth of a mile – 200 metres – the Thompsons could live with that.

“We said, ‘you just run this thing down to the end of our field so it’s not cutting our field in half, and we’ll sign the damn easement,’ ” he says.

TransCanada said no, arguing that the move would require too sharp a bend in the pipe. It threatened expropriation if the family would not sign a deal. Mr. Thompson grew angry. His face, the Stetson-bearing image of the “pissed-off farmer” he calls himself, became the symbol of an opposition that sprung out of the corn fields and spread all the way to the White House. Mr. Thompson would go on to personally meet with some of the most powerful political leaders in the United States to argue against Keystone XL.

But, he says six years later, it didn’t have to be this way – TransCanada could have just moved the pipe route at the time and settled the matter.

It is a common sentiment.

The pipeline industry faces what former U.S. pipeline regulatory official Brigham McCown calls “a decade of activists aggressively targeting pipeline infrastructure.” Keystone XL, whose review has now stretched over 67 months, is a singular example of how badly things can go wrong for the energy industry when those activists dig in – and how costly that resistance can be.

The Keystone XL battle casts a shadow over TransCanada and rival Enbridge Inc. at a time when the companies together are working through some $62-billion in new projects. It is a historic renaissance in pipeline building at a time of unprecedented opposition to what those companies do.

And for Canada, building Keystone XL is of singular importance. The pipeline promises to open a major new outlet for Alberta oil, which has faced difficult and deep swings in prices due to export bottlenecks, and provide support for the continued expansion of the oil sands. For that reason, it has been avidly pursued by the highest levels of both government and industry.

Yet what becomes evident on a drive along the 3,134-kilometre length of Keystone XL is that some of the industry’s pain is self-inflicted. Along the route, many describe TransCanada and its land agents as intransigent, hard-nosed, quick to threaten court-approved expropriation of land and slow to offer reasonable compensation.

In Alberta and Saskatchewan, after a crowd of cattle and wheat farmers gathered to jointly negotiate terms, they succeeded in securing 10 times the money TransCanada first suggested – raising questions over the reasonableness of the initial offer, a question that has echoed elsewhere on the route.

Even those supportive of the pipeline and its benefits have often been left with a bitter taste from dealings with a company that declined to accommodate concerns over pipeline pump stations and worker-housing facilities.

TransCanada has found through polls of its landowners that “we’re doing a good job,” says Andrew Craig, land manager for the Keystone system. He declined, however, to provide more specific approval numbers.

The company says it is, as a general policy, generous with landowners, and has obtained voluntary agreements to access the land on more than 90 per cent of the route. Of course, landowners know that if they don’t sign on their own, the company can force its way onto their land through the courts.

Now, the anger stoked by Keystone XL is propelling a broad set of new demands as landholders feel empowered to push TransCanada for annual payments and changes to legal liability – terms that, if they are met, stand to add new costs and risks to the way pipeline companies operate.

Not that people like Mr. Thompson would care. After all, he was ready to sign and be done with it years ago, had the company drawn a new line on a map 200 metres away.

“They have shot themselves in the foot time and time again because of their total arrogance,” he says.

“Honest to God, if they would have done that when we first started asking them to do it – hell, they would have been pumping oil already.”

A challenge for future pipelines

The sign at the Nebraska state line displays a sunny slogan: “the good life.” Lately, though, the state has become the centre of unrest against Keystone XL. But the image of angry ranchers is in many ways a misleading one. In the 2012 election, at the height of the pipeline debate, more Nebraskans voted for the Republican presidential candidate – unabashed Keystone XL supporter Mitt Romney – than in 2008.

It’s not just Nebraska. Every state – and virtually every county – on the pipeline’s path voted for Mr. Romney.

Those voters are committed pipeline supporters, too. Tim Gravelle, a statistician and former pollster who is principal scientist at Insights Lab in Toronto, matched the Keystone XL route with data from a broad 2012 Pew Research Center survey. He found that the closer people are to the pipeline, the more likely they are to support it. Within the broader American public, 66 per cent think the White House should sign off on the pipeline. Among those 800 to 1,600 kilometres away from the route, 76 per cent back approval. At 160 to 800 kilometres, support rises to 79 per cent. And under 160 kilometres from the route, fully 84 per cent want Keystone XL built.

“It is sort of NIMBY turned on its head,” Mr. Gravelle said. “The people for whom this is in their backyard probably recognize there are economic benefits that are going to accrue to me directly, or indirectly as a result of increased economic activity in my area.”

The numbers add a surprising wrinkle to TransCanada’s Keystone troubles, which have arisen on a route filled with people who are, by nature, overwhelmingly disposed to support its work. Those troubles stemmed in part from its route across the sensitive Sand Hills ecosystem – a route it had to be ordered to amend by the White House, after leaving untouched in the face of ranchers begging for change – and in part from activists living far away and concerned about broader environmental issues.

The company acknowledges that the old way of building pipelines no longer works for an industry facing a barrage of concern about leaks and welds and general safety. “We have to change the way that we approach things,” chief executive officer Russ Girling says in an interview.

But, he says, TransCanada works with some 60,000 landowners, and “we have a great relationship with those people.” The company spends years on twists and turns to find the best path. “Once we get it on the map and we get on the ground, we spend a lot of time on the ground rerouting it,” Mr. Girling says. “I’d say there’s been a couple thousand reroutings of that pipeline along that 1,800-kilometre corridor.”

That much becomes clear on a flyover of the southern leg of Keystone XL, which is already under construction: The pipe zigs and zags to avoid an airport and sloughs. At particularly sensitive areas, like a high school and large river crossings, it leaves no mark at all. In those places, the company uses horizontal boring to open a path for the pipe.

There is no avoiding the fact, though, that any pipeline corridor is going to be made up of people’s pastures and backyards and some will hate it “no matter what you do,” Mr. Girling says.

Still, Nebraska’s opposition underscores the industry’s challenge ahead. “We just secured $16-billion in new projects,” Mr. Girling says. “No illusions in my mind as to how difficult those are going to be to get permitted. And our challenge as a company is to get better at how we approach these things. And it’s getting on the ground talking to the people that are affected and making sure that you meet their needs.”

Playing hardball with ranchers

Daryl Swenson’s cattle graze around gas wells on his farm near the South Saskatchewan River in Alberta. He has worked for natural gas companies. He thinks Keystone XL is a needed export outlet for the oil that powers Alberta’s economy. “Canada needs the oil to be going out of here,” he says.

But when TransCanada called to negotiate access to his land, he was shocked. “The initial offering of money to the landowners was paltry. It was next to nothing,” he says. “So everybody decided to get together.” Mr. Swenson became one of five negotiators for the Alberta Association of Pipeline Landowners (AAPL), a group that dealt with TransCanada on behalf of 60 per cent of those along the new route for the original Keystone pipeline in Alberta and Saskatchewan. They negotiated terms for Keystone XL as well, led by Jim Ness, a man on the Keystone route with an appetite for fighting hard against big companies.

Mr. Ness is no liberal. “I’m a redneck. I like guns. I shoot guns,” he says. But when he was 12, he was beaten up by a bully and “decided that day that I was going to pursue justice the rest of my life,” he says. He has dedicated much of the past decade to fighting what he considers a new kind of bully: the energy industry. He’s even trained for the battle. In 2004, he became a licensed land agent.

“I wanted to know how the system works and find out if there’s any possibility that landowners have some rights. And, of course, we have practically none.” One major problem involves dead pipelines. Once a system is legally abandoned, landowners worry about who will hold liability for the line – particularly if the company that owned it ceases to exist (although in Canada, the National Energy Board is setting up an abandonment fund to help address the issue).

There are other concerns, too. If Keystone XL is built on Mr. Ness’s land, he says a neighbour can’t legally help him harvest crops on the right-of-way without written permission from TransCanada. (The company says that’s only partly true, and that such rules are there to prevent heavy equipment from damaging the pipe.)

“Some of these issues – it’s not about compensation. It’s about risk liability, hazards imposed on your lifestyle,” Mr. Ness says.

With Keystone, the men sought changes. They wanted annual payments and a lightened long-term liability burden. They failed. According to Canadian and Alberta law, companies “are not forced to do anything,” Mr. Ness says – and with the ability to expropriate, he adds, companies don’t have as much incentive to make people happy.

Along much of the Keystone XL route, landowners described fruitless requests for accommodation. The impoverished Faith School District in South Dakota fought for five years – in a battle that ended up in state supreme court – to move a pump station by 1.5 kilometres in hopes of deriving needed new revenues. It failed.

Cody Math, the first American whose house the pipeline will pass, asked TransCanada to shift a northern Montana pump station in hopes of shielding himself from industrial noise that might affect his quiet home on the plains. He lost. The city of Baker, Mont., wanted a new water well to provide for an 800- to 1,000-person worker camp it intends to build there. TransCanada is spending nearly $3.5-million on water and sewer infrastructure upgrades, but declined a new well – it’s rehabilitating an existing one instead – and the city is paying for a new cell in its sewage lagoon, as well as two new police officers to handle the influx of people.

In Alberta, meanwhile, the AAPL negotiators did succeed in securing a TransCanada-funded monitor to spot any environmental or construction problems. And they wrested far more from the company: some 10 times its initial offer. (TransCanada calls that number misleading: Agricultural land values go up over time – in Nebraska, they’re up to $12,000 an acre from $4,500 in 2008 – and changes in crop prices can also raise payments.) But Mr. Swenson called the process “very frustrating.”

“I don’t think of them as evil,” he says. “I think they’re there to make the most profit for their shareholders. And it doesn’t matter if they can get away with paying as little as possible to everybody else.”

Hoping for a better deal

On the road west from Scranton, N.D., Highway 12 crests a hill to a view overlooking a field filled with pipe. Row upon row of green-coated pipe, some stamped “MADE IN CANADA,” lies neatly stacked four high. This is the Gascoyne pipe yard, home to 350 kilometres of Keystone XL – nearly a fifth of the pipeline, in 15,000 different segments. This summer will be the third some of it has sat idle here, a local worker says.

The pipe provides one of the most tangible glimpses of the stakes for TransCanada, which has already spent more than $1.8-billion to buy equipment and prepare to build Keystone XL.

Critics hope they can use that Gascoyne pipe yard – and TransCanada’s eagerness to get going – as leverage for better deals. In Nebraska, those along the new route are seeking terms that stand to alter the relationship between pipeline companies and landowners. They have formed the Nebraska Easement Action Team, or NEAT, which has leaned on the expertise of Domina Law Group, a formidable class-action and personal injury firm in the state. Brian Jorde, the lawyer working with NEAT, figures about 30 per cent of those along the new Nebraska route are members; another 14 per cent aren’t prepared to negotiate at all, while others have shown an interest in signing up if TransCanada gets its permit.

Those willing to talk have serious demands. They want negligence shielding for landowners who accidentally damage the pipeline. They want regular payments – if not every year, then every 10. They want the ability to extract more money if the pipeline easement is ever sold.

“We’ll never get it all, we realize that,” Mr. Jorde says. But the stakes are high – not just for Nebraska, but for the entire pipeline industry. If NEAT can persuade TransCanada to give in on some terms, it will clear the way for others to make similar demands in future expropriation battles. It could change the way pipelines are built. “We would be setting precedent, because they know they don’t have to give these things,” Mr. Jorde says.

As for Mr. Thompson, the Nebraska rancher, in a surprising twist he expects Keystone XL to eventually be built. He believes the White House will impose some sort of carbon regulation – and perhaps demand yet another new route through his state – but eventually clear the way.

“I don’t really have the sense that Obama is going to deny the permit,” he says. And though some have threatened thousand-strong blockades, or even violence if construction begins, once approval is granted the “chances of really stopping it are what? Not very good,” Mr. Thompson says.

Still, he can’t keep from shaking his head at it all. TransCanada, it turned out, eventually moved the pipeline well away from the family ranch amid a broader route change through Nebraska.

“Why didn’t TransCanada come out of the gate and treat people fairly? They would have had a hell of a lot less resistance,” he says. Instead, “they want to screw you into the ground and force it down your throat. And that just doesn’t work.”