OTSUCHI, Japan — The spot where the town’s center once stood is now a dusty construction site filled with diggers and dump trucks toiling amid huge, man-made mesas of earth and gravel. The work is part of an $850 million project to elevate the land by seven feet and shield it behind a towering 48-foot wall.

Four years after a colossal tsunami swept away most of this remote fishing community on Japan’s mountainous northeastern coast, Otsuchi is starting to rebuild.

However, the wait is far from over for thousands of the town’s survivors, many of them still living in temporary apartments after being left homeless by the waves. Otsuchi was so severely crippled by the calamity — 1,284 people died here, including the mayor and many town hall employees, firefighters and police officers — that the town struggled for years even to put together a recovery plan. Reconstruction began only last year and will not be finished until at least 2019, the new mayor says.

Similar stories could be heard across Japan’s tsunami-struck northeast as the nation held prayer ceremonies this week to observe the anniversary of the magnitude 9.0 earthquake and resulting tsunami on March 11, 2011, that left 18,490 people dead or missing. Almost 250,000 people lost their homes in the calamity, and 87,000 still live in cramped, prefabricated housing that was originally meant to last for just two or three years.

It is not clear when, if ever, they will move back. In Fukushima, where the tsunami caused meltdowns that destroyed the Fukushima Daiichi nuclear plant, some areas were so contaminated with radiation that they may not be habitable for decades.

In small coastal communities farther north like Otsuchi, far enough away to escape most of the nuclear fallout, many survivors have simply given up and moved elsewhere, accelerating the depopulation of rural areas in this rapidly graying nation. Those who want to stay worry they could face additional waits as memories of the tragedy fade in the rest of Japan, where attention is now turning to events like the 2020 Olympics in Tokyo.

“Everyone seems to think that life has gone back to normal here, but we are still very much a disaster zone,” said Hiromi Kawaguchi, 66, a retired town employee who lives alone in a tiny two-room apartment in refugee housing after losing his wife, mother and 4-year-old grandson, Shoya, to the tsunami.

“Does this mean more delays if the nation has lost its sense of urgency about us?” he said. “Even big construction companies are starting to leave to get a piece of the Olympics.”

To help in the rebuilding, the central government in Tokyo pledged 25 trillion yen, or about $206 billion, to pay for reconstruction and radiation cleanup as part of a “concentrated recovery program” that was supposed to end in 2016. However, local governments have been so overwhelmed by the task of rebuilding that many have fallen behind schedule and proved unable to even spend all the recovery money made available to them.

Otsuchi is a case in point. Once a quiet community of 15,200 residents on a picturesque bay between rugged mountains and the Pacific Ocean, Otsuchi was struck by seething 50-foot waves that destroyed more than 80 percent of the town’s structures, including the town hall, fire department, police station and main hospital. The tsunami killed the mayor and almost 50 town employees, leaving Otsuchi leaderless in the months after the disaster.

“The town was paralyzed by the chaos,” Otsuchi’s new mayor, Yutaka Ikarigawa, said in an interview in the temporary town hall, which occupies a former elementary school that was damaged in the disaster.

Survivors said it took a year to erect temporary housing and supply all the units with electricity and water; the cleanup of a half-million tons of crumpled cars, shattered wooden beams and other debris was completed only last year.

Survivors also struggled to reach a consensus on what they wanted their reconstructed town to look like. Some favored the huge, expensive wave walls that officials in Tokyo urged them to build. Others pointed out that such walls had failed to save residents in other towns. They argued that the safest thing to do would be completely rebuild the town on higher ground.

In the end, the town settled on a compromise in which commercial structures like factories and stores would be rebuilt on the site of the old town center, which would be elevated and protected behind a wall as wide as half a football field at its base. Most residents will move to new housing at higher elevations, including on flattened hilltops.

Today, the neighborhoods that had been left in ruins are being covered by thick layers of fresh soil. The three-story concrete town hall, its insides gutted by the tsunami that almost completely submerged it, is the only building left standing in the town center. Buddhist statues have been placed in front, turning it into a memorial for those who perished inside.

But the start of construction has brought new delays. Otsuchi has struggled to find construction companies to even bid on its contracts, as a boom in post-tsunami rebuilding has created a shortage of contractors. That shortage has been made worse by Olympic-related construction projects in Tokyo, said the mayor, Mr. Ikarigawa.

As a result, Otsuchi has been unable to spend all the money allocated to its recovery by the central government. In 2012, Otsuchi was able to spend just 28 percent of the $178 million made available to it. Last year, with a new reconstruction plan finally in hand, it did better, spending 62 percent, Mr. Ikarigawa said.

“It doesn’t make sense to have to return unused recovery funds when so much of the town still needs rebuilding,” said Keiichi Sasaki, 53, the head of a neighborhood committee in Otsuchi whose home was washed away by the tsunami.

Until their homes can be rebuilt, about 3,700 residents live in temporary housing, waiting. Thousands of others have already given up: The town hall estimates that Otsuchi has lost at least a quarter of its population to the disaster and the exodus that followed.

Mr. Kawaguchi, the retired town employee, said the number who left may be even higher. He said many lost hope after the construction boom started to drive up the cost of labor and building materials, making it more than twice as expensive to rebuild now as just two years ago.

Nor has the central government provided much relief: It offers subsidies of $40,000 to $60,000 to help rebuild homes that now cost $300,000 to $450,000 to rebuild.

“The country has no problem putting huge sums into big public works projects like wave walls, but it won’t help average people,” Mr. Kawaguchi said. “The longer it takes to recover, the more our town will wither away.”