The latest example of these so-called nontraditional lenders: A venture of Prudential Real Estate Investors and Dutch pension-fund manager Algemene Pensioen Groep has raised a €265 million ($364 million) fund to make junior loans and preferred-equity investments, primarily in the Netherlands.

Like other financiers in this fledgling category, the Pramerica Real Estate PRU -0.07% Capital V fund will try to fill a gap left by banks that have been reducing their exposure to commercial property.

"The Netherlands, in a way, is on the same position where London was two years ago," said Robert-Jan Foortse, head of European property investments at Algemene Pensioen Groep Asset Management. "It is a market where there is limited debt funding available, and the values of properties have slightly corrected."

Nontraditional lenders also include insurance companies and private-equity firms that are increasing their lending throughout Europe

From the first quarter of 2012 to the same period in 2014, the proportion of traditional banking sources lending to commercial-real-estate borrowers fell from 67% to 55%, according to a recent report by Cushman & Wakefield. Their appetite for commercial-real-estate loans has faded partly due to increased regulations in the wake of the downturn.

"This is providing nonbank lenders the opportunity to bridge the gap between the required by borrowers and what is being provided by the banks," said Rawle Howard, a director of BlackRock Inc.'s real-estate-debt team, which also is targeting the Netherlands.

Meanwhile, in the past year, the number of debt funds and private-equity lenders in Europe has increased by 29%, the Cushman & Wakefield report says.

Currently, at least six European so-called mezzanine-debt funds are in the market trying to raise about €1.7 billion, according to alternative-assets researcher Preqin. Mezzanine loans typically are a more risky type of debt that occupies a position in the capital structure between senior debt and equity. If all the funds meet their targets, 2014 could be a record year for mezzanine-debt fundraising in Europe.

Investors have been focusing on the Netherlands in part because prices have been bid up so high in the most desirable markets. In cities like London, Paris and Frankfurt, top office-building values are so high buyers are accepting average annual returns of about 4.5%, while in Amsterdam yields for top office buildings are around 5.5%, according to Machiel Wolters, director of research at CBRE Group Inc. CBG +0.94%

Meanwhile, rent and occupancy trends are more favorable in the Netherlands than they are in other markets investors are turning to, like Italy, Spain and Portugal, experts say.

In 2013, deal volume for commercial real-estate transactions in the Netherlands reached €5.3 billion, compared with €4 billion in 2012, according to CBRE. Deka Immobilien Investment GmbH's purchase of the Symphony Office Tower in Amsterdam for €215.1 million in December 2013 was the largest commercial-real-estate deal of the year. "There is a lot of dynamism, and there are a lot of new entries in the market," said Mr. Wolters. CBRE brokered the deal.

In the first few months of 2014, a joint venture of Mount Kellett Capital Management LP and Sectie5 Investments NV bought a €213 million portfolio of shopping centers from Dutch real-estate-investment trust Corio NV. Also, Dutch property-firm Meyer Bergman acquired the outlet shopping center Batavia Stad for €115 million.

Meanwhile, the Dutch banking sector is recovering slowly. The largest Dutch banks, ING Group and Rabobank Group, have been reducing their portfolios of loans. SNS REAAL has been nationalized and transferred €4.8 billion of commercial-real-estate loans and investments to the newly established so-called bad bank, Propertize.

Prudential Real Estate Investors is a European real-estate branch of Prudential Financial Inc. Pramerica Real Estate Capital V already has a pipeline of deals worth approximately €150 million, with projected annual returns that can vary between the single digits to the lower teens, said Andrew Radkiewicz, managing director and co-head of Europe at Prudential Real Estate Investors.

Other nontraditional lenders considering the Netherlands market include Los Angeles-based Karlin Real Estate, which has raised a €600 million war chest for European investments. The firm, which just opened a London office, says it is looking at possible equity investments and mezzanine loans in the Netherlands.

Some analysts, though, warn that mezzanine lenders shouldn't overpromise. Eleonora Pulci, a director of debt and structured finance at CBRE, says that, in 2011, some of these lenders fell short of their goal of returns in the 18%-to-20% range, partly due to competition.

Also, she says, the European market is getting more complicated for investors. In the past few years, they could focus on one or two large markets. Now, they are increasingly picking and choosing among a number of smaller markets, like Ireland, Spain and the Netherlands.

"Today, the market is complex" Ms. Pulci said.