It is hard to miss inflation in the recent numbers, and maybe even more so, the fear that inflation is going to hang around a lot longer than the Federal Reserve and investors would like. You may see it at the pump, in food prices, or the housing market, and for those who follow stocks, you definitely hear about it in the billionaire investor talking points.

Billionaire hedge fund manager Paul Tudor Jones said inflation may be worse than feared for both the markets and society. Bill Ackman called last week for the Federal Reserve to start raising rates as soon as possible. David Tepper said stocks don’t look like a great investment from here, but it all depends on rates.

You get the picture.

Warren Buffett, not fond of short-term calls on the economy or market, had plenty to say earlier in his career about what inflation can do to stock market wealth. Buffett’s view of inflation was heavily influenced by the runaway inflation of the 1970s. “Inflation is a far more devastating tax than anything that has been enacted by our legislatures,” he wrote in 1977. “The inflation tax has a fantastic ability to simply consume capital. … If you feel you can dance in and out of securities in a way that defeats the inflation tax, I would like to be your broker — but not your partner.”

What the billionaires think and do is pretty far from lining up with the investment reality of most individuals, so take it down a few zeroes, what are wealthy do-it-yourself investors — individuals with at least $1 million in a brokerage account that they manage on their own — thinking right now?

They are increasingly worried, too, according to a survey from Morgan Stanley’s E-Trade Financial of millionaire investors provided exclusively to CNBC. According to the quarterly E-Trade data, these wealthy investors have not been this concerned about their stock market holdings or the economy since Q2 2020, right after the March 2020 Covid crash and shutdown of the U.S. economy

“We’re definitely seeing a downtick in optimism,” said Mike Loewengart, managing director of investment strategy at E-Trade. “They are starting to see some cracks in the economic recovery and it’s dampening bullishness.”

More of the wealthy still describe themselves as bullish, but just barely, with that indicator dropping from 65% in Q3 2021 to 52% in the current quarter. That’s the lowest level of bullishness since Q2 2020.

The E-Trade survey was conducted October 8 to October 16 among 119 investors with $1 million or more of investable assets.

Last quarter, 72% of these wealthy investors said inflation was “transitory,” supporting the Fed view. That has now fallen to 53%. Those who “strongly disagree” with the Fed’s transitory view increased from 9% to 19%.

Inflation has been a concern all year, but “while it isn’t new, it is a lot stickier,” Loewengart said.

A new CNBC Fed Survey out on Tuesday finds respondents — which includes money managers, strategists, and economists — forecasting the first rate hike to move up to September 2022 from December in the last survey. And almost half (44%) of the 25 respondents believe the Fed will raise rates by July. A majority (60%) believe inflation is a big enough concern that the Fed should halt all asset purchases now.

The E-Trade millionaire set, meanwhile, includes many business owners who have firsthand experience with the supply chain challenges and the bite of inflation, and they may also be seeing it in broader investments they make beyond public equities, and that is contributing to these investors being more attuned to the issue. 

Michael Sonnenfeldt, founder of Tiger 21, an investing network for the affluent which includes many former and current business owners, said no practical business person he knows would now make a bet against inflation heating up for a period of time. “To most of our members it feels like something more than transitory and the start of a secular trend. There is more consistent evidence of inflation coursing through the economy than we’ve seen in decade, or longer.”

Part of it is timing, not only the Fed’s.

As fears about the delta variant’s impact on the economy diminish, and vaccines become available even more widely, people are expecting that this time the virus really is under control and they can go out again and spend.

“The relationship between spending and inflation fears is pretty high,” said Lew Altfest, CEO of Altfest Personal Wealth Management. “The delta variant caused a postponement of the inflation discussion, but now it’s back and it does have the power to prevent the markets from making progress. So we are in the rethink period. Delta just pushed out the inflation discussion by limiting growth in the short-term, but now it is back to the fundamentals that increase visibility farther out,” he said.

While the recent GDP reading of 2% was a slowdown, Altfest noted that that services component was up significantly and there is likely more spending to come from the federal government, another tailwind for growth. “If growth really picks up again, then inflation will pick up,” he said.

The bond market is expecting it to, with a key reading moving up to a level indicating higher-than- anticipated inflation will last for years.

“Now as the dust has started to clear, the new normal of business looks like it will include supply chain and labor issues, and commodities surging. On all fronts, the new normal more people see us going back to includes a lot more inflation,” Sonnenfeldt said. “Takes a while to have enough data points … and all of a sudden there are 10 things in a pattern starting to really ring some bells.”

The percentage of millionaires who expect the market to end this quarter with a gain declined from 70% to 47%. Those who expect the market to drop increased by 21 percentage points, from 14% to 35%.

That fits the virus versus inflation paradigm as it’s the first time these wealthier investors have expected a quarterly decline in stocks since Q2 2020.

“That’s a key shift,” Loewengart said. “In past quarters millionaire investors were near uniform in their belief that the market would continue to rise.”

A big disconnect, though: earnings are still coming in strong, despite some notable disappointments towards the end of last week including Amazon and Apple. Overall, the market is exceeding expectations in Q3 earnings — and based on were analysts were into this earnings periods, forward estimates will have to be revised upward, which supports the market, which already is at a record.

Q2 2020 ended up being one of the best on record, but that came after the crash in Q1 2020 rather than stocks reaching new records on a regular basis. Investor sentiment trailed the actual market comeback in 2020, but this time around, there is less room in price-to-earnings ratios to quickly make up.

“The more stocks go up, the more investors feel validated, but that’s a big weakness in thinking because the more stocks go up the more risky they become,” Altfest said. “Investors know the market is high and they just ignore it. I wouldn’t call a 45 P/E company safe because it has a strong position in this market.”

Even with a shift in Fed policy influencing some investors, it still should be described today as “fairly accommodative,” Loewengart said.

The downward revision in wealthy investor expectations has not led to major changes in portfolio allocations, though in the E-Trade survey there was a slight decrease among those saying they were making no changes to their investments, with that falling from 47% to 41%. And there was an increase of 5 percentage points in those moving out of investments and into cash, from 17% to 22%.

But as Loewngart said, “If you sit in cash in the current inflationary environment you’re a net loser. Your purchasing power will decline and that’s not up for debate.” 

In the Tiger 21 investor community, holdings in currencies, commodities and crypto all are edging up.

“I don’t think there is a single reason for interest in crypto. Inflation is one of them, but it also represents an extraordinary new frontier in the world of assets,” Sonnenfeldt said. “But people are really thinking about what if there were hyper inflation, what kind of assets will hold their value, and for some portion of people crypto is the answer and an investment opportunity.”

Over the last decade, fixed-income allocations among Tiger 21 members have declined from low teens to now resting at 7%, the lowest level ever for bonds in the history of its investor surveying. “People don’t want interest rate exposure going out at all,” Sonnenfeldt said.

Public equity has risen above private equity, which is uncommon for the Tiger 21 community, and that is a measure of the flexibility investors want and in public stocks the feeling that investors can shift on quicker notice, he added.

Among the E-Trade group, more millionaires cited treasury-inflation-protected securities and interest-rate sensitive sectors including financials, materials and energy as their inflationary preferences.

But overall, the sector view is still skewed to the long-time favorites among these investors: information technology and health-care, which Loewengart said to a large degree have the ability to weather inflation shocks, and pass on additional costs to end consumers. Energy is the notable inflation winner right now as commodity prices surge and winter nears with energy shortages around the world, from natural gas in Europe to coal in China, reaching an equal 45% (the same as tech and health care) among millionaires asked to pick the best sector for Q4.

“It’s about reckoning the environment will be more challenging and you have to pick your spots accordingly. But you still have to be invested to capture future returns,” Loewengart said.

Wealthy investors surveyed by E-Trade who describe the economy as expanding fell from 41% to 23% while those who see a recession increased from 14% to 23%. Those who described economic conditions as peak, including by definition that “inflation takes hold” and economic growth stops or slows, increased from 22% to 29%.

“It’s such a drastic shift in economic outlook quarter over quarter,” Loewengart said.

While it may seem surprising to see more of the wealthy describe a still-growing economy as being in a recession, there is a historical precedent for why they may feel this way. Inflation can run hot, at 3% to 4%, and co-exist with rising corporate profits, at least for a while, before action needs to be taken. It’s that potential action, from the Fed, which Altfest says has the attention of investors.

“Cutting inflation means raising rates and in most cases the quickest way to get rid of serious inflation is by raising rates fast enough to cause a recession. If things really pick up that’s what is going to happen. Volcker got rid of inflation,” he said, alluding to the decision by then-Fed chairman Paul Volcker to sharply raise rates starting in the late 1970s and cause a recession as a way to control inflation.

Among the Tiger 21 investors, more have been investing based on the belief that new technology will transform the economy (34%) than based on fear of a market correction or inflation (25%). Though even more (42%) say neither is their main investing rationale.

“An investor needs a component of risk-taking to preserve wealth, and the engine of wealth preservation is called growth,” Sonnenfeldt said. “Innovation is the flipside of inflation in the investor mindset and many wealthy investors do feel like this is a unique period of innovation.”

Venture capital is increasing to an allocation level it has never had among these investor portfolios as people see an economy becoming much more complicated and new opportunities in areas including energy transition. “People are being forced to find business that can pivot to new opportunities and inflation can create a lot of opportunities if it is managed well,” Sonnenfeldt said.

He views the inflation versus innovation puzzle as two trains leaving a station.

“One is the inflation train and it’s already picking up speed, so for other train, call it the innovation train, to go faster it will need some amazing themes and success. And the faster the inflation train goes, the harder you have to push the innovation train to get past it. But inflation doesn’t preclude innovation from creating new industries and products.”