The job market in the U.S. has remained hot this year despite the Federal Reserve’s best efforts to cool it down, and unemployment is at just 3.7%, close to a 50-year low.

But part of the population has been silently walking away from work for several decades—well before the COVID-19 pandemic began.

Men without four-year college degrees, between the ages of 25 and 54, have left the workforce in higher numbers than other groups. And they’re leaving in part because of their perceived social status relative to better-educated men of similar age, according to a new study from the Federal Reserve Bank of Boston.

Non-college-educated men have seen their pay shrink by more than 30% since 1980 compared to the average earnings of all other prime-age workers. Their weekly earnings have declined 17%, while those of college-educated men rose by 20%, adjusting for inflation. That earnings loss has caused a decline in their social status, prompting them to walk away from work entirely, Pinghui Wu, the author of the study, wrote.

“For many workers, a job not only offers financial security, it also affirms their status, which is tied to their position relative to their age peers and many social outcomes,” Wu wrote.

The study found that the decline in earnings for non-college-educated men over the last four decades has increased their likelihood of leaving the labor force by nearly half a percentage point. That also accounts for 44% of the increase in their exit rate.

Even though the pattern of men withdrawing from the labor force dates back several years, it has been more pronounced in recent years, as a number of men who left the workforce during the pandemic chose not to return.

Nearly 89.7% of men between 35 to 44 years were looking for jobs or already working as of November. That’s down from 90.9% before the pandemic, according to the New York Times.

Younger white men in particular were more likely to leave when their expected wages fell relative to their more educated peers, according to the Fed study. Unlike men, women have not seen the same level of decline in their wages based on education. That group has seen a 32% increase in weekly earnings, irrespective of their educational qualifications.

There have been numerous studies investigating the decline of non-college educated prime-age men in the labor force over the past 40 years. Some have linked the decline in male labor force participation to the contraction of manufacturing jobs in the U.S., since men were disproportionately represented in the manufacturing sector. However, Wu says that explanation does not explain the persistence of the decline in prime-age men’s labor participation over several years.

“If the increasing wage gap between high and low earners directly or indirectly affects men’s aggregate labor supply, wage inequality might have carried wider implications to the economy than previously believed,” Wu wrote.

This story was originally featured on Fortune.com