One in four American workers surveyed by Bankrate.com say they can't take a vacation, and another third won't use all the vacation they're due.

Most say they can't afford it.

A sizable chunk of workers in all age groups expressed doubts about taking time off this summer, with millennials saying they're just trying to put their best foot forward in a new job.

"They are at a point where they are still trying to advance their career and might be in a position where they're trying to impress their employer," says Bankrate's Amanda Dixon.

For whatever reason workers are forgoing vacation, Dixon says it's leaving money on the table, and worse.

"If you're constantly working without taking a vacation, you're going to eventually burn out," she warns.