Congress has reached a deal to give federal employees paid time off to care for new children. The policy change is seen as a big step forward in the push for a national policy of paid family leave for all workers — America is the only wealthy country in the world not to require businesses to provide that benefit.

But it isn’t as sweeping as some might hope. While more than two million federal workers are to get 12 weeks of paid leave for the birth of a newborn or an adoption, new parents make up only about 20% of workers who take family leave, according to Michelle McGrain, federal affairs manager with the National Partnership for Women and Families.

“We don’t want to limit it to just one type of leave, one type of need, and tell other people that their reasons for needing time off from work are not as important,” McGrain said.

As it stands, caring for a sick child, spouse or elderly parent won’t be covered. More than 40 million Americans regularly provide care for a family member according to AARP, and, as Meshal Desantis with Paid Leave for the US pointed out, the responsibility falls disproportionately to women and people of color.

“Women and people of color are also the least likely to have access to paid leave, so the people who are facing the highest burdens are actually the least likely to have the resources to face those,” she said.

That can lead workers to drop out of the labor force, slowing down economic growth and cutting those workers’ long-term earning potential, said to Eileen Appelbaum, co-director of the Center for Economic and Policy Research.

“It’s like an escalator,” she said, “you got off the escalator and when you got back on, you’re getting back on at a much lower level.”