Black workers and those without college degrees were forced out of jobs at the highest rates during the pandemic.