Blamed for everything from decreased productivity to increased flu cases, the majority of workers hate them, a survey says.