Limiting riskier, illiquid investment could make banks and the financial system more stable — but the Federal Reserve isn’t so sure.