Groups with finitely many normalizers of infinite index