Groups with finitely many normalizers of non-nilpotent subgroups