Groups with finitely many normalizers of non-subnormal subgroups