Groups with finitely many normalizers of non-abelian subgroups