Groups with few normalizer subgroups