On the uniqueness of higher-spin symmetries in AdS and CFT