As team admin you can remove team members from your team, which will also remove the team activity controls from their home screen. , as the removal disassociates it from your organization unit and thus hides it from your "+ Add" user search results.
To re-add a user once they were removed from a team, is only possible by tenant admins as they can see and access the full tenant user directory.