Currently if you just make someone an Activity Leader of an Activity Group the required way for that person to add/drop members from the group is to follow these steps:
Select the Group
As a work around if you make a Content Editor Access Group, add the Activity Leader to it and give that group access to the Activity Group in question they will be able to use the "Manage" button directly on the Activity page under the Roster. I feel this should be the DEFAULT action and not require the additional steps of creating and maintaining this Activity Content Editor Access group.
As an aside, that I may open another idea for, it's worth noting that while the Activity Content Editor Access groups and Editors are rolled over with the Master Rollover the permissions given to these groups on the Activity Groups are not. So those permissions will need to be manually created for each group.