Only creator or other participants can invite other members to group, this is how it works currently. To change this logic is considered as a custom code modification.
Please create a post in the custom code modification section of forum or you're free to hire a developer to do that for you.
Ironically, I searched before posting this. I still haven't gotten the hang of this forum. I searched "admin adding to groups" and other variations & came up with nothing at all. In addition, it seems that I can't add any of the site members to the group that I created (as admin). Can you only add people that're on your friends list? If so, why would the admin be bound to this?
To be honest, I'm surprised that a plugin would have to be purchased just to allow the admin to add members to specific groups. The admin should be all powerful over the site -how is this not already a feature? Maybe I'm just not used to the culture of Oxwall...