Daniel, that is how "Friends" and "Groups" plugins works together. If you deactive "Friends" plugin, you will be able to invite all users from your site. What you would like to achieve also requires some custom code modifications.
1. "I would also like to invite without being a member"
See my reply to Jeff.
2. "and without having to be friends with invitees.".
You will need to edit ow_plugins/groups/components/invite_widget.php:
if ( OW::getEventManager()->call('plugin.friends') )
{
$users = OW::getEventManager()->call('plugin.friends.get_friend_list', array(
'userId' => $userId,
'count' => 100
));
}
3. Even more complex changes are required if you would like to make modifications #1 and #2 for admin only ( and regular way for all other users).
On another note, there is a plugin up in the store that allows admin to become friends with everybody automatically. You might want to check it out.