If you want to get rid of all the groups and all the group data and start over you an always clear the group tables.
ox_groups_group
ox_groups_group_user
ox_groups_invite
This should get rid of all your groups and all the data for the groups if you clear those tables.
TBird,
Try looking at the url when you hover over the uninstall link of the plugin. Is the url structure different of the other plugin uninstalled link? If so copy the working uninstall link and remodify it with group. like e.g
WRONG-(will keep you hanging) url: plugins/groups/uninstall
RIGHT-(will delete it off your admin) url: plugins/uninstall-request/key/groups
If you do a fresh copy of oxwall with all the basic plugin that they have, you will see that you wont be able to uninstall all of them. (i don't know if it's a bug) but I remodified the urls and it removed the plugin.
I don't know if this is your issue but hope that helps.