matt, you are right. Privacy plugin just gathers actions from other plugins.
If you would like to remove certain privacy option you will need to look into source code of corresponding plugin. Usually action gets declared within
public function genericInit() .
Let's take "Photo" plugin as an example. It adds "Who can view my photo albums" action into "Privacy" plugin. To remove this privacy settings you will need to edit ow_plugins/phot/classes/event_handler.php file.
Line to remove: $em->bind('plugin.privacy.get_action_list', array($this, 'addPrivacyAction'));
Another example "Blogs" plugin. It adds "Who can view my blogs" action. To remove it you will need to edit ow_plugins/blogs/classes/event_handler.php file.
Action to remove:
$action = array(
'key' => PostService::PRIVACY_ACTION_VIEW_BLOG_POSTS,
'pluginKey' => 'blogs',
'label' => $language->text('blogs', 'privacy_action_view_blog_posts'),
'description' => '',
'defaultValue' => 'everybody'
);
$event->add($action);