Remove something added by a particular even - c#

I have a piece of code that will add the selected item of a combobox to a listbox when a checkbox is checked. I would like to remove that selected item to be removed from the listbox when the checkbox is unchecked.

My problem is I cant simply repeat the code for removal to be the same as add because the combobox selection will different or empty when unchecked.

This is how it currently looks:

private void CBwasher_CheckedChanged(object sender, EventArgs e)
{
        if (checkBox1.Checked == true)
        {
            listBox2.Items.Add(comboBox1.SelectedItem);
        }
        if (checkBox1.Checked == false)
        {
            listBox2.Items.Remove(comboBox1.SelectedItem);
        }

So I need a way of removing whatever was added by that check change instead of removing the selected index of the combobox. Please consider that there may be multiple lines in the listbox added by multiple different checkboxes.