CheckBox in List Adapter acts strange

I have a list adapter , and each item has a checkbox. I also have an option to delete from that list all the items the user checks. The part that should delete the selected items works well, but.. for example if I have

1. 0000
2. 5555
3. 9999

If I check 5555 and 9999, and click remove, they disappear but then 0000 gets checked even though I haven't pressed it.

(historyList is all the items in history fragment. listToRemove is only the checked items)

 @Override
public View onCreateView( LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState ) 
{ 
    View view = inflater.inflate( R.layout.fragment_history, container, false );
    adapter = new PasswordAdapter( historyList );
    setListAdapter(adapter);
    ImageView removeButton = ( ImageView ) view.findViewById( R.id.removeButton );
    removeButton.setOnClickListener( new OnClickListener(){
        @Override
        public void onClick( View v )
        {
            if ( !listToRemove.isEmpty() )
            {
                listToRemove.clear();

                adapter.updateList( historyList );
            }
        }
    });

    return view;
}
...
...
...
@Override
    public View getView(int position, View convertView, ViewGroup parent) {
        final View result;

        if (convertView == null) 
        {
            result = LayoutInflater.from(parent.getContext()).inflate(R.layout.history_list_item, parent, false);
        } 
        else 
        {
            result = convertView;
        }

        final Password item = getItem( position );

        ( (TextView) result.findViewById( R.id.historyPasswordTextView) ).setText(item.getPasswordString());
        ( (TextView) result.findViewById( R.id.historyDateTextView ) ).setText(item.getPasswordString());
        CheckBox checkBoxToRemove = (CheckBox) result.findViewById( R.id.removePasswordFromHistoryCheckBox ) ;
        checkBoxToRemove.setOnCheckedChangeListener( new OnCheckedChangeListener(){
                @Override
                public void onCheckedChanged( CompoundButton buttonView, boolean isChecked )
                {
                    if( isChecked )
                    {
                        listToRemove.add( item );
                    //  Log.d( "TAG", listToRemove.toString() );

                        for( int i=0; i<listToRemove.size(); i++)
                            Log.d("TAG","checked after add: "+ listToRemove.get(i).getPasswordString());
                    }
                    else
                    {
                        listToRemove.remove( item );
                        for( int i=0; i<listToRemove.size(); i++)
                            Log.d("TAG","checked after remove: "+ listToRemove.get(i).getPasswordString());
                    }

                }
            });

Have I missed something?