CheckBox state not saved when scrolling

I've tried reading the many answers on that issue, but I can't make it work in my app.. When I check items (which I can do both by clicking on an item, and by clicking the checkbox itself) , they go to a list (listToRemove) that once I click on a button that list is passed to another fragment. The issue is every time I scroll away from a checked checkbox, it changes its state to unchecked again. I've seen multiple questions regarding this issue here on SO, but I couldn't make it work. A help would be much appreciated.

Thanks.

@Override
    public void onListItemClick( ListView l, View v, int position, long id ) 
    {
        Password item = ( Password ) ( getListAdapter() ).getItem( position );
        CheckBox check = ( CheckBox ) v.findViewById( R.id.removePasswordFromHistoryCheckBox );
        if( !check.isChecked() )
        {
            if( !listToRemove.contains( item ))
            {
                listToRemove.add( item );
            }
            check.setChecked( true );
        }
        else
        {
            if( listToRemove.contains( item ))
            {
                listToRemove.remove( item );
            }
            check.setChecked( false );
        }
    }

....
....

  @Override
    public View getView(int position, View convertView, ViewGroup parent) 
    {
        ViewHolder holder;
        final Password item = getItem( position );

        if (convertView == null) 
        {
            holder = new ViewHolder();
            convertView = LayoutInflater.from(parent.getContext()).inflate(R.layout.history_list_item, parent, false);
            holder.dateText = (TextView) convertView.findViewById( R.id.historyDateTextView ) ;
            holder.passwordText =  (TextView) convertView.findViewById( R.id.historyPasswordTextView ) ;
            holder.checkBoxToRemove = (CheckBox) convertView.findViewById( R.id.removePasswordFromHistoryCheckBox ) ;
            convertView.setTag( holder );

        } 
        else 
        {
            holder = (ViewHolder) convertView.getTag();
        }

        if( listToRemove.contains( item ) )
        {
            holder.checkBoxToRemove.setChecked(true);
        }
        else
        {
            holder.checkBoxToRemove.setChecked(false);
        }

            holder.dateText.setText( historyList.get( position ).getDate() );
            holder.passwordText.setText( historyList.get( position ).getPasswordString() );
            holder.checkBoxToRemove.setOnCheckedChangeListener( new OnCheckedChangeListener(){
                    @Override
                    public void onCheckedChanged( CompoundButton buttonView, boolean isChecked )
                    {
                        if( isChecked )
                        {
                            if( !listToRemove.contains( item ))
                            {
                                listToRemove.add( item );
                            }
                        }
                        else
                        {
                            if( listToRemove.contains( item ))
                            {
                                listToRemove.remove( item );
                            }
                        }
                    }
                });

        return convertView;
        }