RecyclerView and notifyDataSetChanged LongClick mismatch

I'm having a weird problem with notifyDataSetChanged() in my Recycler Adapter. If I keep 5 items in an array the code works fine and I can check the checkbox at the item I LongClick, but when I add 5 items or more to the array other checkboxes get checked in my list.

I am using a boolean to toggle between VISIBLE and GONE on the checkboxes when the user LongClicks as well.

Here is my code:

class RecyclerAdapter(private val listActivity: ListActivity) : RecyclerView.Adapter<RecyclerAdapter.Holder>() {

    lateinit var binding: ActivityListItemRowBinding
    var checkboxesVisibility = false
    val dummyArrayWorks = arrayOf("000", "111", "222", "333", "444")
    val dummyArrayFails = arrayOf("000", "111", "222", "333", "444", "555")

    override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): Holder {
        binding = ActivityListItemRowBinding.inflate(LayoutInflater.from(parent.context), parent, false)
        return Holder(binding)
    }

    override fun getItemCount(): Int = dummyArrayFails.size

    @SuppressLint("NotifyDataSetChanged")
    override fun onBindViewHolder(holder: Holder, position: Int) {

        val item = dummyArrayFails[position]
        
        holder.binding.checkbox.visibility = if (checkboxesVisibility) VISIBLE else GONE
        holder.bindItem(item)

        holder.itemView.setOnLongClickListener {
            if (!checkboxesVisibility) {
                checkboxesVisibility = true
                holder.binding.checkbox.isChecked = true
                notifyDataSetChanged()
                true
            } else {
                false
            }
        }
        holder.itemView.setOnClickListener {
            if (!checkboxesVisibility) {
                //Some other unrelated code
            } else {
                holder.binding.checkbox.isChecked = !holder.binding.checkbox.isChecked
                notifyDataSetChanged()
            }
        }
    }

    class Holder(internal val binding: ActivityListItemRowBinding) : RecyclerView.ViewHolder(binding.root) {

        var item = String()

        fun bindItem(item: String) {
            this.item = item
            binding.itemPlaceHolder.text = item
        }
    }
}

I should add that when I remove the toggle for the checkboxes, and just show the checkboxes on first load, the clicks match the checkmarks without a problem.

Does anybody have any idea of what is going on? All help will be much appreciated!