Android onCheckedChangeListener bizarre behavior

So I am attempting to code some CheckBox logic for a page in which a user can +1 or -1 a specific entry. The first time my RecyclerViewAdapter is bound, I iterate through the list of votes and check if the user id is present, and if the vote value is +1 or -1 here

fun bindTask(lunchVote: LunchVoteGroupDestination) {
        mItem = lunchVote
        mIdView.text = lunchVote.groupDestination?.nickname
        mScoreView.text = lunchVote.score.toString()
        destinationId = lunchVote.groupDestinationId
        votes = lunchVote.votes

        for(vote: Vote in votes) {
            if(prefs!!.currentId == vote.userId && vote.voteValue == 1) {
                plusOneBox.setOnCheckedChangeListener(null)
                plusOneBox.isChecked = true
            }
            if (vote.userId.equals(prefs!!.currentId) && vote.voteValue == -1) {
                minusOneBox.setOnCheckedChangeListener(null)
                minusOneBox.isChecked = true
            }
        }

Which works fine! If I previously voted +1 and reload, boom, the box is checked. The issue, however, comes when I then try to interact with the boxes again. If I hit the +1 box, the check remains and doesnt do anything. This also occurs for the -1 box. I also set my listeners for the CheckBoxes in the bindtask function like so

fun bindTask(lunchVote: LunchVoteGroupDestination) {
        mItem = lunchVote
        mIdView.text = lunchVote.groupDestination?.nickname
        mScoreView.text = lunchVote.score.toString()
        destinationId = lunchVote.groupDestinationId
        votes = lunchVote.votes

        for(vote: Vote in votes) {
            if(prefs!!.currentId == vote.userId && vote.voteValue == 1) {
                plusOneBox.setOnCheckedChangeListener(null)
                plusOneBox.isChecked = true
            }
            if (vote.userId.equals(prefs!!.currentId) && vote.voteValue == -1) {
                minusOneBox.setOnCheckedChangeListener(null)
                minusOneBox.isChecked = true
            }
        }

        System.out.println("PlusOneBox " + plusOneBox.isChecked + mIdView.text)
        plusOneBox.setOnCheckedChangeListener(object : CompoundButton.OnCheckedChangeListener {
            override fun onCheckedChanged(buttonView: CompoundButton, isChecked: Boolean) {
                System.out.println("PLUSONE CHECKED")
                if(isChecked) {
                    val upvoteData = VoteObject(prefs!!.accessToken, prefs!!.currentGroup, prefs!!.currentLunchVote, destinationId)
                    socket.upvote(upvoteData)
                } else {
                    val downvoteData = VoteObject(prefs!!.accessToken, prefs!!.currentGroup, prefs!!.currentLunchVote, destinationId)
                    socket.downvote(downvoteData)
                }
            }
        })

        minusOneBox.setOnCheckedChangeListener(object : CompoundButton.OnCheckedChangeListener {
            override fun onCheckedChanged(buttonView: CompoundButton, isChecked: Boolean) {
                System.out.println("MINUSONE CHECKED")
                if(isChecked) {
                    val downvoteData = VoteObject(prefs!!.accessToken, prefs!!.currentGroup, prefs!!.currentLunchVote, destinationId)
                    socket.downvote(downvoteData)
                } else {
                    minusOneBox.setOnCheckedChangeListener(null)
                    val upvoteData = VoteObject(prefs!!.accessToken, prefs!!.currentGroup, prefs!!.currentLunchVote, destinationId)
                    socket.upvote(upvoteData)
                    minusOneBox.setOnCheckedChangeListener(this)
                }
            }
        })
    }

It should also be worth mentioning that I have a socket listener in my fragment class that uses the adapter listening for a +1 and -1 event, in which case it updates the score of the item and notifys the adapter that the data set has changed, like here

private val onUpvote = Emitter.Listener { args ->
    val data = args[0] as JSONObject
    var newScore: Int
    var updatedId: Int
    System.out.println(data)
    if(activity != null) {
        activity.runOnUiThread({
            try {
                newScore = data.getInt("score")
                updatedId = data.getInt("id")
                for (i in LUNCHVOTE_DESTINATIONS.size - 1 downTo 0) {
                    val destination = LUNCHVOTE_DESTINATIONS.get(i)
                    if(updatedId == destination.id) {
                        LUNCHVOTE_DESTINATIONS.get(i).score = newScore
                        recyclerView!!.adapter.notifyItemChanged(i, LUNCHVOTE_DESTINATIONS.get(i))
                        break
                    }
                }
            } catch (e: JSONException) {
                e.printStackTrace()
            }
        })
    }
}

Can any of you see why my CheckBoxes work fine when nothing is previously selected, but on a page reload it takes multiple clicks just to uncheck the boxes. Does it have to do with notifying the data set has changed in my listener. Am I binding the listeners multiple times so it's messing with my logic? Is my checkbox logic simply wrong? Any help here would be much appreciated.