set elevation to custom checkbox button only in selector android

I want to set elevation/shadow to a custom checkbox button only and not its text. This is what I am doing currently

  <CheckBox
        android:id="@+id/custom_checkbox"
        style="@style/CustomCheckBox"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_marginTop="@dimen/standard_page_margin"
        android:elevation="20dp"
        android:outlineProvider="bounds"
        android:text="This is an example." />

and the style is

<style name="CustomCheckBox" parent="Base.Widget.AppCompat.CompoundButton.CheckBox">
    <item name="android:button"> @drawable/custom_selector_check_box_button</item>
    <item name="android:textAppearance">@style/CustomTextStyleBody</item>
    <item name="android:textColorLink">@color/custom_bright_sky_blue</item>
    <item name="android:textColorHighlight">@android:color/transparent</item>
    <item name="android:paddingStart">16dp</item>
</style>

where the selector is

<?xml version="1.0" encoding="utf-8"?>
<selector xmlns:android="http://schemas.android.com/apk/res/android">
<item android:state_checked="true">
    <layer-list>
        <item>
            <shape android:shape="rectangle">
                <size android:width="40dp" android:height="40dp" />
                <solid android:color="@color/slate_gray" />
            </shape>
        </item>
        <item android:gravity="center" android:drawable="@drawable/custom_tick" />
    </layer-list>
</item>
<item android:state_checked="false">
    <shape android:shape="rectangle">
        <size android:width="40dp" android:height="40dp" />
        <solid android:color="@color/slate_gray" />
    </shape>
</item>

But this puts shadow to the entire checkbox along with the text. Also, I am not able to change the elevation value in selector