Used .prop() to create a Select All checkbox but don't want the checkbox removed

So I created a select all checkbox to select and deselect my list. Everything works great but when I remove the list it also removes the select all checkbox. Please tell me what am I doing wrong. Here is the code:

HTML:

         <div class="item-list">
                <H2>Shopping List</H2>              
                <ul class="check-list">
                    <li><input type="checkbox" id="select-all"/>Select All</li>
                </ul>
            </div>

jQuery:

$(document).ready(function(){

    $(".add-btn").click(function (){
        var addItem = $('#item-add').val();
        $(".check-list").append('<li ><input type="checkbox" class="check-box">' + addItem + '</li>');
    });

    $('#select-all').change(function(){
        $('.check-box').prop('checked', $(this).prop('checked'));
    });

    $(".rem-btn").click(function(){
        $(".check-list input:checked").parent().remove();
    });

});