I've done this exact thing several years ago, using the jQuery fancyBox plugin (http://fancyapps.com/fancybox/). There very well may be better popup/modal window alternatives by now, although I was satisfied with fancyBox as it was very easy to work with.

FancyBox solves the problem of displaying the popup when the video thumbnail link is clicked. Inside fancyBox, you can embed the YouTube video (or HTML, or an image, or anything - not just videos).

Directly on the fancyBox documentation page, they link to this JsFiddle: http://jsfiddle.net/M78zz/, which demonstrates how to launch a YouTube video from a popup link. The fiddle also demonstrates how to use the YouTube API to go to the next video when the first is finished playing.

Here's the JsFiddle code:

HTML

<!-- This loads the YouTube IFrame Player API code -->

<script src=http://www.youtube.com/player_api></script>



<a class=fancybox fancybox.iframe href=http://www.youtube.com/embed/L9szn1QQfas?enablejsapi=1&wmode=opaque>Video #1</a>



<br />



<a class=fancybox fancybox.iframe href=http://www.youtube.com/embed/cYplvwBvGA4?enablejsapi=1&wmode=opaque>Video #2</a>

JavaScript

// Fires whenever a player has finished loading

function onPlayerReady(event) {

    event.target.playVideo();

}



// Fires when the player's state changes.

function onPlayerStateChange(event) {

    // Go to the next video after the current one is finished playing

    if (event.data === 0) {

        $.fancybox.next();

    }

}



// The API will call this function when the page has finished downloading the JavaScript for the player API

function onYouTubePlayerAPIReady() {



    // Initialise the fancyBox after the DOM is loaded

    $(document).ready(function() {

        $(.fancybox)

            .attr('rel', 'gallery')

            .fancybox({

                openEffect  : 'none',

                closeEffect : 'none',

                nextEffect  : 'none',

                prevEffect  : 'none',

                padding     : 0,

                margin      : 50,

                beforeShow  : function() {

                    // Find the iframe ID

                    var id = $.fancybox.inner.find('iframe').attr('id');



                    // Create video player object and add event listeners

                    var player = new YT.Player(id, {

                        events: {

                            'onReady': onPlayerReady,

                            'onStateChange': onPlayerStateChange

                        }

                    });

                }

            });

    });



}

CSS

.fancybox-nav {

    width: 60px;       

}



.fancybox-nav span {

    visibility: visible;

}



.fancybox-next {

    right: -60px;

}



.fancybox-prev {

    left: -60px;

}