Idea is basically giving every name an id and removing them by id on button click.

Updated fiddle

this gives every li an id and adds a button with removeName('itemid') on it's onClick attribute.

entry.setAttribute('id','item'+lastid);

var removeButton = document.createElement('button');

removeButton.appendChild(document.createTextNode("remove"));

removeButton.setAttribute('onClick','removeName("'+'item'+lastid+'")');

entry.appendChild(removeButton);

removeName function simply finds item and remove it from the list.

function removeName(itemid){

    var item = document.getElementById(itemid);

    list.removeChild(item);

}

EDIT:
As to why zozo's answer would cause problems:
it allows you to add multiple items with same id. I explained in a comment on their answer.

EDIT

So for your 2 requests:

1) Getting data from DOM

Going for this first because it builds up on above answer.

FIDDLE

So we have our names, and able to add and remove them to the list but now we need the data as an array.

added this line for easier access to the name

entry.setAttribute('data-name',firstname);

and getting all names from the list like this:

function getNames(){

    var names = [];

    for(var i=0;i<list.children.length;i++){

        names.push(list.children[i].getAttribute("data-name"));//get previously set attribute and add to array

    }

    return names;

}

We can simply get the attribute we added to get the name. If we do not add that attribute we would have need to go in to the inner nodes of the <li> and grab the text node to get the names. We can't simply get innerText or innerHtml of the li because we added a button( the remove) button inside the li element.

I added another button at the end of the form to test the getNames function.

2) Storing data in js from the begining

So the other approach is different, this time we do not add to/read from DOM but we store everything in an array and. Every time the array changes we render the list for display purposes.

FIDDLE

So, we have a names array that we will add to and remove from.

var names = []; // [] is same as new Array();

We simply add the name to the list and re-render the list

function changeText2() {

    var firstname = document.getElementById('firstname').value;

    document.getElementById('boldStuff2').innerHTML = firstname;

    names.push(firstname);//simply add new name to array;

    //array changed re-render list

    renderList();

}

And when we need to remove we simply remove it from the array and re-render the list:

function removeName(nameindex){

    names.splice(nameindex,1);

    //array changed re-render list

    renderList();

}

As you might guess everything happens in the renderList function:

function renderList(){

we remove everything from the DOM first:
According to this answer this is faster than simply doing list.innerHTML = ""

//clean the list

while (list.firstChild) {

    list.removeChild(list.firstChild);

}

Then we create each item again

for(var i=0;i<names.length;i++){

    var entry = document.createElement('li');

    entry.appendChild(document.createTextNode(names[i]));

    var removeButton = document.createElement('button');

    removeButton.appendChild(document.createTextNode("remove"));

    removeButton.setAttribute('onClick','removeName('+i+')');

    entry.appendChild(removeButton);

    list.appendChild(entry);

}

Note that removeName function now takes an array index instead of id.
you need to call renderList everytime the array changes so the changes are visible in the dom.