You need to create just one InfoWindow object, keep a reference to it, and reuse if for all the markers. Quoting from the Google Maps API Docs:

If you only want one info window to display at a time (as is the behavior on Google Maps), you need only create one info window, which you can reassign to different locations or markers upon map events (such as user clicks).

Therefore, you may simply want to create the InfoWindow object just after you initialize your map, and then handle the click event handlers of your markers as follows. Let's say you have a marker called someMarker:

google.maps.event.addListener(someMarker, 'click', function() {

   infowindow.setContent('Hello World');

   infowindow.open(map, this);

});

Then the InfoWindow should automatically close when you click on a new marker without having to call the close() method.