You could set a container element or maybe even the body to overflow: hidden with a width and height of the browser window. This way any overflowing content will fall off the page and scroll bars are never displayed. This can be set in a css statement like body.dialog-open { overflow: hidden; }. You can then add and remove the .dialog-open classname when the dialog opens and closes.

The width and height might not be required if setting this on the body, but I'd have to check browser compatibility on that one. Might get some unexpected results there.

edit:
If you want scrolling inside your dialog you can set overflow: auto there, with a height set on that element.

Older browsers (most notably IE) might show a horizontal scrollbar as well, you might have to set overflow-x: hidden if that is the case.

Also see: CSS div element - how to show horizontal scroll bars only? for more information on scrollbars.