Question

49

Screen Scraping from a web page with a lot of Javascript

rated 0 times [ 54] [ 5] / answers: 1 / hits: 18402 / 15 Years ago, wed, may 13, 2009, 12:00:00

I have been asked to write an app which screen scrapes info from an intranet web page and presents the certain info from it in a nice easy to view format. The web page is a real mess and requires the user to click on half a dozen icons to discover if an ordered item has arrived or has been receipted. As you can imagine users find this irritating to say the least and it would be nice to have an app anyone can use that lists the state of their orders in a single screen.

Yes I know a better solution would be to re-write the web app but that would involve calling in the vendor and would cost us as small fortune.

Anyway while looking into this I discovered the web page I want to scrape is mostly Javascript (although it doesn't use any AJAX techniques). Does anyone know if a library or program exists which I could feed with the Javascript and which would then spit out the DOM for my app to parse ?

I can pretty much write the app in any language but my preference would be JavaFX just so I could have a play with it.

Thanks for your time.

Ian

Answers

Only authorized users can answer the question. Please sign in first, or register a free account.

jayden

Add To Favorites

Follow

Total Points: 108

Total Questions: 109

Total Answers: 107

Location: Kenya

Member since Mon, Jun 14, 2021

3 Years ago

answered 15 Years ago kaleyv · Accepted Answer

You may consider using HTMLunit
It's a java class library made to automate browsing without having to control a browser, and it integrates the Mozilla Rhino Javascript engine to process javascript on the pages it loads. There's also a JRuby wrapper for that, named Celerity. Its javascript support is not really perfect right now, but if your pages don't use many hacks things should work fine the performance should be way better than controlling a browser. Furthermore, you don't have to worry about cookies being persisted after your scraping is over and all the other nasty things connected to controlling a browser (history, autocomplete, temp files etc).