Monday, June 3, 2024
 Popular · Latest · Hot · Upcoming
178
rated 0 times [  179] [ 1]  / answers: 1 / hits: 18159  / 6 Years ago, sun, february 4, 2018, 12:00:00

I am performing web scraping to grab data from a website as part of my project. I can make the request and grab the data which is present in the dom. However, some data is getting rendered on javascript onClick function.


One way could be, using the selenium to click on the link (which calls the javascript function) and grab the rendered data, but this process is time-consuming, and I don't want to open the browser.


Is there any way other than selenium to achieve this?


Website: http://catalog.fullerton.edu/preview_entity.php?catoid=16&ent_oid=1849


In the courses section of this webpage, all the courses are hyperlinks, and as soon as someone clicks on the courses, a javascript method gets called. I need the data which gets rendered after the javascript function call.


More From » python

 Answers
13

You can't. If you want to run JavaScript, you'll need to use a headless browser. Otherwise, you'll have to disassemble the JavaScript and see what it does.



Click on the element while your browser's developer tools are open in the Network tab:



enter



You can now see that the JavaScript downloads new HTML from that URL. You can easily send the same request with urllib.


[#55269] Thursday, February 1, 2018, 6 Years  [reply] [flag answer]
Only authorized users can answer the question. Please sign in first, or register a free account.
quinn

Total Points: 160
Total Questions: 86
Total Answers: 101

Location: Belarus
Member since Tue, Mar 14, 2023
1 Year ago
quinn questions
;