Sunday, May 12, 2024
 Popular · Latest · Hot · Upcoming
96
rated 0 times [  97] [ 1]  / answers: 1 / hits: 45684  / 15 Years ago, sun, september 13, 2009, 12:00:00

I am writing a screen scraper script in python with module 'mechanize' and I would like to use the mechanize.click_link() method on a link that has javascript:__doPostBack in href.
I believe the page I am trying to parse is using AJAX.



Note: mech is the mechanize.Browser()



>>> next_link.__class__.__name__
'Link'
>>> next_link
Link(base_url='http://www.citius.mj.pt/Portal/consultas/ConsultasDistribuicao.aspx', url=javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$lnkNext',''), text='2', tag='a', attrs=[('id', 'ctl00_ContentPlaceHolder1_Pager1_lnkNext'), ('title', 'Pxc3xa1gina seguinte: 2'), ('href', javascript:__doPostBack('ctl00$ContentPlaceHolder1$Pager1$lnkNext',''))])
>>> req = mech.click_link(next_link)
>>> req
<urllib2.Request instance at 0x025BEE40>
>>> req.has_data()
False


I would like to retrieve the page source after clicking the link.


More From » python

 Answers
26

I don't use mechanize, but I do a lot of web scraping myself with python.


When I run into a javascript function like __doPostBack, I do the following:

I access the web site in Firefox, and use the HttpFox extension to see the parameters of the POST request the browser sent to the web server when clicking the relevant link.

I then build the same request in python using urllib.parse.urlencode to build the query strings and POST data I need.

Sometimes the website uses cookies as well, so I just use python's http.cookiejar.


I have used this technique successfully several times.


[#98707] Tuesday, September 8, 2009, 15 Years  [reply] [flag answer]
Only authorized users can answer the question. Please sign in first, or register a free account.
everardo

Total Points: 406
Total Questions: 104
Total Answers: 92

Location: Albania
Member since Sun, Nov 22, 2020
4 Years ago
everardo questions
Fri, Dec 24, 21, 00:00, 2 Years ago
Thu, Jan 21, 21, 00:00, 3 Years ago
Fri, Jul 31, 20, 00:00, 4 Years ago
;