Question

100

Detect Search Crawlers via JavaScript

rated 0 times [ 102] [ 2] / answers: 1 / hits: 47658 / 11 Years ago, tue, november 19, 2013, 12:00:00

I am wondering how would I go abouts in detecting search crawlers? The reason I ask is because I want to suppress certain JavaScript calls if the user agent is a bot.

I have found an example of how to to detect a certain browser, but am unable to find examples of how to detect a search crawler:

/MSIE (d+.d+);/.test(navigator.userAgent); //test for MSIE x.x

Example of search crawlers I want to block:

Google 

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 

Googlebot/2.1 (+http://www.googlebot.com/bot.html) 

Googlebot/2.1 (+http://www.google.com/bot.html) 



Baidu 

Baiduspider+(+http://www.baidu.com/search/spider_jp.html) 

Baiduspider+(+http://www.baidu.com/search/spider.htm) 

BaiDuSpider

Answers

Only authorized users can answer the question. Please sign in first, or register a free account.

aileent

Add To Favorites

Follow

Total Points: 556

Total Questions: 107

Total Answers: 101

Location: Croatia

Member since Fri, Sep 11, 2020

4 Years ago

aileent questions

1 Next.js, getStaticProps not working with component but does with page

Tue, Mar 1, 22, 00:00, 2 Years ago

1 Why am I getting Failed to load extension. Invalid value for 'content_security_policy'?

Fri, Apr 16, 21, 00:00, 3 Years ago

1 Laravel livewire model is not working when updating input field from javascript?

Sun, Nov 29, 20, 00:00, 4 Years ago

1 How to get Input field type as props in react typescript

Wed, Nov 11, 20, 00:00, 4 Years ago

1 React Native Render error missing semicolon

Fri, May 8, 20, 00:00, 4 Years ago

View All

answered 11 Years ago jacie · Accepted Answer

This is the regex the ruby UA agent_orange library uses to test if a userAgent looks to be a bot. You can narrow it down for specific bots by referencing the bot userAgent list here:

/bot|crawler|spider|crawling/i

For example you have some object, util.browser, you can store what type of device a user is on:

util.browser = {

   bot: /bot|googlebot|crawler|spider|robot|crawling/i.test(navigator.userAgent),

   mobile: ...,

   desktop: ...

}