How to automatically ping search engine crawlers using python.

In today’s post we will see how to make a python bot that will automatically ping the search engine crawlers to come and crawl your website/blog. For this. we will be using the help of ping-o-matic.com.

Ping-o-matic is a website that dose the same thing i.e it pings the search engine crawlers to come and crawl your website/blog. This helps in getting better ranks in the search results. I don’t know if it really works, but it is definitely worth a try.

Once you go into the ping-o-matic website, it will ask you for your website name, website URL, etc. once you hit submit, it will start the pinging. After that it will ask you to bookmark the page so that you don’t have to type in the details again and again. The next time you want to ping, you just have to click on the bookmark.

What we will be doing is, we will be opening the bookmarked URL through python. It is not at all complicated. The coding is very simple. Because most of the work will done by ping-o-matic.

If you have never worked with the web in python, you don’t have to worry this does not require any experience. For opening any URL through python, there are many ways. One is the raw sockets way, which is a lower level of doing it, but it is not necessary. Two there is module called ‘requests’, which allows you to send GET, POST and PUT requests to any web server. Three, is using urllib module for sending a simple GET request.

The syntax for accessing the URL or sending a GET request is very simple. First you have to import the urllib2 module. Then you have to call the function urlopen() with the bookmarked URL as the argument.

That’s about it. That will be as good as clicking the bookmark. But this will only ping once. If we want it to keep sending pings forever, then we will have to run it in an infinite loop. But of course there is a minimum amount of waiting time before you can ping the second time. So we will have to call the urlopen() function after a certain time delay. Just like C/C++ has the delay() function, python has something similar. It has the sleep() function, which comes under the module ‘time’. But in delay() the time you specify as argument is in milliseconds. Whereas in sleep() it is in seconds. Refer to the below picture to see how to use the sleep() function in an infinite loop.

So that is the whole code required. As you can see, there is a time delay of 1800 seconds i.e.  30 min. So the program will open the bookmarked URL every 30 min. you can even increase the frequency of pinging but I don’t think you can go below 10-15 min.

I have added an exception for URL error i.e. if the URL is wrong, of if the site is not responding or any such issue, it will not crash the program and print ‘URL error’ instead.

Now for running this bot, you can either execute it and minimize it in your computer. But you will have to keep it on forever. Or you can host the script on a cloud service like Heroku which will keep it running for you.

I personally use this for this blog, and the results are astonishing. So I recommend that you also try it yourself. I will cover how to use Heroku in another post. Till then you can refer to Youtube for clarity. So that’s it for today’s post, will be back with more useful ideas and concepts. Until then, happy coding.