Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

How do you guys manage masking the IP address when you want to scrape using your python script?


I find there is really no need to hide or mask the IP address when web scraping. The use of proxies or Tor to do so is completely unnecessary and maybe prohibitive e.g. try using Google in Tor.


When you are hitting sites thousands of times you have to make yourself as human like and anonymous as possible. Which isn't even as hard as it sounds. Just your address, user agent and random timers are the three most important things in botting.

http://www.blackhatunderground.net/forum/the-deep-web/9-blac...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: