Scrapy

python -m pip install -U pip setuptools

代理

爬取/购买代理,然后自验证

cat proxy.txt while read line;do curl -x $line https://www.baidu.com -m 5 –connect-timeout 5 -o /dev/null -s -w “$line “%{http_code}”\n”;done>~/ip2.txt
cat ip2.txt awk ‘{if($2==200)print $1}’ >ip-good.txt

购买动态代理

搭建动态代理

Yan Peipan 25 May 2015