長(zhǎng)效ip在Python scrapy框架的教程 使用固定長(zhǎng)效代理ip的使用教程
時(shí)間:2023-01-27 來(lái)源:楓之葉網(wǎng)絡(luò)
最近楓之葉網(wǎng)絡(luò)客服遇到客戶用python語(yǔ)言使用scrapy框架用長(zhǎng)效ip代理來(lái)進(jìn)行大數(shù)據(jù)采集,以下是小編匯總的scrapy的demon:
import base64
# 代理服務(wù)器
proxyServer = "http://www.xbltjxp.cn:65535"
# 代理隧道驗(yàn)證信息
proxyUser = "pachongdaili"
proxyPass = "pachongdaili"
# for Python2
proxyAuth = "Basic " + base64.b64encode(proxyUser + ":" + proxyPass)
# for Python3
#proxyAuth = "Basic " + base64.urlsafe_b64encode(bytes((proxyUser + ":" + proxyPass), "ascii")).decode("utf8")
class ProxyMiddleware(object):
def process_request(self, request, spider):
request.meta["proxy"] = proxyServer
# 適配 scrapy 2.6.2
request.meta["_auth_proxy"] = proxyServer
request.headers["Proxy-Authorization"] = proxyAuth
request.headers["Connection"] = "close"
當(dāng)前地址:http://www.xbltjxp.cn/support/a466.html 客服聯(lián)系QQ:475685360
最近楓之葉網(wǎng)絡(luò)客服遇到客戶用python語(yǔ)言使用scrapy框架用長(zhǎng)效ip代理來(lái)進(jìn)行大數(shù)據(jù)采集,以下是小編匯總的scrapy的demon:
import base64
# 代理服務(wù)器
proxyServer = "http://www.xbltjxp.cn:65535"
# 代理隧道驗(yàn)證信息
proxyUser = "pachongdaili"
proxyPass = "pachongdaili"
# for Python2
proxyAuth = "Basic " + base64.b64encode(proxyUser + ":" + proxyPass)
# for Python3
#proxyAuth = "Basic " + base64.urlsafe_b64encode(bytes((proxyUser + ":" + proxyPass), "ascii")).decode("utf8")
class ProxyMiddleware(object):
def process_request(self, request, spider):
request.meta["proxy"] = proxyServer
# 適配 scrapy 2.6.2
request.meta["_auth_proxy"] = proxyServer
request.headers["Proxy-Authorization"] = proxyAuth
request.headers["Connection"] = "close"
當(dāng)前地址:http://www.xbltjxp.cn/support/a466.html 客服聯(lián)系QQ:475685360