人妖在线一区,国产日韩欧美一区二区综合在线,国产啪精品视频网站免费,欧美内射深插日本少妇

新聞動(dòng)態(tài)

Python抓取今日頭條街拍圖片數(shù)據(jù)

發(fā)布日期:2022-02-02 08:19 | 文章來(lái)源:腳本之家

(1)抓取今日頭條街拍圖片

(2)分析今日頭條街拍圖片結(jié)構(gòu)

keyword: 街拍
pd: atlas
dvpf: pc
aid: 4916
page_num: 1
search_json: {"from_search_id":"20220104115420010212192151532E8188","origin_keyword":"街拍","image_keyword":"街拍"}
rawJSON: 1
search_id: 202201041159040101501341671A4749C4
可以找到規(guī)律,page_num從1開(kāi)始累加,其他參數(shù)不變

(3)按功能不同編寫(xiě)不同方法組織代碼

獲取網(wǎng)頁(yè)json格式數(shù)據(jù)

def get_page(page_num):
 global headers
 headers = {
  'Host': 'so.toutiao.com',
  #'Referer': 'https://so.toutiao.com/search?keyword=%E8%A1%97%E6%8B%8D&pd=atlas&dvpf=pc&aid=4916&page_num=0&search_json={%22from_search_id%22:%22202112272022060101510440283EE83D67%22,%22origin_keyword%22:%22%E8%A1%97%E6%8B%8D%22,%22image_keyword%22:%22%E8%A1%97%E6%8B%8D%22}',
  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36',
  'X-Requested-With': 'XMLHttpRequest',
  'Cookie': 'msToken=S0DFBkZ9hmyLOGYd3_QjhhXgrm38qTyOITnkNb0t_oavfbVxuYV1JZ0tT5hLgswSfmZLFD6c2lONm_5TomUQXVXjen7CIxM2AGwbhHRYKjhg; _S_DPR=1.5; _S_IPAD=0; MONITOR_WEB_ID=7046351002275317255; ttwid=1%7C0YdWalNdIiSpIk3CvvHwV25U8drq3QAj08E8QOApXhs%7C1640607595%7C720e971d353416921df127996ed708931b4ae28a0a8691a5466347697e581ce8; _S_WIN_WH=262_623'
 }
 params = {
  'keyword': '街拍',
  'pd': 'atlas',
  'dvpf': 'pc',
  'aid': '4916',
  'page_num': page_num,
  'search_json': '%7B%22from_search_id%22%3A%22202112272022060101510440283EE83D67%22%2C%22origin_keyword%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22image_keyword%22%3A%22%E8%A1%97%E6%8B%8D%22%7D',
  'rawJSON': 1,
  'search_id': '2021122721183101015104402851E3883D'
 }
 url = 'https://so.toutiao.com/search?' + urlencode(params)
 print(url)
 try:
  response=requests.get(url,headers=headers,params=params)
  if response.status_code == 200:
  #if response.content:
#print(response.json())
return response.json()
 except requests.ConnectionError:
  return None

從json格式數(shù)據(jù)提取街拍圖片

def get_images(json):
 images = json.get('rawData').get('data')
 for image in images:
  link = image.get('img_url')
  yield link

將街拍圖片以其md5碼命名并保存圖片

實(shí)現(xiàn)一個(gè)保存圖片的方法 save_image(),其中 item 就是前面 get_images() 方法返回的一個(gè)字典。在該方法中,首先根據(jù) item 的 title 來(lái)創(chuàng)建文件夾,然后請(qǐng)求這個(gè)圖片鏈接,獲取圖片的二進(jìn)制數(shù)據(jù),以二進(jìn)制的形式寫(xiě)入文件。圖片的名稱可以使用其內(nèi)容的 MD5 值,這樣可以去除重復(fù)。相關(guān)代碼如下:

def save_image(link):
 data = requests.get(link).content
 with open(f'./image/{md5(data).hexdigest()}.jpg', 'wb')as f:#使用data的md5碼作為圖片名
  f.write(data)

main()調(diào)用其他函數(shù)

def main(page_num):
 json = get_page(page_num)
 for link in get_images(json):
  #print(link)
  save_image(link)

(4)抓取20page今日頭條街拍圖片數(shù)據(jù)

這里定義了分頁(yè)的起始頁(yè)數(shù)和終止頁(yè)數(shù),分別為 GROUP_START 和 GROUP_END,還利用了多線程的線程池,調(diào)用其 map() 方法實(shí)現(xiàn)多線程下載。

if __name__ == '__main__':
 GROUP_START = 1
 GROUP_END = 20
 pool = Pool()
 groups = ([x for x in range(GROUP_START, GROUP_END + 1)])
 #print(groups)
 pool.map(main, groups)
 pool.close()
 pool.join()
import requests
from urllib.parse import urlencode
from hashlib import md5
from multiprocessing.pool import Pool
def get_page(page_num):
 global headers
 headers = {
  'Host': 'so.toutiao.com',
  #'Referer': 'https://so.toutiao.com/search?keyword=%E8%A1%97%E6%8B%8D&pd=atlas&dvpf=pc&aid=4916&page_num=0&search_json={%22from_search_id%22:%22202112272022060101510440283EE83D67%22,%22origin_keyword%22:%22%E8%A1%97%E6%8B%8D%22,%22image_keyword%22:%22%E8%A1%97%E6%8B%8D%22}',
  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36',
  'X-Requested-With': 'XMLHttpRequest',
  'Cookie': 'msToken=S0DFBkZ9hmyLOGYd3_QjhhXgrm38qTyOITnkNb0t_oavfbVxuYV1JZ0tT5hLgswSfmZLFD6c2lONm_5TomUQXVXjen7CIxM2AGwbhHRYKjhg; _S_DPR=1.5; _S_IPAD=0; MONITOR_WEB_ID=7046351002275317255; ttwid=1%7C0YdWalNdIiSpIk3CvvHwV25U8drq3QAj08E8QOApXhs%7C1640607595%7C720e971d353416921df127996ed708931b4ae28a0a8691a5466347697e581ce8; _S_WIN_WH=262_623'
 }
 params = {
  'keyword': '街拍',
  'pd': 'atlas',
  'dvpf': 'pc',
  'aid': '4916',
  'page_num': page_num,
  'search_json': '%7B%22from_search_id%22%3A%22202112272022060101510440283EE83D67%22%2C%22origin_keyword%22%3A%22%E8%A1%97%E6%8B%8D%22%2C%22image_keyword%22%3A%22%E8%A1%97%E6%8B%8D%22%7D',
  'rawJSON': 1,
  'search_id': '2021122721183101015104402851E3883D'
 }
 url = 'https://so.toutiao.com/search?' + urlencode(params)
 print(url)
 try:
  response=requests.get(url,headers=headers,params=params)
  if response.status_code == 200:
  #if response.content:
#print(response.json())
return response.json()
 except requests.ConnectionError:
  return None
def get_images(json):
 images = json.get('rawData').get('data')
 for image in images:
  link = image.get('img_url')
  yield link

def save_image(link):
 data = requests.get(link).content
 with open(f'./image/{md5(data).hexdigest()}.jpg', 'wb')as f:#使用data的md5碼作為圖片名
  f.write(data)def main(page_num):
 json = get_page(page_num)
 for link in get_images(json):
  #print(link)
  save_image(link)if __name__ == '__main__':
 GROUP_START = 1
 GROUP_END = 20
 pool = Pool()
 groups = ([x for x in range(GROUP_START, GROUP_END + 1)])
 #print(groups)
 pool.map(main, groups)
 pool.close()
 pool.join()

到此這篇關(guān)于Python抓取今日頭條街拍圖片數(shù)據(jù)的文章就介紹到這了,更多相關(guān)Python抓取今日頭條圖片內(nèi)容請(qǐng)搜索本站以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持本站!

美國(guó)服務(wù)器租用

版權(quán)聲明:本站文章來(lái)源標(biāo)注為YINGSOO的內(nèi)容版權(quán)均為本站所有,歡迎引用、轉(zhuǎn)載,請(qǐng)保持原文完整并注明來(lái)源及原文鏈接。禁止復(fù)制或仿造本網(wǎng)站,禁止在非www.sddonglingsh.com所屬的服務(wù)器上建立鏡像,否則將依法追究法律責(zé)任。本站部分內(nèi)容來(lái)源于網(wǎng)友推薦、互聯(lián)網(wǎng)收集整理而來(lái),僅供學(xué)習(xí)參考,不代表本站立場(chǎng),如有內(nèi)容涉嫌侵權(quán),請(qǐng)聯(lián)系alex-e#qq.com處理。

相關(guān)文章

實(shí)時(shí)開(kāi)通

自選配置、實(shí)時(shí)開(kāi)通

免備案

全球線路精選!

全天候客戶服務(wù)

7x24全年不間斷在線

專屬顧問(wèn)服務(wù)

1對(duì)1客戶咨詢顧問(wèn)

在線
客服

在線客服:7*24小時(shí)在線

客服
熱線

400-630-3752
7*24小時(shí)客服服務(wù)熱線

關(guān)注
微信

關(guān)注官方微信
頂部