Helo

Python学习之单线程爬取360看看所有电视剧
  博主最近在跟@小伟学习Python,刚好有一个学习任务是爬取360影视的所有电视剧和电...
扫描右侧二维码阅读全文
28
2019/01

Python学习之单线程爬取360看看所有电视剧

  博主最近在跟@小伟学习Python,刚好有一个学习任务是爬取360影视的所有电视剧和电影的图片,播放链接,评分,名字。这个项目使用分布式+队列来做最好,但是我还没学到,所以勉强使用单线程来做一做看。以下代码基于Python3,只爬取了360看看电视剧板块的全部电视剧的名字、评分、图片链接及播放链接。这个代码还有许多不足,等我学一段时间再来改进一下。具体代码如下:

#author = Xiaofeng
import re 
from urllib import request
from bs4 import BeautifulSoup

userAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"

def fact(url):  #主函数
    data={}
    html = request.urlopen(url)
    soup = BeautifulSoup(html.read(),"html.parser")
#获取名字
    for namelist in soup.find_all('h1'):
        name = namelist.get_text()
        data["名字"] = name
#获取评分
    for grade in soup.find_all(attrs={'class':"s-title-right"}):
        sorce=grade.get_text().strip()
        data["评分"] = sorce
#获取播放链接
    for link in soup.find_all(attrs={'class':"g-playicon s-cover-img"}):
        playurl = link.get('href')
        data["播放链接"] = playurl
#获取图片链接
    img = link.find_all(name='img')
    for url_img in img:
        imgurl = url_img.get('src')
        data["图片链接"] = imgurl
    return data

url = []
for i in range(1,26):
    listurl = "https://www.360kan.com/dianshi/list.php?rank=rankhot&cat=all&area=all&act=all&year=all&pageno="+"{i}".format(i=i)
    html = request.urlopen(listurl)
    soup = BeautifulSoup(html.read(),"html.parser")
#获取详细页的url
    for postfix in soup.find_all(attrs={'class':"js-tongjic"}):
        inurl = 'https://www.360kan.com'+postfix.get('href')
        url.append(inurl)
num = len(url)
for i in range(num):
    a=fact(url[i])
    print('名字:',a["名字"])
    print('评分:',a["评分"])
    print('图片链接:',a["图片链接"])
    print('播放链接:',a["播放链接"])
    print('-'*40)

  以下是运行时的部分截图:

Last modification:September 19th, 2019 at 12:05 am
If you think my article is useful to you, please feel free to appreciate

Leave a Comment