594 Open source projects that are alternatives of or similar to Py3_scripts

爬虫, http代理, 模拟登陆!

💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

ScrapingOutsourcing专注分享爬虫代码 尽量每周更新一个

A multi-thread crawler framework with many builtin image crawlers provided.

Lite version of Crawlab. 轻量版 Crawlab 爬虫管理平台

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台，支持任何语言和框架

Scrapy-based Crawlers for news of Taiwan

scrapy专利爬虫（停止维护）

Scrapoxy hides your scraper behind a cloud. It starts a pool of proxies to send your requests. Now, you can crawl without thinking about blacklisting!

《数据采集从入门到放弃》源码。内容简介：爬虫介绍、就业情况、爬虫工程师面试题 ；HTTP协议介绍； Requests使用 ；解析器Xpath介绍； MongoDB与MySQL； 多线程爬虫； Scrapy介绍 ；Scrapy-redis介绍； 使用docker部署； 使用nomad管理docker集群； 使用EFK查询docker日志

HTTP API for Scrapy spiders

码云仓库链接:AJay13/ECommerceCrawlers Github 仓库链接:DropsDevopsOrg/ECommerceCrawlers 项目展示平台链接:http://wechat.doonsec.com

Scrapy as a Web Crawler for Azure Search Samples

Redis-based components for Scrapy.

[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

Parser and database to index the terpene profile of different strains of Cannabis from online databases

基于Scrapy的QQ音乐爬虫(QQ Music Spider)，爬取歌曲信息、歌词、精彩评论等，并且分享了QQ音乐中排名前6400名的内地和港台歌手的49万+的音乐语料

A Facebook crawler

A framework for creating semi-automatic web content extractors

Some scrapy and web.py exmaples

Simple but useful Python web scraping tutorial code.

DotnetCrawler is a straightforward, lightweight web crawling/scrapying library for Entity Framework Core output based on dotnet core. This library designed like other strong crawler libraries like WebMagic and Scrapy but for enabling extandable your custom requirements. Medium link : https://medium.com/@mehmetozkaya/creating-custom-web-crawler-with-dotnet-core-using-entity-framework-core-ec8d23f0ca7c

Github 仓库及用户分析爬虫

crawler framework, distributed crawler extractor

基于搜狗微信搜索的微信公众号爬虫接口

Crawlera middleware for Scrapy

Dynamic file detection tool based on crawler 基于爬虫的动态敏感文件探测工具

Python爬虫实战 - 模拟登陆各大网站 包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

PTT 網路版爬蟲

swiss army knife for hackers

Python + Scrapy + MongoDB . 5 million data per day !!!💥 The world's largest website.

Distributed crawler powered by Headless Chrome

python爬虫

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Faster requests on Python 3

Scrapy middleware to handle javascript pages using selenium

The Prime Cross Site Request Forgery (CSRF) Audit and Exploitation Toolkit.

Word2vec 千人千面 个性化搜索 + Scrapy2.3.0(爬取数据) + ElasticSearch7.9.1(存储数据并提供对外Restful API) + Django3.1.1 搜索

Possibly the best practice of Scrapy 🕷 and renting a house 🏡

京东商品价格监控：监控用户设定商品价格，降价邮件/微信提醒。技术：Python爬虫/IP代理池/JS接口爬取/Selenium页面爬取

支援 PTT 還有 PTT2 的 PTT API

带你了解一下Golang的市场行情

Random User-Agent middleware based on fake-useragent

两只蠢萌京东的分布式爬虫.

豆瓣电影top250、斗鱼爬取json数据以及爬取美女图片、淘宝、有缘、CrawlSpider爬取红娘网相亲人的部分基本信息以及红娘网分布式爬取和存储redis、爬虫小demo、Selenium、爬取多点、django开发接口、爬取有缘网信息、模拟知乎登录、模拟github登录、模拟图虫网登录、爬取多点商城整站数据、爬取微信公众号历史文章、爬取微信群或者微信好友分享的文章、itchat监听指定微信公众号分享的文章

Doujinshi downloader 绅士漫画下载

a new crawler based on python with more function including Network fingerprint search

Python入门网络爬虫之精华版

A collection of awesome web crawler,spider in different languages

🐨实时监控1000家中国企业的新闻动态

[Unmaintained] A simple and clean video/music/image downloader 👾

A high performance web crawler in Elixir.

TweetScraper is a simple crawler/spider for Twitter Search without using API

🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载。

use multiple proxies with Scrapy

一个超级轻量的百度图片爬虫

Declarative web scraping

Crawler for Nintendo Switch eShop

✌️ Python3 BitTorrent DHT crawler