Name | Date | Size | #Lines | LOC | ||
---|---|---|---|---|---|---|
.. | 23-Jun-2021 | - | ||||
CHANGELOG.md | H A D | 23-Jun-2021 | 470 | 15 | 11 | |
CONTRIBUTING.md | H A D | 23-Jun-2021 | 4.4 KiB | 68 | 42 | |
README.md | H A D | 23-Jun-2021 | 5.5 KiB | 113 | 73 | |
VERSION | H A D | 23-Jun-2021 | 6 | 2 | 1 | |
colly.go | H A D | 23-Jun-2021 | 35.9 KiB | 1,300 | 983 | |
context.go | H A D | 23-Jun-2021 | 2.2 KiB | 88 | 50 | |
htmlelement.go | H A D | 23-Jun-2021 | 3.8 KiB | 121 | 73 | |
http_backend.go | H A D | 23-Jun-2021 | 5.5 KiB | 228 | 185 | |
request.go | H A D | 23-Jun-2021 | 5.3 KiB | 181 | 121 | |
response.go | H A D | 23-Jun-2021 | 2.7 KiB | 100 | 69 | |
unmarshal.go | H A D | 23-Jun-2021 | 4.6 KiB | 172 | 126 | |
xmlelement.go | H A D | 23-Jun-2021 | 4.6 KiB | 171 | 125 |
README.md
1# Colly 2 3Lightning Fast and Elegant Scraping Framework for Gophers 4 5Colly provides a clean interface to write any kind of crawler/scraper/spider. 6 7With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving. 8 9[![GoDoc](https://godoc.org/github.com/gocolly/colly?status.svg)](https://godoc.org/github.com/gocolly/colly) 10[![Backers on Open Collective](https://opencollective.com/colly/backers/badge.svg)](#backers) [![Sponsors on Open Collective](https://opencollective.com/colly/sponsors/badge.svg)](#sponsors) [![build status](https://img.shields.io/travis/gocolly/colly/master.svg?style=flat-square)](https://travis-ci.org/gocolly/colly) 11[![report card](https://img.shields.io/badge/report%20card-a%2B-ff3333.svg?style=flat-square)](http://goreportcard.com/report/gocolly/colly) 12[![view examples](https://img.shields.io/badge/learn%20by-examples-0077b3.svg?style=flat-square)](https://github.com/gocolly/colly/tree/master/_examples) 13[![Code Coverage](https://img.shields.io/codecov/c/github/gocolly/colly/master.svg)](https://codecov.io/github/gocolly/colly?branch=master) 14[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fgocolly%2Fcolly.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2Fgocolly%2Fcolly?ref=badge_shield) 15[![Twitter URL](https://img.shields.io/badge/twitter-follow-green.svg)](https://twitter.com/gocolly) 16 17 18## Features 19 20 * Clean API 21 * Fast (>1k request/sec on a single core) 22 * Manages request delays and maximum concurrency per domain 23 * Automatic cookie and session handling 24 * Sync/async/parallel scraping 25 * Caching 26 * Automatic encoding of non-unicode responses 27 * Robots.txt support 28 * Distributed scraping 29 * Configuration via environment variables 30 * Extensions 31 32 33## Example 34 35```go 36func main() { 37 c := colly.NewCollector() 38 39 // Find and visit all links 40 c.OnHTML("a[href]", func(e *colly.HTMLElement) { 41 e.Request.Visit(e.Attr("href")) 42 }) 43 44 c.OnRequest(func(r *colly.Request) { 45 fmt.Println("Visiting", r.URL) 46 }) 47 48 c.Visit("http://go-colly.org/") 49} 50``` 51 52See [examples folder](https://github.com/gocolly/colly/tree/master/_examples) for more detailed examples. 53 54 55## Installation 56 57``` 58go get -u github.com/gocolly/colly/... 59``` 60 61 62## Bugs 63 64Bugs or suggestions? Visit the [issue tracker](https://github.com/gocolly/colly/issues) or join `#colly` on freenode 65 66 67## Other Projects Using Colly 68 69Below is a list of public, open source projects that use Colly: 70 71 * [greenpeace/check-my-pages](https://github.com/greenpeace/check-my-pages) Scraping script to test the Spanish Greenpeace web archive 72 * [altsab/gowap](https://github.com/altsab/gowap) Wappalyzer implementation in Go 73 * [jesuiscamille/goquotes](https://github.com/jesuiscamille/goquotes) A quotes scrapper, making your day a little better! 74 * [jivesearch/jivesearch](https://github.com/jivesearch/jivesearch) A search engine that doesn't track you. 75 * [Leagify/colly-draft-prospects](https://github.com/Leagify/colly-draft-prospects) A scraper for future NFL Draft prospects. 76 * [lucasepe/go-ps4](https://github.com/lucasepe/go-ps4) Search playstation store for your favorite PS4 games using the command line. 77 78If you are using Colly in a project please send a pull request to add it to the list. 79 80## Contributors 81 82This project exists thanks to all the people who contribute. [[Contribute]](CONTRIBUTING.md). 83<a href="https://github.com/gocolly/colly/graphs/contributors"><img src="https://opencollective.com/colly/contributors.svg?width=890" /></a> 84 85 86## Backers 87 88Thank you to all our backers! [[Become a backer](https://opencollective.com/colly#backer)] 89 90<a href="https://opencollective.com/colly#backers" target="_blank"><img src="https://opencollective.com/colly/backers.svg?width=890"></a> 91 92 93## Sponsors 94 95Support this project by becoming a sponsor. Your logo will show up here with a link to your website. [[Become a sponsor](https://opencollective.com/colly#sponsor)] 96 97<a href="https://opencollective.com/colly/sponsor/0/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/0/avatar.svg"></a> 98<a href="https://opencollective.com/colly/sponsor/1/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/1/avatar.svg"></a> 99<a href="https://opencollective.com/colly/sponsor/2/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/2/avatar.svg"></a> 100<a href="https://opencollective.com/colly/sponsor/3/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/3/avatar.svg"></a> 101<a href="https://opencollective.com/colly/sponsor/4/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/4/avatar.svg"></a> 102<a href="https://opencollective.com/colly/sponsor/5/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/5/avatar.svg"></a> 103<a href="https://opencollective.com/colly/sponsor/6/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/6/avatar.svg"></a> 104<a href="https://opencollective.com/colly/sponsor/7/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/7/avatar.svg"></a> 105<a href="https://opencollective.com/colly/sponsor/8/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/8/avatar.svg"></a> 106<a href="https://opencollective.com/colly/sponsor/9/website" target="_blank"><img src="https://opencollective.com/colly/sponsor/9/avatar.svg"></a> 107 108 109 110 111## License 112[![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2Fgocolly%2Fcolly.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2Fgocolly%2Fcolly?ref=badge_large) 113