pluginGeek
Web Crawling / ScrapingRuby41,277EditCrawl, spider and scrape websites with Ruby.
Scrapy, a fast high-level web crawling & scraping framework for Python.https://scrapy.org
A Powerful Spider(Web Crawler) System in Python.http://docs.pyspider.org/
Anemone web-spider frameworkhttp://anemone.rubyforge.org
Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.http://felipecsl.com/wombat/
Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...https://github.com/jaimeiniesta/metainspector
A task based API for taking screenshots and scraping text from websites.https://github.com/Netflix/sketchy
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.http://spidr.rubyforge.org/
Ruby gem to inspect completely a web page. It scrapes a given URL, and returns you its meta, links, images more.https://github.com/davidesantangelo/webinspector
A data scraping framework based on Open Civic Data's Pupahttps://github.com/jpmckinney/pupa-ruby
Spider is a Web spidering library for Ruby. It handles the robots.txt, scraping, collecting, and looping so that you can just handle the data.https://github.com/johnnagro/spider
A DSL to write web spider. Depend on capybara and capybara-webkithttps://github.com/zires/micro-spider
A batteries-included framework for easy web-scraping. Just add CSS! (Or do more.)https://github.com/propublica/upton
A simple Ruby web spider that uses Anemone to crawl every page of a site looking for email addresses. Stores the results with SQLite3 using Data Mapper.https://github.com/endymion/email_spider
Ronin Web is a Ruby library for Ronin that provides support for web scraping and spidering functionality.http://ronin-ruby.github.com/
ScrApify is a library to build APIs by scraping static sites and use data as models or JSON APIs. It powers APIfy which is used to create JSON APIs from any html or wikipedia pagehttp://apify.heroku.com/resources
Around the webAdd