Tuesday, May 22, 2012

Ruby scraper. How to export to CSV?

I wrote this ruby script to scrape product info from the manufacturer website. The scraping and storage of the product objects in an array works, but I can't figure out how to export the array data to a csv file. This error is being thrown:
scraper.rb:45: undefined method `send_data' for main:Object (NoMethodError)



I do not understand this piece of code. What's this doing and why isn't it working right?



  send_data csv_data, 
:type => 'text/csv; charset=iso-8859-1; header=present',
:disposition => "attachment; filename=products.csv"


Full code:



#!/usr/bin/ruby

require 'rubygems'
require 'anemone'
require 'fastercsv'

productsArray = Array.new

class Product
attr_accessor :name, :sku, :desc
end

# Scraper Code

Anemone.crawl("http://retail.pelicanbayltd.com/") do |anemone|
anemone.on_every_page do |page|

currentPage = Product.new

#Product info parsing
currentPage.name = page.doc.css(".page_headers").text
currentPage.sku = page.doc.css("tr:nth-child(2) strong").text
currentPage.desc = page.doc.css("tr:nth-child(4) .item").text

if currentPage.sku =~ /#\d\d\d\d/
currentPage.sku = currentPage.sku[1..-1]
productsArray.push(currentPage)
end
end
end

# CSV Export Code

products = productsArray.find(:all)
csv_data = FasterCSV.generate do |csv|
# header row
csv << ["sku", "name", "desc"]

# data rows
productsArray.each do |product|
csv << [product.sku, product.name, product.desc]
end
end

send_data csv_data,
:type => 'text/csv; charset=iso-8859-1; header=present',
:disposition => "attachment; filename=products.csv"




No comments:

Post a Comment