Comments on: Tips on Writing a Scraper

By: Joseph

Joseph — Mon, 28 Jan 2008 16:45:15 +0000

DO you know of any example php scripts that could help me learn. I want to learn how to grab links, images, and content. Thanks

By: rob

rob — Thu, 24 Jan 2008 11:54:06 +0000

Nice post Would you keep [a] tags and style tags etc in place in the content or would you strip them out – or by taking ‘everything you can’ do you mean scraping everything in the [body] tags?

I’ve tried using perl’s HTML::TreeBuilder module to strip out content from [p] tags, but that massacres any line breaks etc a bit too much.

By: Brent

Brent — Wed, 16 Jan 2008 00:28:45 +0000

Good stuff. I’d say though that Google isn’t even that good at rooting out scraper sites. Especially programming related sites. There’s a ton of forum type programming sites in their index. Maybe the code syntax throws off their ability to detect scraped content.

By: » Von Oldschool Spam und Lamazüchtern | seoFM - der erste deutsche PodCast für SEOs und Online-Marketer

Tue, 15 Jan 2008 19:55:57 +0000

[...] Tipps fürs Scraper bauen – Gute Tipps von Dave! [...]

By: david

david — Sat, 12 Jan 2008 00:16:20 +0000

1) Extremely rare.

2) Most of the time its easier. The java applet or Flash app is running on YOUR computer. That means its talking to a server and exchanging data. Sniff that traffic, figure out how to mimic it, and then connect to the server with your application that harvests data.

By: Kristian Schmidt

Kristian Schmidt — Sat, 12 Jan 2008 00:11:26 +0000

Regarding the last line, what about flash files? Or, god forbid, java applets?

How would you go about scraping those?

By: Andre

Andre — Fri, 11 Jan 2008 20:20:19 +0000

Good info, I’ve recently wrote several in PHP. Good point about grabbing all of the data you can, why waste time going back.