This weekend I was going through the logs for The Market Ticker and Tickerforum, and noted that a few rogue "robots" had managed to come at the site with some very high-rate download requests (basically "spidering" the site) while evading the general rules and policies of good conduct on the Internet.

That is, they were essentially grabbing pages as fast as the system could return them (which is very fast), a rather anti-social practice.

Forum spam and data-mining have become a rather serious problem over the last couple of years and the volume of this sort of crap has increased substantially, with the worst offenses coming from (no surprise) places like China, India and (surprise!) Europe.

Tickerforum has a number of "defensive" measures it employs to stop abusive practices; it is able, for example, to detect "robotic" attempts to register accounts (a common spammer tactic; register an account and then spam away, all automatically without a human involved), and will "ip ban" the source addresses where such attempts come from.  There are also a number of well-placed "Honey Pot" traps that will never be seen by a human, but a robot will find them, and if they're followed that draws an instant ip ban as well.

Nonetheless this is basically a cyber arms race, where the malefactors develop their weapons, you improve your shields, they rotate their phaser frequencies a bit, you rotate your shields, and so on.  I've done pretty well keeping most of these guys out -- many other forums and discussion areas on the web, not so much.

So this weekend I added a "rate monitor" tracking system that is entirely content-insensitive and adaptive, with the intent that it should be able to automatically detect attempts to play the game I had caught and squash it in real-time, barring subsequent attempts by blocking the originating IP address.  But after coding up a first pass and turning it on, then watching the results for a while, I found something else interesting in the logs it generated.

There's a new "kid in town" in the spidering game -- and it's not the ordinary sort of "search engine."  These are data miners who are not looking to provide a public service such as Google or Yahoo does, but rather are spidering sites for the purpose of selling "brand management."

This is something I believe that online forum operators need to think about from a policy perspective.  After all, we have a database of messages that has value (otherwise nobody would come visit us, right?)  These folks are mining our data and then selling this data to third parties -- not making it available for free, but rather marketing the intelligence they gather -- without compensating us!

I have a problem with that and if you run one of these sites you should too.

If you do a google search for "Web Reputation Management" you will find a huge number of firms that are plying this trade. I wondered where they were getting the data they were using -- whether they were, when contracted, going out and performing searches at that time, then doing their thing to "counter" whatever the client might not like.

That's not their model folks. 

They are data mining your site on a continual basis and then selling that data to other people.  They are adding to your system and network load on a continual basis and yet you receive zero value from these "hits" as they are not people, they're robots and simply grabbing your data store and analyzing it for their paying customers.

You ought to be getting paid for this; you're certainly not getting value in-kind as you do with real humans -- whether by whatever subscription or donation model you may have, from viewed advertising or from that person's contribution to the discussions underway. 

Nope -- these folks are doing nothing other than taking and reselling your data, without your consent and in many cases in direct violation of your terms of service and compilation copyrights. And they're not doing it to provide a search function to the public at no charge to -- they're doing it to compile a proprietary database of information they intend to keep and sell.

I'm going to implement a means to block this sort of thing here, since I've now found that I can identify the firms doing it with a high degree of success.   And I believe that others who run sites similar to this one should do so as well.

Let's put a stop this sort of business model.