How good is your antispyware? Can yours "detect 40,000 parasite definitions"? Can yours search for 53,248 spyware components?" Only 22, 984? Wimp! But wait, how can the wimp be rated 5th best out of 20 in a comparative review? What are we counting here? Are we all using base 10 arithmetic?
If you're confused by the disparities in claims of numbers of spyware detected, and nervous that the antispyware software you just purchased doesn't measure up, join the club. Dozens of antispyware software companies are waging war on two fronts. To the east, software engineers and spyware hunters battle against spyware developers. To the west, marketing wonks wage a competition amongst themselves to catch your attention and ultimately, another sale. In our "super-size it" society, what better way than to pile up the statistics?
In host and network security technology circles, numbers are never more deceiving than when they are applied to intrusion and malicious code detection. In the case of spyware, the numbers are doubly deceptive. As you may have already surmised, the first deception lies in what's counted as spyware. I know of no standard definition of what constitutes "one spyware" (if you find one, send it to Congress). Is each ad cookie one spyware instance? Each OLE object? DLL? Executable programs? Can we count a program stored on disk and a process running in memory as two instances? Is each registry item added by a spyware installer package one spyware instance? What if the spyware changes a registry item: can we count that? What if two spyware use the same registry value or substitute their own DLL for a legitimate one? Can I count my competitors by labeling them scamware?
Common sense tells me that this is all nonsense and borders on deceptive advertising. But how often does common sense prevail in a competitive market? When I asked several antispyware vendors how they counted, I discovered "what counts as spyware" is quite a hot button. So I decided I'd comparing how antispyware vendors count spyware myself. I also decided to forego formal, methodical testing. Instead, I would "inspect my system for spyware" the way an average consumer might.
The one test area where I did impose some rigor was the method of infection. I asked Aluria Software's Research, Analysis, and Response Team to provide me with some spyware samples. I visited sites RAR suggested to further infest the test PC. [Disclosure: I have done consulting work for Aluria Software and earned their trust. Under normal circumstances, they do not distribute spyware samples.]
I began with a laptop running a clean install of Windows XP SP2 and downloaded ten "free antispyware scanners" at random. I installed each scanner and disabled any active protection provided by the product. I ran a full system scan from each scanner to be certain they all detected no spyware. This in itself was an interesting exercise, as several products identified competing products as scamware; humorously, some products point accusing fingers at each other. Adjusting for this behavior proved non-trivial. I didn't want to remove the scamware because I was fairly confident these products would help me prove my point. Moreover, I was only interested in obtaining coarse measures, so I simply added the counts of scamware detected to the total counts.
This was a very informal test so I do not intend to publish the product names nor the results. Suffice to say that the range in the numbers of spyware infections reported was between 14 and 187. By my count, the number should have been 19. At the high end, I suspected several false positives but it was evident from the way the scan results were presented that the objective were to deceive and persuade the consumer to purchase the product. Spread Fear, Uncertainty and Doubt and ye shall profit.
Now that I had a basis for comparison, what conclusions could I draw? The first is that raw numbers of spyware detected are deceiving. Without standards for what constitutes one (1) spyware infection, it's impossible to say whether one scanner is superior to another. Without certification to assure that products comply with such standards and hence compete on a level field, deception is too often rewarded: unsophisticated users can easily be misled or frightened into purchasing products that claim to detect the most spyware. Lastly, new spyware appears frequently, and existing spyware is morphed to evade detection even more frequently. Counting can actually conceal the fact that a product isn't keeping pace with new threats.
Rating antispyware products based on claims of the number of spyware detected diverts attention from what I believe is the more important metric for scanning: accuracy. Scanning accuracy is extremely important, especially for large-scale deployments. Accuracy can be measured in terms of false positives and completeness. False positives - crying "Wolf!" - distract IT from productive tasks. If you're about to deploy antispyware to hundreds of desktops, you don't want to be barraged with false alarms. Comprehensiveness of detection is even more important. Products that do not identify all the components of a spyware infection and cannot keep accurate track of components as spyware morphs can be dangerous. In large desktop deployments, you don't want products that perform automated removal upon detection to do The Wrong Thing, remove a required dll, and cause hundreds of PCs to crash and burn.
Ultimately, you run the risk of getting exactly - or less - than you pay for when you rely on the performance of free scanners as the sole basis for purchasing antispyware software or finance it through a software lease. You have no useful basis for comparison. You are also overlooking two equally important features of antispyware software: blocking and removal. Unfortunately, there are too few reliable comparative tests for antispyware, and too many web sites that post contrived and biased test results. If you're searching for effective antispyware software, I recommend you search for product tests performed under editorial supervision by reputable trade publications. A second, reliable source is to follow the leads of Internet Service Providers who offer antispyware software to customers: top-tier ISPs aren't going to recommend or offer software that's going to increase support calls.
Originally published in 2005 by Security Pipeline, reprinted courtesy of CMP Technology and Dark Reading.