Trusted Reviews is supported by its audience. If you purchase through links on our site, we may earn a commission.Learn more.
Understanding antivirus test results
In This Article
Our reviews of thebest free antivirusandbest paid-for antivirussoftware use publicly available data from three anti-malware testing labs: AV-TEST, AV-Comparatives and SE Labs. Here’s how to access and read that test data for yourself.
By way of full disclosure, I worked with the SE Labs team during its earlier incarnation as Dennis Technology Labs. All three testing houses provide a range of services and test results for antivirus makers, enterprise and consumers for a range of platform. Here, I’ll focus on consumer AV testing for Windows.
See the FAQ below for more detail, but ‘real-world’ malware testing involves pointing a browser or email client at a live malware source, while a static ‘reference set’ exposure usually has the anti-malware suite scan a set of known recent malware samples.
Save 81% on a VPN with SurfShark
Surfshark has dropped the price of its VPN to £1.94 a month. Head over to Surfshark now to pay a one time price of £46.44 for 24 months of Surfshark and save 81%.
AV-Comparatives
Based in Austria and founded in 1999,AV-Comparativescarries out tests of consumer and enterprise anti-malware solutions for Windows, macOS and Android. AV-Comparatives typically publishes itsresultsin comparative reports with interactive graphs and tables showing the protection and false positive performance of each antivirus suite in the group test. I focus on its real-world (live exposure) tests for the results we use in Trusted’s antivirus reviews.
Key figures to look for in the firm’s real-world tests are Blocked, which shows how many malicious programs were blocked outright, Compromised, which shows how much malware took hold on the system, and User dependent, which shows how often the antivirus tool asks the user to decided about a potential threat. These results are make up the at-a-glance Protection Rate percentage.
AV-TEST
Founded in 2004, Germany’s AV-TEST carries out a wide range of enterprise and consumer anti-malware testing for Android, macOS and Windows, as well testing the security and performance of other software and devices.
AV-TEST provides at-a-glanceresultsfor each anti-malware suite it tests, using scores out of six for protection, performance, and usability. However, to see how that maps to actual performance, you’ll want the summary test results spreadseets published in the company’s publicly-accessiblePress Area.
For Windows home anti-malware, these show the percentage of blocked real-world and reference set malware exposures, false positive detections of benign software, and the impact on system performance in a range of categories.
Save 81% on a VPN with SurfShark
Surfshark has dropped the price of its VPN to £1.94 a month. Head over to Surfshark now to pay a one time price of £46.44 for 24 months of Surfshark and save 81%.
SE Labs
UK-based SE Labs, founded in 2015, tests security software for the consumer, small business and large enterprise.Reportsare available as downloadable PDFs and include information on the threat landscape, a summary test methodology and a breakdown of the performance of each anti-malware suite.
Protection accuracy ratings are weighted to differentiate between blocked malware, which never gets a hold on the system, and neutralised malware, which gets onto the system but is then successfully removed by the antivirus tool – a Protection Score hashes these together, while detailed charts and graphs show the number of threats detected, blocked, neutralised or compromised. False positive detection of legitimate software is also tested, and weighted based on what kind of interaction is required from the user.
You might like…
FAQ
FAQs
Also known as reference set tests, static malware testing generally involves an on-demand scan of a range of malware, typically introduced to a test system via an external storage medium. This is useful in that it can replicate the kind of infection you might see via a USB drive or even a local network share, but it doesn’t accurately reflect everything going on.The fact that the malware threat samples have to be collected in advantage also means that static tests tend to assess how up-to-date malware detection engines’ signature libraries are, rather than the accuracy of their behavioural or heuristic detection.
‘Real-world’ testing, also known as dynamic testing or live threat exposure is the most realistic kind of malware test. The most simple example would be pointing test systems for each antivirus suite at a website known to attempt a drive-by download or opening an email with an infected attachment and seeing how each antivirus suite responds.In practice, testing houses often use more reproduceable systems, for example by recording and using replays of attacks using an HTTP/S traffic capture tool such as Fiddler2.The accurate collection of logs and recording of results, and the use of a transparent methodology, is critical to the reliability and trustworthiness of such tests, leading to the formation of bodies such as AMTSO.
The Anti-Malware Testing Standards Organisation publishes aprotocol standardfor the testing of anti-malware solutions, with a particular focus on fairness and transparency of the testing process. While it’s not in any way a guarantee of strict accuracy, AMTSO compliant tests provide enough data and information on the testing process to make their results clear and easy to follow.
TheEICAR antivirus Test Fileis a rudimentary “is this thing on?” test for malware detection engines, developed and distributed by the European Institute for Computer antivirus Research. Every antivirus program in the world is configured to detect it as a demonstration of what the detection of a real malicious file would look like.An Eicar test file detected using Clam-TK for LinuxIt’s safe for everyday users to use, and can thus be used to ensure that your antivirus suite is actually working. Itdoes nottest the real malware detection capabilities of your AV in any way whatsoever.If a review of antivirus software relies exclusively or largely on an antivirus suite’s ability to detect EICAR files for its conclusions on the malware detection engine’s effectiveness, you should go and read a different review.
A false positive occurs when antivirus software incorrectly flags up a benign program as a potential threat. The most often occurs with unknown software.
Save 81% on a VPN with SurfShark
Surfshark has dropped the price of its VPN to £1.94 a month. Head over to Surfshark now to pay a one time price of £46.44 for 24 months of Surfshark and save 81%.
K.G. Orphanides is a writer and developer whose areas of expertise include internet security, VPNs, Linux for the desktop, small-scale game development, software preservation and computer audio techno…
Founded in 2003, Trusted Reviews exists to give our readers thorough, unbiased and independent advice on what to buy.
Today, we have millions of users a month from around the world, and assess more than 1,000 products a year.
Editorial independence means being able to give an unbiased verdict about a product or company, with the avoidance of conflicts of interest. To ensure this is possible, every member of the editorial staff follows a clear code of conduct.
We also expect our journalists to follow clear ethical standards in their work. Our staff members must strive for honesty and accuracy in everything they do. We follow the IPSO Editors’ code of practice to underpin these standards.
Why trust our journalism?
Founded in 2003, Trusted Reviews exists to give our readers thorough, unbiased and independent advice on what to buy.
Today, we have millions of users a month from around the world, and assess more than 1,000 products a year.
Editorial independence means being able to give an unbiased verdict about a product or company, with the avoidance of conflicts of interest. To ensure this is possible, every member of the editorial staff follows a clear code of conduct.
We also expect our journalists to follow clear ethical standards in their work. Our staff members must strive for honesty and accuracy in everything they do. We follow the IPSO Editors’ code of practice to underpin these standards.
