Definition and need of mobile antivirus
Mobile Antivirus(AV) refers to a software developed for a mobile device to detect and block malicious applications and suspicious threat behaviors. StatCounter’s report states that from last year, Android OS has been the most used worldwide, including personal computer OS such as Microsoft’s Windows and Apple’s macOS. In other words, Android OS has the highest market share across all operating systems. Figure 1. shows StatCounter’s statistics on OS market share worldwide. Android could become the most popular OS because it allows developers to join Android OS projects to improve or upgrade the system freely, like open-source projects, so the OS can update bugs or newer services quickly. These user-participated open OSes can be stably optimized and give less restrictions for the users. However, this open-source-based operating system has become a favorite target for hackers. Malware apps that threaten Android smartphones can be categorized into six main types based on attack methods and purposes.
Figure 1. Operating system market share worldwide [1]
| • Trojan – Misleads users of its true intent by disguising itself as a normal program • Ransomware – A type of cryptovirological malware that permanently blocks access to the victim’s personal data unless a ransom is paid • Spyware/Worm – It aims to gather information about a person or organization and send it to another entity in a way that harms the user by violating their privacy, endangering their device’s security, or other means • Banking Trojan – It is designed to collect online banking credentials and other sensitive information from infected machines • Downloader on Android – After installing a malware app induces downloading and installing additional malicious app(s) on a mobile device • Fake App – It mimics official app stores or imitates popular apps to trick users into downloading malicious apps |
Mobile AVs, developed to protect against various malware users’ smartphones, works like legacy AVs for PCs. However, due to the limited resources of mobile devices, mobile AVs use a detection method that sends the malicious app’s metadata to a cloud server for analysis and lighter detection methods than PC AVs. Due to their performance limitations, false positives and false negatives due to incorrect metadata transfer can often occur. Therefore, based on these issues and the characteristics of mobile devices, our research team developed and established test scenarios to analyze the functionality and performance of mobile AVs.
What’s essential about mobile antivirus?
1. UX(User xperience) oriented design
The user experience of a mobile AV is one of the most important factors. Users want a convenient and intuitive user experience, so mobile AV should consider their needs, providing them with an intuitive and efficient interface and considering the responsiveness and stability of the app.
2. Platform & device compatibility
It should work on various OSs (iOS, Android, etc.) and devices (smartphones, tablets, etc.). Compatibility for different platforms, including various device screen sizes and resolutions, should be considered to ensure stable operation.
3. Performance optimization
Mobile devices have limited hardware resources (memory, battery, network bandwidth, etc.), so they must be optimized for performance. Mobile AV is required to minimize memory and battery usage by optimizing resource occupancy for background tasks, network requests, and more.
4. Malware detection performance
The most crucial feature of mobile antivirus is the capability of detecting malicious apps. With so much sensitive information being handled on mobile, such as users’ personal and payment information, malware detection is essential to protect from stealing and performing malicious behavior.
5. Continuous update and maintenance
Mobile AV requires continuous automated engine maintenance, updates, and performance improvements to resist newer malware and security vulnerabilities.
We designed and established functional and performance evaluation considering various essential factors in mobile AV. In addition, we conducted newer test methods which has not been conducted by existing mobile AV test institutes, such as SD memory card inspection, large-scale malicious app inspection, and recently collected malware app inspection. Malware apps subject to the tests were provided by financial and security-related institutes and our crawling system.
The Mobile AV candidates for functionality and performance analysis were selected based on the following criteria.
| • Criterion 1. AVs with high domestic popularity and downloads on Google Play Store • Criterion 2. Reliable product with a 3.5+ star rating • Criterion 3. Products that can be installed on Android OS and tablets |
Under the above three criteria, we chose 10 mobile AVs, including V3 Mobile Security, Alyac M, ESET Mobile Security, and SK Shielders Mobile Guard, as listed below.
| Mobile Antivirus | Version | Mobile Antivirus | Version |
| V3 Mobile Security | 3.8.0.9 | Norton 360 | 5.76.0.231201002 |
| Alyac M | 3.0.4.7 | Avira Antivirus Security | 7.22.0 |
| ESET Mobile Security | 8.2.15.0 | Bitdefender Mobile Security | 3.3.224.2368 |
| SK Shielders Mobile Guard | 23.1.2 | Kaspersky Standard | 11.109.4.11153 |
| Avast Mobile Security | 23.24.0 | Malwarebytes Mobile Security | 5.3.4 |
Table 1. Version of Mobile AV
Malware apps collected from finance institutes, security-related institutes, and our own crawling system were organized into a total of six sample groups according to the evaluation scenarios to analyze the functionality and performance of mobile AV.
| • Sample 1. 1,000 malware apps collected by malicious type: phishing, spyware, financial, etc. • Sample 2. 10,000 malware apps from the last 3 years selected by random sampling • Sample 3. 1,000 malware apps that occurred within the past year • Sample 4. 5,000 malware apps related to financial phishing in the last 3 years • Sample 5. 100 finance malware apps within 3 months • Sample 6. 140 malware apps within 1 month |
Figure 2. (Top) Mobile AV test environment, (Bottom) Test software (Controller/Agent)
The test environment of the mobile AV was completely separated inside and outside through two firewalls (allowing networks for analysis such as the cloud), as shown in Figure 2(top). Inside the network behind the firewalls, 10 PCs connected to each mobile device, and a controller that delivers commands and a malware app was built. All test PCs and mobile devices used in functional and performance analysis used the same hardware and OS, where the details are shown in Table 1. To minimize human involvement, we developed our own command execution software (Controller/Agent) to execute commands received from the controller (ADB) and deliver malware apps, as shown in Figure 2(bottom). They can simultaneously download and install malware apps to each mobile device via a PC and save test logs of the commands executed by each PC and the files transferred to the DB.
| PC | Mobile Device | ||
| CPU | i5-12400 | Product | Samsung Galaxy Tab A7 Lite |
| Memory | 16GB | Internal Memory | 32GB |
| Storage | SSD 500GB | External Memory | SD Card 128GB |
| OS | Windows10 Pro(64bit) | OS | Android 13 |
Table 2. Mobile antivirus device specifications for testing
Establishment of evaluation criteria and test critical features to measure mobile AV detection performance and functionality
Total of 50 evaluation criteria were established to analyze the functionality and performance of mobile AVs, following the Korean “Software Technical Evaluation Guidelines [2]” and the evaluation items of international AV testing institutes. We conducted 14 detailed performance tests based on six significant scenarios and around 36 additional functionalities. In addition, some of the malware used in the experiments was finance apps (voice phishing, fake apps, etc.) and was only found in Korea.
| Evaluation Categories | Evaluation items | Evaluation Criteria |
| Functionality | Accuracy | (1) Real-time detection accuracy with malware app installations (1,000) |
| (2) Manual scan detection accuracy test of 10,000 malware apps (last 3 years) | ||
| (3) Detection accuracy test of malware apps (1,000) from the past year (since Jul 2022) | ||
| (4) Detection accuracy test of finance phishing malware apps (5,000) over 3 years | ||
| (5) Detection accuracy test of finance malware apps (100) occurring within 3 months (Aug 2023) | ||
| (6) Detection accuracy test of malware apps (140) that occurred within 1 month (Nov 2023) | ||
| Speed | (7) Test speed for the manual scan of 10,000 malware apps (Test 2) | |
| (8) Test speed for recent malware apps (1,000) within 1 year (Test 3) | ||
| (9) Test speed for financial phishing-related malware apps (5,000) over 3 years (Test 4) | ||
| (10) Test speed for financially related malware apps (100) within 3 months (Aug 2023) (Test 5) | ||
| (11) Test speed for malware apps (140) within 1 month (Nov 2023) (Test 6) | ||
| Resource efficiency | Battery usage | (12) Check the battery level during the Test 4 scan. Method: Check the battery remaining after running an antivirus scan for 5,000 malware for 3 minutes from a 100% charge |
| (13) Antivirus background mode battery level Method: Check the battery level after 72 hours from 100% charge | ||
| Temperature | (14) Temperature check during the Test 4 scan Method: Check the temperature change when running an antivirus scan for 5,000 malware for 3 minutes | |
| Usability | Users’ ease of operation | (15) How many languages do products support? |
| (16) Does the product offer manuals? | ||
| Input data support | (17) How many ways (or additional options) do the Quick and Deep Scans offer to specify what to scan? (e.g., email files, other folders, compressed files, etc.) | |
| Ease of understanding progress | (18) Does it provide a UI/UX that makes it easy to understand the scan’s progress being performed? | |
| Installation environment suitability | (19) Does the product installation process not prompt you to install other external programs? | |
| Ease of uninstallation | (20) Is the product easy to install and uninstall? | |
| Report Generation | (21) Does it provide detection and quarantine result reports in a file? | |
| Custom detections and scans | (22) Can users exclude certain conditions (folder, filename, extension, detection name, etc.) from detection and inspection? | |
| Real-time detection | (23) Is it possible to specify a specific location (entire system, specific folder, etc.) for real-time detection? | |
| Add-ons | Manual Scanning | (24) After real-time detection, can the AV set actions (quarantine and delete) for detected malware? |
| (25) How many manual scanning methods does the AV offer? | ||
| (26) Can the AV allow specific locations (files, drives, specific folders, etc.) to be scanned during a manual scan? | ||
| (27) Does it have a scheduled scan feature? | ||
| Network Security | (28) Block harmful sites or manage user-defined sites? | |
| (29) Ability to prevent or detect specific network-based intrusions (spoofing, remote, etc.)? | ||
| (30) VPN or proxy capabilities? | ||
| (31) WiFi management capabilities? | ||
| System Security | (32) Is it possible to see a history of recently installed apps? | |
| (33) Does it have phishing-specific detection or blocking capabilities (URL, SMS, Email)? | ||
| (34) Does it have the ability to clean up and block old (unused) apps? | ||
| (35) Does it have the ability to manage app permissions (location, mic, contacts, call logs, messages, etc.)? | ||
| (36) Does it have an app lock feature? | ||
| (37) Does it have the ability to tamper with the OS and check for rooting? | ||
| Privacy | (38) Does it have a junk file deletion feature? | |
| (39) Does it allow you to delete your browsing history? | ||
| (40) Does it have the ability to delete user history (recently opened files, list of running documents, etc.)? | ||
| (41) Does it have the ability to manage internet banking (check for banking app fraud)? | ||
| (42) Does it have the ability to manage payment information (e.g., PayPal)? | ||
| Others | (43) Does it have a QR scanner scan function? | |
| (44) Does it have the ability to handle exceptions in detection? | ||
| (45) Does it have a function to clean smartphone memory? | ||
| Vendor support | Maintenance | (46) Does the product continue to receive regular product updates and feature additions? |
| Schedule updates | ||
| (47) Does it allow users to control engine (DB) updates automatically or manually? | ||
| Troubleshooting and support | (48) Does the product have a Q&A or FAQ on the homepage or within the AV product? | |
| (49) Does it provide technical support in the Korean language on the homepage? | ||
| (50) Does the provider have a contact support or service (such as a chatbot) that can respond quickly in case of a problem? |
Mobile AV performance results and analysis
Five scenarios were used in the quantitative evaluation, which is the main analytics of the mobile AV test.
• (Test 1) ( Real-time) Real-time detection accuracy test with 1,000 malware app installations
• (Test 2) ( Load Test) Manual scan of 10,000 malware apps to test detection accuracy (last 3 years)
• (Test 3) (Update) Detection accuracy test of malware apps (1,000) from the past year (since Jul 2022)
• (Test 4) (Financial) Detection accuracy test of financial phishing-related malware apps (5,000) over 3 years
• (Test 5) (Trend) Detection accuracy test of finance malware apps (100) occurring within 3 months (Aug 2023)
• (Test 6) (Trend) Detection accuracy test of malware apps (140) that occurred within 1 month (Nov 2023)
The analysis results for the quantitative evaluation are shown in Table 3, with the 10 mobile AV products anonymized and labeled A through J.
| A | B | C | D | E | F | G | H | I | J | |
| Test 1 | 95.57% | 97.22% | 97.01% | 75.28% | 96.29% | 96.60% | 94.75% | 98.04% | 78.48% | 96.81% |
| Test 2 | 99.90% | – | 98.99% | 86.23% | 93.02% | – | 96.85% | 99.67% | 93.06% | 99.98% |
| test 3 | 91.30% | – | 87.60% | 55.80% | 75.10% | – | 88.80% | 94.9% | 86.30% | 91.20% |
| Test 4 | 99.80% | – | 99.62% | 100% | 90.58% | – | 99.98% | 99.98% | 90.74% | 99.98% |
| Test 5 | 100% | – | 100% | 100% | 100% | 100% | 100% | 100% | 100% | 100% |
| Test 6 | 85.71% | – | 91.43% | 87.86% | 90.00% | – | 91.43% | 90.71% | 32.86% | 90.71% |
[Test 1] Real-time detection accuracy test with 1,000 malware app installations
We tested how well mobile AVs detect 1,000 malware apps installed via Agent and Controller. Of the 1,000 malware apps, 29 were not installed due to unknown issue with the app itself, and the most common types of malware detected were trojans and spyware. In addition, four malware apps were detected by only one or two AVproducts and were determined to be normal apps when manually analyzed. This is thought to be a false positive by the AVproducts.
[Test 2] Manual scan of 10,000 malware apps to test detection accuracy
This test evaluates the detection performance of mobile AVs by storing 10,000 malware apps from the last three years on an SD memory card and then running it as a manual scan. The test results showed that some mobile AVs use internal memory for scanning, causing mobile AVs to terminate due to insufficient internal memory, and some products do not support SD memory card scanning. In addition, some products showed increased detection accuracy when installing malware apps in Test 1. This is assumed to be due to the difference in real-time and manual scanning.
[Test 3] Detection accuracy test of malware apps (1,000) from the past year
We tested the detection performance of 1,000 malicious apps from the past year. This test was designed to determine how quickly mobile engines are updated. The results showed that some mobile AVs significantly differed from the detection accuracy in Test 2. This result indicates that such mobile AV engines have longer update cycles.
[Test 4] Detection accuracy test of financial phishing-related malware apps (5,000) over 3 years
We tested detection performance against 5,000 malware apps related to financial phishing over three years. Financial malware can directly harm users’ financial assets, such as their payment information. These tests determine how well mobile AVs prevent someone from stealing sensitive financial information. The results showed that most products had high detection rates.
[Test 5] Detection accuracy test of finance malware apps (100) occurring within 3 months
This is a detection test of 100 recent finance malware apps collected from financial institutes within 3 months. The malware apps used in the test are unknown or recently detected finance malware apps in Korea. The methodology is the same as Test 2, where the test is performed by manually scanning the SD memory card after saving it to the SD memory card. In test 5, we can assume that all antivirus products have 100% detection accuracy, which is very fast for financial malware apps.
[Test 6] Detection accuracy test of malware apps (140) that occurred within 1 month
Test 6 is a detection test for malware apps that occurred within one month. This test shows how quickly the product can update its engine against unknown malware apps. Compared to Test 3, Test 6 shows a general increase in detection accuracy, indicating that most mobile AVs respond quickly. One mobile AV had a significant drop in detection accuracy. We need to analyze this cause in detail, but we can assume that it results from not updating the engine quickly enough.
Conclusion
The ultimate goal of a mobile AVs are to effectively protect users’ mobile devices (phones, tablets, etc.) by detecting all malware. However, based on the analysis presented in this post, it is clear that not every malware can be prevented using a single mobile AV.. The various mobile AVs have their own strengths and weaknesses. While we can’t say which is the best, our research team found a small but significant difference in detection performance.
The cyber threats in the mobile device environment are continuously evolving and diversifying. We suggest the user self-protect the checklist following the box. This checklist will help users to protect their mobile devices.
| • Remember to install a mobile AV product on your mobile device and periodically update and scan the engine. • Do not install apps downloaded from unofficial sources or unreliable websites. • Carefully consider if an app requires excessive permissions unrelated to its function. • If you must install an app from a non-official app store, compare the name and icon to the official app. • Carefully check apps that start automatically when your mobile device boots or runs in the background. • If an app lacks developer information, ratings, or reviews, think carefully before installing it. • Run a mobile AV scan if your mobile device’s data and battery consumption increases significantly. |
Our research team will continue to evaluate PC and mobile AV products annually and publish detailed results, so keep watching our website.
Reference
[1] https://gs.statcounter.com/os-market-share
[2] (과학기술정보통신부 고시 제2021-98호)소프트웨어 기술성 평가기준 지침 일부개정
KAIST 사이버보안연구센터 사이버위협분석팀 연구원으로 블록체인 및 소프트웨어 테스팅 연구를 진행하고 있다.
KAIST 사이버보안연구센터 사이버위협분석팀 연구원으로 악성코드 분석 프로그램 및 연구를 수행하고 있다.