Part 2. Functionality and Performance Evaluation of Antivirus For 2024

In the last article, “Part 1. Functionality and Performance Evaluation of Antivirus For 2024,” we introduced the process for selecting AV products for evaluation and a new approach to analyzing effective functionality and performance. This post presents a functionality and performance analysis study for private or public versions of antivirus products based on the five criteria specified below.

1) Select an antivirus product for home users
2) Build a user experience-centered functionality and performance analysis environment
3) Analyze detection performance by malware family
4) Measure real-time detection accuracy
5) Analyze the detection performance of mutant malware based on generative AI

Generative AI to Create Mutant Malware

To generate mutant malware, we used MalRNN, released about four years ago, to generate and test malware. MalRNN’s malware generation method consists of the following three steps.

① Perform data sampling by converting binary data categorized as benign and malicious to hexadecimal.
② Generates a new byte sequence at the end of the existing malware’s byte sequence.
③ Generate malware using own deep learning detection model to check whether it is detected or not.

Figure 1. MalRNN generation structure and procedure

Only about 15% of all original malware could be generated by MalRNN, and the remaining 85% could not be generated due to technical issues. We verified that the generated mutant malware is a different file by comparing the hash value of the original malware. Moreover, we also verified that the generated mutant malware operates the same as the real file by executing it. Through detailed analysis, we confirmed that the overall operation of the process, network, file access, registry, etc., is the same, and the malware generation is entirely successful.

Figure 2. (top) Process and network execution comparison, (bottom) Registry execution comparison
How to Choose and Test Antivirus

We selected 10 antivirus products out of 15 candidates for testing based on the following three criteria.
• (Criteria #1) For private user
• (Criteria #2) For Windows OS
• (Criteria #3) Support for real-time detection

Figure 3. How to choose antivirus for a test

A malware sample is configured as follows to effectively evaluate an antivirus product’s functionality and performance.

First, a sample of 2,000 malware, 200 of each of 10 families (Trojan, Worm, Virus, Spyware, etc.), randomly selected from 20,000 malware collected over the past year. These were chosen randomly with no specific conditions and used to test the antivirus’s real-time detection performance when malware is downloaded to the PC.

The second is a sample of 421 executable malware extensions. We categorized the executable malware extensions as EXE, HTML, PDF, etc., and excluded JS, DLL, and Bat extensions. These samples have been pre-run and analyzed in an isolated environment to verify that they are behavioral malware. They are used to test the real-time detection performance of antivirus software.

Third, a sample of mutant malware generated using generative AI is used to test whether generative AI can create a variant of malware and detect it compared to the original malware.

• (Sample #1) 2,000 malware per family, randomly selected from 20,000 collected within the past year
• (Sample #2) 421 executable malware
• (Sample #3) 149 mutant malware generated using generative AI

Figure 4. (top) Build a test bed environment (bottom) Develop scenario-specific test tools and systems

The test PCs used for functionality and performance evaluation all had the same specifications and OS and were configured to be as similar as possible to actual personal user PCs by installing applications used by individual users, such as messenger(KakaoTalk), compression application(Alzip), and text editor(MS Office), and were tested with minimal user intervention using the Controller and Agent developed by ourselves.

The antivirus products’ functionality and performance analysis results show a detection difference for each product, as seen in Figure 5. For more detailed analysis results, please refer to our upcoming paper (the paper name/link will be posted after publication in September 2024).

Figure 5. Download malware detection (top left), malware execution detection (top right), malware detection per family (bottom)
Conclusion

In this article, we discussed the methods and results of analyzing the functionality and performance of antivirus products using generative AI-based mutant malware and family-specific malware. We found that generative AI is still insufficient to bypass antiviruses effectively, but the rapid growth of technology makes it necessary to continue researching. We also found that the detection rate of malware per family shows that detection deviations appear depending on the characteristics of the detection engine of each antivirus product. The research team will continue to develop various test methods to analyze the functionality and performance of antiviruses and share the results.

Reference

[1] https://csrc.kaist.ac.kr/blog/2024/03/04/안티바이러스-기능-및-성능-분석-1부/
[2] https://github.com/johnnyzn/MalRNN

1 명이 이 글에 공감합니다.

답글 남기기

이메일 주소는 공개되지 않습니다. 필수 필드는 *로 표시됩니다