LLM & RAG based Cyber Threat Prediction Part. 2 (RAG-XAI Collaboration)

In the previous post, “LLM and RAG-based Cyber Threat Prediction (Part 1)” (link), we examined existing approaches to cyber threat prediction and discussed their inherent limitations. We also introduced a combination of Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG) to understand the flow of cyber attacks and predict subsequent attack stages.

In this blog post, we take a step further toward more effective threat prediction. Specifically, we present a method that integrates cyber threat detection results from T9 Detect, introduced in the T9 Project blog (link), with evidence derived from Explainable Artificial Intelligence (XAI). By leveraging XAI-generated explanations obtained during the detection process, we demonstrate how to extract more relevant contextual information and utilize it for improved cyber threat prediction.

Figure 1. T9 Project Detection Framework

XAI for RAG

Before leveraging XAI in RAG, we first examine the form in which XAI results are provided and what processing is required to utilize them for RAG-based query generation.

Figure 2. XAI-RAG Prediction Concept

Cyber Threat Detection with XAI

The cyber threat detection model (T9 Detect) developed in the T9 Project is designed to analyze network traffic data at the packet, flow, and session levels to detect various types of attacks[1]. In particular, by incorporating Explainable Artificial Intelligence (XAI), the model is designed to provide the underlying evidence behind its detection decisions, enabling the identification of which packet features or communication patterns contributed to the detection of a specific attack.

The evidence derived from XAI for attack detection can be leveraged to generate queries in a RAG-based cyber threat analysis process. Because network packets contain a wide range of fields and data, directly using them may introduce unnecessary or irrelevant information; however, by focusing only on the key evidence and features identified through XAI, queries can be constructed around information that is highly relevant to the actual attack. This approach enables the retrieval of more precise and contextually relevant information from the RAG dataset, ultimately improving the LLM’s ability to interpret the attack flow and predict the most likely subsequent attack stages.

Semantic Based Token Reconstruction for XAI

Figure 3. Before and After Semantic Token Reconstruction

The results provided by XAI primarily consist of tokens and their corresponding contribution scores[2]. However, in the case of conventional BERT-based tokenizers, a single semantic unit can be fragmented into multiple tokens when representing network traffic data[3]. For example, numerical values such as session length or packet size, as well as specific network field values, may be split into several pieces during the tokenization process, leading to fragmentation of their original semantic meaning and potential distortion in interpretation. This token fragmentation can cause XAI outputs to be misinterpreted as information unrelated to actual network characteristics if used without proper processing, making it a critical factor in securing explainability in threat detection and prediction tasks [4].

Therefore, we perform a Semantic Token Reconstruction process that recombines fragmented tokens from XAI outputs to restore the original network feature units. Specifically, among the tokens split by a BERT-based tokenizer, we identify consecutive tokens corresponding to the same network feature and merge them into a single semantic unit [5]. Through this process, network statistical values such as session length (SESSION), average packet size (AVG_PKT), and average inter-arrival time (AVG_IAT), as well as field values like IP addresses and port numbers, can be reconstructed in a form that preserves their original semantic meaning.

The XAI results reconstructed through semantic token reconstruction play a crucial role in the RAG-based query generation process. As illustrated on the left side of Figure 3, when network features are represented as fragmented tokens, the meaning of the search query can become distorted. In contrast, by utilizing tokens restored as coherent semantic units, as shown on the right, queries can be constructed around key features that are directly related to actual attack behaviors. This enables more accurate retrieval of threat intelligence from the RAG dataset and ultimately improves the LLM’s ability to interpret attack scenarios and predict subsequent attack stages in a more reliable and effective manner.

RAG-XAI Collaboration

Figure 4. Example of RAG Query using XAI

As described earlier, the XAI results reconstructed into semantic units are utilized for query generation in RAG-based cyber threat analysis. To ensure that only meaningful features are used, as illustrated in Figure 4, we select only those tokens whose contribution scores exceed a predefined threshold based on the token-level contributions provided by XAI[6]. This process is intended to extract only the network features that have a significant impact on attack detection and incorporate them into the RAG query.

The queries constructed in this manner are used to retrieve the most relevant threat intelligence from the RAG dataset, and the retrieved results serve as additional context in the LLM’s reasoning process[7]. The LLM then comprehensively analyzes both the XAI-derived network features and the attack knowledge retrieved through RAG to interpret attack scenarios associated with the currently observed behaviors, and based on this analysis, it presents the attack stages that are most likely to occur next to the user.

Conclusion

So far, we have explored methods for leveraging XAI-based attack detection results in RAG-based cyber threat analysis. In particular, we introduced a process in which XAI outputs provided alongside detection results are reconstructed into original network feature units through semantic token reconstruction, and then used to generate RAG queries. This approach enables the effective utilization of evidence identified by the detection model to retrieve threat intelligence that is highly relevant to actual attacks, and to integrate this information into the LLM’s reasoning process.

In the next post, we will present the overall architecture and practical implementation of a cyber threat prediction system based on LLM and RAG, including the integration of XAI and RAG introduced in this article. We hope you look forward to it.

References

[1] T9 Project의 세 번째 공격 데이터(T9 Attack) 공개와 사이버 위협 탐지(T9 Detect) 프리뷰, KAIST CSRC 블로그, 2025
[2] Adadi, Berrada, Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence(XAI), IEEE Access, 2018
[3] Devlin, Jacob, et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL, 2019
[4] Sennrich, et al., Neural Machine Translation of Rare Words with Subword Units, ACL, 2016
[5] Jain et al., Attention is not Explanation, NACCL 2019
[6] Lundberg, Scott, Lee, A Unified Approach to Interpreting Model Predictions, NeurIPS, 2017
[7] LLM 및 RAG 기반 사이버 위협 예측 1탄(이해하기), KAIST CSRC 블로그, 2025

2 명이 이 글에 공감합니다.