AI models have advanced by leaps and bounds in recent years, but as they have grown ever larger, they have also taken on a new challenge: the enormous amount of computation they require. According to Stanford HAI’s The 2025 AI Index Report[1], the compute poured into training major AI models has been growing explosively of late, roughly doubling every five months.
Figure 1 Training compute trends of major AI models by developer (2003–2024)[1]
The problem is that this massive computation is not confined to the training stage. At the inference stage, when a trained model is actually run, performing the operations and storing and moving intermediate results also takes a great deal of energy[2].
The resource burden arising from computation, memory use, and data movement becomes a major constraint when high-performance AI models are deployed in real-world environments. A model that runs perfectly well on a server may slow down or run out of memory on a smartphone or other small device, and it may draw more power and heat up. The model-execution constraints seen in on-device AI environments are a prime example of the resource problems that arise when large AI models meet real-world deployment.
So building high-performance AI is only half the job — making the model run efficiently in its actual environment matters just as much. This work of reducing an AI model’s resource consumption and overcoming its execution-efficiency limits is called AI model optimization.
AI optimization can be thought of in terms of two stages: training and inference. At the training stage, what matters is reducing the computation, memory, time, and power needed to train the model. At the inference stage, the focus is on reducing the model size, runtime memory, processing time, and power consumption needed to run the trained model in its actual environment.
In this article, we focus on inference-stage optimization — getting a trained AI model to run on real devices. There are many approaches to inference optimization, and in some cases the model undergoes additional training to adapt to the structure or operations that changed during optimization. The point of this extra training is not to cut training costs such as compute or training time; it is to help the model retain as much of its performance as possible in its optimized form while running inference on fewer resources. Which optimization method to choose, how to apply it, and whether to pair it with additional training can all vary with the model’s architecture and characteristics, the target device, and the runtime environment.
Our research team is also studying optimization approaches that consider both the model and the runtime environment together, so that the AI models we develop in-house can be used efficiently across a variety of environments. Before introducing that research in detail, let us first look at what AI model optimization means, what counts as success, and the most representative methods.
What Is at the Heart of AI Model Optimization?
To understand what optimization achieves, it helps to first look at what drives a model’s resource usage. An AI model stores a huge number of values that were adjusted during training to reflect the patterns in the data and the relationships between inputs and outputs. These are called parameters, and the more parameters a model has, the larger its file generally becomes — and storing or running the model takes hardware with capacity to match.
While a model processes an input, each step produces intermediate results called activations — and even if the model file itself is small, large activations can demand a lot of memory at runtime. Beyond that, the amount and kind of operations the model performs, and the process of moving parameters and intermediate data around, also affect processing time and power consumption[2]. So when optimizing an AI model, you need to consider not only parameters, activations, computation, and data movement, but also actual runtime memory, processing time, power consumption, and heat.
Just because a metric computed from the model’s structure — like parameter count or operation count — has gone down does not mean the costs in the actual environment fall by the same amount. Depending on the combination of model architecture, target hardware, and execution engine, the real memory usage, processing time, and power consumption can all differ.
For example, even if you cut the theoretical operation count dramatically, processing time may not drop as much as expected if the target device or execution engine does not support the changed operations efficiently. Conversely, even without a large reduction in parameters or operations, reducing data movement and switching to a structure that favors memory access and parallel processing can cut actual processing time substantially[3]. Runtime memory and power consumption are likewise shaped by the model’s structure and the execution environment[2].
For these reasons, the effect of optimization must ultimately be verified in the environment where the model will actually be used. Compare the model before and after optimization on the same data and input sizes, on the same device and runtime, and measure what matters in that environment — runtime memory, processing time, power consumption, and so on. For real-time services, processing time may be what counts; on devices with limited memory or battery, runtime memory, power consumption, and heat may be the more important criteria.
Figure 2 Conceptual diagram of AI model optimization with the runtime environment in mind
So, if runtime memory, processing time, and power consumption have all fallen enough in the real environment, can we call the optimization a success? These items correspond to the cost we set out to reduce, and in cost terms the result is certainly meaningful. But that alone is not enough. However few resources the model uses, if it can no longer perform its original function properly, it is hard to put to practical use.
The yardstick for checking this is quality. Quality means how well the optimized model performs the functions its purpose requires. For example, if an AI that separates people from the background runs very fast but frequently misses faces or bodies, it is hard to call that a successful optimization.
Whether optimization has succeeded must therefore be judged on cost and quality together. Improving both at once is the ideal, but in practice you often have to find the right balance between the two. Apply optimization aggressively to push costs down further and quality may drop; to keep quality high, you may have to accept a certain amount of cost. The key is not to blindly minimize or maximize either side, but to balance cost and quality to fit the actual purpose.
Figure 3 AI model optimization that weighs quality and cost together
In short, inference-stage AI model optimization is the process of first setting the quality level you need and the cost you can accept in a given environment, and then refining the model to satisfy both.
Methods for Optimizing AI Models
There are many ways to optimize an AI model. The most representative techniques are quantization, pruning, and knowledge distillation[4].
Quantization lowers the precision of the numbers used to represent parameters and activations. For example, converting numbers stored in 32 bits down to 8 bits reduces the model file size and memory usage. If the target device supports low-precision arithmetic efficiently, processing time and power consumption can come down as well[5]. Lower the precision too far, though, and the error introduced by representing numbers more coarsely can degrade quality.
Pruning removes parameters or connections of relatively low importance from the model — much like trimming unnecessary branches from a tree. This reduces the parameter count and the amount of computation[6]. Remove too many elements, however, and the model’s quality can suffer. And if the removal leaves behind an irregular structure, processing time on real devices may not improve much even though the parameter count has fallen.
Knowledge distillation has a small model learn the way a large, high-performing model makes its judgments. The large model is called the teacher model and the small one the student model. The student learns not only from the correct answers but also from the teacher’s outputs, achieving higher quality than it would have by training alone[7].
Because knowledge distillation requires training the student model separately, it comes with additional training cost. And if the student model’s size or structure is too constrained, it cannot fully absorb the teacher’s judgment. The purpose of this training, however, is not to cut training costs — it is to produce the small, efficient model that will actually be used for inference.
Table 1 Comparison of representative AI model optimization techniques
| Optimization method | Principle | What it means | Characteristics | Pros | Cons |
| Quantization |  | Reduce numerical precision | Can be applied without changing the structure Effective when low-precision operations are supported | Smaller model size Less memory Possible speed and power gains | Quality degrades if precision is cut too far Hardware-dependent |
| Pruning |  | Remove low-importance elements | Structure depends on the unit of removal | Fewer parameters Less computation Simpler model structure | Quality degrades if too much is removed Sparse structures give limited speed gains |
| Knowledge Distillation |  | Transfer a large model’s knowledge to a small model | Student model trained separately Only the student model is used at inference | Higher quality than training the small model alone | Additional training cost required Limited if the student model is too small |
Beyond applying individual techniques to an existing model, there are also approaches that make the model architecture itself more efficient, or that design a lightweight model from the start. Costly operations can be replaced with simpler ones, and the arrangement of layers and channels can be adjusted. Another option is to design a small model from scratch, choosing structures the target device can process efficiently[8]. Simplify the structure too far, though, and quality can suffer — and designing and training a new model is a burden of its own.
These methods are sometimes used one at a time, but they are often applied in combination. You might simplify the model’s structure and then apply quantization, or prune away parts of the structure and then use knowledge distillation. If the optimized model falls short of the target quality, you can retrain the modified model through fine-tuning, or fold the errors introduced by quantization into the training process to shore up quality.
Which methods to combine, and how strongly to apply them, depends on the model and its environment. Each method differs in which costs it can reduce and how it affects quality, so the methods and their intensity should be tuned to the quality and cost criteria set earlier.
Decisions about optimization methods and how strongly to apply them can be made not only for the model as a whole but at the module level as well. AI models often consist of multiple modules with different roles, and each module differs in how much computation and memory it uses and how much it affects final quality. Modules with high costs can be reviewed as priority targets for optimization, while quality-sensitive modules can keep more of their original structure. It is also possible to choose different optimization methods and intensities to suit each module’s characteristics.
After optimizing a module, you should evaluate not only the module’s own ability to do its job, but also the quality of the final output once it is connected back into the whole model or service. Even if an individual module’s performance dips somewhat, if the impact on the final result is small and the cost savings are large, it can count as a practical optimization. Conversely, even with strong average performance metrics, if the system repeatedly misses elements that matter in the actual task, it is hard to call the result a good fit.
You also need to confirm that the optimization gains carry through the entire execution pipeline. A real AI service runs preprocessing and postprocessing, data conversion, memory copies, and more alongside model inference, so if any of these becomes a new bottleneck, it must be improved as well. Optimization, then, is not a matter of applying one technique uniformly — it is a process of adjusting methods and intensity to the characteristics of the model and its modules, and verifying that the whole system meets the quality and cost targets.
Figure 4 Example of selecting and evaluating per-module optimization strategies
Conclusion
In this article, we looked at the criteria for judging whether AI model optimization has succeeded, and at the representative methods: quantization, pruning, knowledge distillation, and efficient architecture design.
The goal of AI model optimization is not to make the model as small as possible. The heart of it is optimizing the model so that, within a given environment, it maintains the target quality while meeting an acceptable cost budget.
In the next article, we will show how the criteria and methods covered here are applied in actual research, with concrete examples — stay tuned!
References
[1] Stanford Institute for Human-Centered Artificial Intelligence, The 2025 AI Index Report, 2025.
[2] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, “Efficient Processing of Deep Neural Networks: A Tutorial and Survey,” Proceedings of the IEEE, 2017.
[3] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, “ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design,” European Conference on Computer Vision (ECCV), 2018.
[4] Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “A Survey of Model Compression and Acceleration for Deep Neural Networks,” arXiv:1710.09282, 2017.
[5] B. Jacob et al., “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[6] S. Han, H. Mao, and W. J. Dally, “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding,” International Conference on Learning Representations (ICLR), 2016.
[7] G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” NIPS Deep Learning Workshop, 2015.
[8] M. Tan et al., “MnasNet: Platform-Aware Neural Architecture Search for Mobile,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
KAIST 사이버보안연구센터 AI 기술보안팀 선임연구원으로 AI 및 LLM을 이용한 AI 모델 경량화와 보안 연구를 수행하고 있다.
KAIST 사이버보안연구센터 사이버위협분석팀 연구원으로 AI에이전트 보안과 위협모델링 연구를 수행하고 있다.