Lee, Jung Hyun

Thank you for giving me the opportunity to introduce myself!

I am a self-motivated researcher at NAVER Cloud in the Foundation Research team. Before joining NAVER Cloud, I worked as a software engineer at Samsung Research. Prior to that, I completed my M.S. from the Graduate School of AI at KAIST, where I was fortunately advised by Prof. Eunho Yang, and did my B.S. in mathematics from POSTECH.

You are more than welcome to contact me anytime, if you want to know more about my work :-)

onliwad101 at gmail dot com  /  CV  /  Google Scholar  /  GitHub  /  LinkedIn

By leveraging trillions of high-quality tokens during pre-training, recent pre-trained models have accumulated extensive knowledge across multiple domains and languages. However, they still struggle to elicit the appropriate knowledge required for complex reasoning tasks, resulting in limitations in their reasoning abilities. To address this challenge, my current work focuses on post-training and optimally scaling inference-time compute to enhance mathematical reasoning capabilities. Specifically, I am dedicated to refining a new verifier to more effectively utilize additional test-time computation by improving the prediction of which partial solutions are on the right track toward a correct final answer during search. Additionally, I am exploring the generation of synthetic data for post-training by using different search techniques with the verifier. This approach could pave the way for self-improvement when both generator and verifier originate from the same pre-trained model.

Previously, my research primarily focused on streamlining the inference and fine-tuning of large language models (LLMs). Specifically, my expertise developed in the areas of LLM quantization and parameter-efficient fine-tuning for quantized LLMs.

We presented token-supervised value models (TVMs), new token-level verifiers trained to estimate the probability of reaching the correct final answer for each token in a solution.

We proposed a new post-training weight quantization method for LLMs, LRQ that learns low-rank weight-scaling matrices instead of dense ones to decrease learnable parameters, thus enhancing the generalization capability of quantized LLMs. As a result, Llama and Llama 2 models can be quantized via LRQ with a minor impact on the accauracy of commonsense reasoning tasks and MMLU.

We introduced a quantization-aware parameter-efficient fine-tuning technique, PEQA that fine-tunes only the quantization step sizes of quantized LLMs to (i) reduce both the model size and the number of training parameters during fine-tuning, and (ii) accelerate inference latency after fine-tuning. Therefore, on a single NVIDIA A100-80GB GPU, Llama 65B can be fine-tuned via PEQA and then accelerated.

We developed a new weight-rounding mechanism, FlexRound that can flexibly quantize pre-trained weights of not only computer vision models but also language models based on the magnitude of each weight. Specifically, Llama models can be quantized via FlexRound with marginal performance degradation on both commonsense reasoning tasks and causal language modeling.

We formulated Cluster-Promoting Quantization (CPQ), which promotes cohesive clustering of full-precision weights around quantization grids. On top of that, we devised DropBits that drops bits rather than neurons/filters to lower the bias resulting from our multi-class straight-through estimator used in CPQ. We quantized ResNet-18 and MobileNetV2 on ImageNet with low-bit weight-activation quantization schemes.

Conference reviewer

• AAAI Conference on Artificial Intelligence (AAAI): 2025
• Association for Computational Linguistics (ACL) Rolling Review: 2024
• International Conference on Machine Learning (ICML): 2024
• International Conference on Learning Representations (ICLR): 2024, 2025
• Neural Information Processing Systems (NeurIPS): 2022, 2023, 2024

• Research Track TOP 2, N INNOVATION AWARD 2023 (NAVER's internal technology excellence awards ceremony)

Website template credit: Jon Barron