Gukhua Lee, Joon Woo, Taeyoung Hong, Journal of Internet Computing and Services, Vol. 25, No. 5, pp. 41-51, Oct. 2024
10.7472/jksii.2024.25.5.41, Full Text:
Keywords: High-Performance Computing, Customized container image, Image manager, Supercomputing environment

Abstract

This paper introduces a novel approach for managing and building customized container images in high-performance computing (HPC) environments, addressing the growing need for flexibility, scalability, and efficiency in computational workflows. Our contributions include the development and integration of a custom container image manager and builder within a container-based HPC infrastructure. This system enables users to effortlessly create, manage, and deploy personalized AI service platforms, significantly enhancing the user experience by reducing the time and effort required to configure essential packages and frameworks. The image manager we developed is capable of processing multiple user requests concurrently, distributing tasks efficiently to image builders operating on compute nodes. Meanwhile, the image builder is designed to handle queued tasks, generate customized container images based on active instances, and store these images in a private container registry, ensuring seamless access and reusability. We validated our system's effectiveness by implementing it on HPC cluster-based systems, including the Nurion supercomputer and the Neuron GPU system, demonstrating its scalability and interoperability in real-world environments. Additionally, we established an architecture and mechanism that ensures seamless integration with existing container-based supercomputing frameworks, underscoring our system's capability to optimize resource utilization and streamline the deployment of AI service platforms.

Statistics
Cite this article