Exploring the Trade-Off Space of Hierarchical Scheduling for Very Large HPC Centers
Student: Stephen Herbein (University of Delaware)
Supervisor: Dong H. Ahn (Lawrence Livermore National Laboratory)
Abstract: Hierarchical batch scheduling has many advantages, but a lack of trade-off studies against traditional scheduling precludes the emergence of effective solutions for very large HPC centers. Under hierarchical scheduling, any job can instantiate its own scheduler to schedule the sub-jobs with a distinct policy. While such scheduling parallelism and specialization are attractive for the expanding scale and resource diversity of the centers, hierarchical resource partitioning can lead to poor resource utilization. In this poster, we explore the trade-off space between scheduling complexity and resource utilization under hierarchical scheduling. Our approach includes novel techniques to create a hierarchical workload automatically from a traditional HPC workload. Our preliminary results show that one additional scheduling level can reduce scheduling complexity by a factor of 3.58 while decreasing resource utilization by up to 24%. Our study, therefore, suggests that poor utilization can happen under hierarchical scheduling and motivates dynamic scheduling as a complementary technique.
Two-page extended abstract: pdf