Enhancing job scheduling with adaptive prediction model based on user profiling

Abstract

The recent advances in Grid and Cloud Computing have brought about a number of challenges on computing resource management. In particular, the ability to fulfil the Service Level Agreement (SLA) and Quality of Service (QoS) on job scheduling. Service providers are obliged to meet the user’s expectation that stated in the SLA. A job scheduler’s efficiency is graded based on how well the user’s expectations are met. However, the user’s Requested Time is at best an estimate and fraught with inaccuracies to begin with. This thesis focuses on developing a new runtime prediction algorithm to improve the scheduling efficiency rather than merely relying on the user’s Requested Time. In this work, an analysis on the job submission characteristics has been carried out. The result revealed that there is a trend in the job submission characteristics. An adaptive approach is used to categorize those trends into different profiles that lead to similar predictable behaviour. A novel user profile-aware method, Runtime Prediction using Dynamic Weighted Moving Average (RP-DWMA) in making runtime prediction is proposed. Based on the simulation results on 11 production workloads obtained from Grid and supercomputer, RP-DWMA has successfully improved the scheduling efficiency. The results demonstrated an average of 41.2% performance improvement with inclusive of the resubmission cost and an acceptable average error rate of about 12.4%. Furthermore, RP-DWMA is able to adapt quickly to the dynamic changes in the submission patterns with less overhead and performs well in handling user inaccuracies in runtime estimates. In conclusion, RP-DWMA is well suited for job submission environment that requires service-oriented approaches.