A Processing Element aware Map-Reduce framework for heterogeneous cluster environment

Friday, 21st January 2011, 2:00 pm (PDCC Meeting Room)
Speaker: Tan Yu Shyang (PDCC)
Title: A Processing Element aware Map-Reduce framework for heterogeneous cluster environment
 
Abstract:

In this talk, I will present, Pamar a Processing Element Aware Map-Reduce Framework, for supporting distributed computing on clusters with heterogeneous processing elements configuration, e.g. different configuration of CPU and GPGPU for each node. Pamar can identify both the processing elements available to the different nodes in the cluster and the processing element requirements for each user submitted jobs, with minimum or even no user intervention in the code. I will also present briefly HPE, a custom resource aware scheduler, for Pamar to perform the scheduling of jobs with different processing elements requirements to the different nodes with different processing elements. Our team integrate HPE and Pamar into Hadoop so as to demonstrate such a framework. Finally I will show that Pamar can provide improvements to the job queue completion time in heterogeneous clusters via scheduling tasks to the appropriate nodes with the respective processing elements.


 
Biography: