Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center

Monday, 27th September 2010, 10:30 am (PDCC Meeting Room)
Speaker: Assoc/Prof. Ion Stoica (UC Berkeley)
Title: Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center

Today's datacenters need to support a variety of applications, and an even higher variety of dynamically changing workloads. In this talk, I will present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI. Sharing improves cluster utilization and avoids per-framework data replication. Mesos shares resources in a fine-grained manner, allowing frameworks to achieve data locality by taking turns reading data stored on each machine. To support the sophisticated schedulers of today's frameworks, Mesos introduces a distributed two-level scheduling mechanism, called resource offers. Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to schedule on these resources. Our experimental results show that Mesos can achieve near-optimal locality when sharing the cluster among diverse frameworks, can scale up to 50,000 nodes, and is resilient to node failures.


Ion Stoica received his PhD from the Carnegie Mellon University in 2000. He is an Associate Professor in the EECS Department at University of California at Berkeley, where he does research on network architectures and distributed systems. Stoica is the recipient of the 2007 CoNEXT Rising Star Award, a Sloan Foundation Fellowship (2003), a Presidential Early Career Award for Scientists & Engineers (PECASE) (2002), and the ACM doctoral dissertation award (2001). He also serves as a CTO at Conviva, which he co-founded in 2006.