Map Reduce is a programming model and an associated implementation for processing and generating large data sets with a parallel, distributed algorithm on a cluster. Hadoop is an open-source implementation of Map Reduce enjoying wide adoption and is often used for short jobs where low response time is critical. Hadoop’s performance is closely tied to its task scheduler implicitly assumes that cluster nodes are homogeneous and tasks make progress linearly, and uses these assumptions to decide when to speculatively re-execute tasks that appear to be stragglers. In practice, the homogeneity assumptions do not always hold. Specifically this occurs in a virtualized data center, such as Amazon’s Elastic Compute Cloud (EC2).but that the Hadoop’s scheduler can cause severe performance degradation in heterogeneous environments. To address this problem, a new scheduling algorithm Longest Approximate Time to End (LATE) that is highly robust to heterogeneity and it leads to improvement in response time.
