Mesos: The definitive cluster manager

Why Mesos?

Before trying to explain what is Mesos, is necessary to answer the question: Why Mesos?

In recent years modern applications are becoming more and more data-dependent. Storing unstructured data facilitates the collocation of data in various schemas and structural forms for further analysis.

These data lakes are large enough to need to be stored in a data center.

Exponential growth of unstructured data (social media, internet of things, log analytics, …) makes these data lakes distributed among hundreds of commodity servers in a cluster.

This horizontal scaling strategy makes necessary to analyze data with a distributed computing framework.




Any current software architecture has not a single computing framework. Several and different frameworks sharing the same data center is a common situation. This means that stream processing, batch processing, data integration or storage frameworks can share share resources (cpu, ports, disk, …).

For this reason, there is a need for fine grained cluster resource management between multiple frameworks.
In other words, manage the cluster as a single computer. As Ben Hindman said: “we wanted people to be able to program for the data center just like they program for their laptop”.



What is Apache Mesos?

Mesos is an open source cluster manager capable of sharing resources between multiple and different frameworks.



How Does Mesos Work?

These are the main components of Mesos:
Agent: daemon running on cluster node that executes tasks from frameworks and notifies about available node resources
Framework: application that runs on Mesos and solve a specific use case and consists of a scheduler (decides whether to accept or reject resources) and a executor (assigns resources accepted to tasks and controls its execution).
Master: mediates between agents and frameworks.




The Mesos master implements a resource allocation mechanism called resource offers: mesos agents report to master about available resources, master decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them.

Letting frameworks the scheduling decision, mesos simplify the cluster resource management because implementing a centralized scheduler would have exponentially increased as new frameworks were included.

The communication between mesos components is provided by libraries that implements APIs and make mesos extensible to new frameworks.




Resource allocation is not enough without resource isolation. Mesos must ensure that the amount of resources allocated for any task will not be able to be consumed by another task. Mesos resource isolation mechanism is based on a containerizer API that supports a wide range of containerizer implementations (docker containers, linux containers).




Mesos was made for resource cluster management and when it comes to managing a cluster, high availability must be guaranteed with fault tolerance:
Master: thanks to zookeeper and a list of inactive masters ready to leader election if the active one has a failure.
Agent: if agent fails, master notices and informs frameworks. They can choose to reschedule task in other healthy agents.
Framework: Mesos delegates framework to implement its own failover mechanism.






Apache Mesos is not just a platform for efficient resource sharing in a cluster. Mesos is a solution for current situations in distributed systems:
Multiple applications that belong to hybrid computation frameworks or a microservice architecture can share the same cluster resources.
An ever-changing technology will have a way to fit with mesos thanks to its extensible API.
Multiple datacenter hardware (on premise, cloud) is not a problem due to its infrastructure agnostic design.



We are a team that has as main objective the dissemination and training of the technological stack that sustains Datio. We also ensure the quality and coherence of our technical documentation.

More Posts