本文分享自微信公众号 - dotNET跨平台（opendotnet）。
如有侵权，请联系 support@oschina.cn 删除。
本文参与“OSC源创计划”，欢迎正在阅读的你也加入，一起分享。

Apache Mesos (4)-Mesos 的资源调度、分配及隔离机制

OSC 请你来轰趴啦！1028 苏州源创会，一起寻宝 AI 时代

出处： https://andyyoung01.github.io/ 或 http://andyyoung01.16mb.com/

前面的文章，我们对 Mesos 架构已经有了基本的认识，并且也安装配置了一个有 3 个主节点的高可用集群。虽然我们目前还没有实际在集群上部署 Frameworks 来运行具体的任务，但后面的文章我们会详细探索 Marathon、Chronos 和 Aurora 等框架。现在我们先更深入得了解一下 Mesos 的资源调度、分配及隔离机制。

Mesos 的资源调度机制

Mesos 实现了一个两层的调度系统：Mesos slave 将它的可用资源汇报给 master，然后 master 通过可插拔的分配模块（Allocation module）向 Framework 的 schedule 发出资源邀约，这个 schedule 可以接受整个、部分或者拒绝这个资源邀约。如下图所示：

1. Mesos slave 向 master 通告它的资源：8CPUs，16GB 内存，64GB 磁盘可用空间。星号（*）表示这些资源属于默认角色（default role）。我们后面会介绍到角色。
2. Mesos 的分配模块决定 master 应该向 Framework A 的 schedule 发出资源邀约。
3. Framework A 的 schedule 接受了资源邀约的一半的资源，剩下 4CPUs，8GB 内存及 32GB 的可用磁盘空间给其它程序。
4. Mesos 的分配模块决定 master 应该将所有剩下的未分配的资源通告给 Framework B 的 schedule。

上面的过程一直重复不断地进行，因为每过一段时间，slaves 会因为任务运行完毕而空出可用的资源来。

因为大部分的程序都需要一定的 CPUs，内存，磁盘及网络，Mesos 预定义了一些资源。

• 默认资源 - 默认情况下，来自于 Mesos slave 的资源通告包括：
  • cups
  • mem
  • disk
  • ports

在 mesos-slave 启动时，系统的空余资源决定了上面这些值。

关于资源通告的以及 Mesos 系统架构的其它信息，可以参考官方文档 http://mesos.apache.org/documentation/latest/architecture/ 。

当 slave 持续不断地将自己的可用资源通告给 master 时，Mesos 的另一个部分，可插拔分配模块，负责决定哪个 Framework 应该得到给定的资源邀约。

Mesos 的资源分配机制

Mesos master 的分配模块（Allocation module）决定了将资源分配给某个 Framework。此模块可插拔的特性，允许系统工程师根据自己组织的需要，来实现他们自己的分配策略和算法。内置的分配模块使用的是 Dominant Resource Fairnes（DRF）算法，它可以满足大部分 Mesos 用户的需要。

关于默认的分配模块和算法的更多信息，请参考 http://mesos.apache.org/documentation/latest/allocation-module/ 。

Mesos 默认提供了几种方法来对资源分配进行调整，而不需要替换或者重写整个分配模块。这些方法包括 角色 ， 权重 和 资源保留 。

角色

在 Mesos 集群中，角色允许您将 Frameworks 和资源分成任意的组。

要使用角色，需要在启动 master 时加入–roles 的配置选项。例如，下面的配置允许 Frameworks 以三个在数据中心里常见的角色注册：

--roles="dev,stage,prod"

之后，当 Framework 要在 master 上注册时，Framework 可以指定为这些角色中的任意一种。这就允许许多团队或许多环境来共享一个大的 Mesos 集群，而不用创建多个小的集群。您也可以使用角色来确保特定类型的工作只运行在特定的一组机器上，例如负载均衡器或反向代理只运行在专用的边缘节点上。

权重

集群也可以给每个角色配置权重，来使不同的角色对于资源的分配有不同的优先权。当 Mesos 决定将资源先分配给哪个 Framework 时，它先将资源分配给最低于该权重的合理份额的 Framework。例如，参考如下配置：

--weights="dev=10,stage=20,prod=30"

具有 prod 角色的 Frameworks 将得到比具有 dev 角色的 Frameworks 的三倍的资源。

资源保留

虽然权重可以确保某个角色得到比其它角色更多的资源，但 Mesos 也提供资源保留的方法。资源保留确保某个角色总是可以得到 slave 的一定的资源，但这样做可能会导致整个集群使用率的降低。

假如有一台机器有 16CPUs，32GB 内存，128GB 磁盘，你想要确保这台机器有一半的资源总是可以提供给以 prod 角色注册的 Frameworks，可以这样在这台 Mesos slave 机器上配置：

--resources="cpus(prod):8; mem(prod):16384; disk(prod):65536"

手动配置 Mesos slave 的资源和属性

前面几节我们知道资源可以分配给 slave 上的一定的角色，也可以配置资源保留。但如果您想创建定制的资源或想重写 slave 上资源的默认的值呢？和 Mesos 中的其它东西一样，资源通告的内容也是可配置的。当你想引入一些新的资源，或者想将 CPUs 和内存等资源硬编码到资源通告中，这个特性是非常有用的。

手动配置 Mesos slave 的资源

Mesos 提供了三种数据类型来指定资源： 标量，范围，和集合 。

• 标量：cpus 资源给定值 8；mem 资源给定值 16384。
• 范围：端口资源给定从 10000 到 20000 的值。
• 集合：磁盘资源给定值 ssd1，ssd2，和 ssd3。

可以使用与资源保留类似的方式指定 slave 资源：

--resources="cpu(*):4; mem(*):8192; disk(*):32768; ports(*):[40000-50000];cpu(prod):8; mem(prod):16384; disk(prod):65536"

手动配置 Mesos slave 的属性

属性可以是任意的键 / 值对，用来给 master 或 Frameworks 提供关于机器的一些数据。可以这样设置：

--attributes="datacenter:pdx1; rack:1-1; os:rhel7; pythons:python2,python3"

关于怎样配置 Mesos slave 资源和属性的更多信息，请参考 http://mesos.apache.org/documentation/latest/attributes-resources 。

Mesos 的资源隔离机制

容器是榨取您的基础设施效率的一个非常理想的方法。容器比起虚拟机来更加轻量级，并且允许您在一个与其它负载隔离的环境中运行程序和代码。Mesos 中的一个基础理念是使用容器隔离进程是使用计算资源最有效率的方式。

Mesos 默认支持 Linux cgroups 和 Docker，两种当下最流行的容器技术。通过在容器中运行 executors 和任务，Mesos slave 允许多个 Framework 的 executor 一起运行，而不影响其它负载。这有点像在每个物理机上同时运行多个虚拟机，容器不用启动整个操作系统而比虚拟机轻量化许多。

Mesos 的一个基础的组件称为 containerizer。可以使用 –containerizers 配置选项在 Mesos slave 上配置，目前包括两个 containerizer：mesos 和 docker。mesos 配置使用 cgroups 隔离和监视负载；docker 配置调用 Docker 容器运行时，允许您在 Mesos 集群上启动已经编译好的镜像。

除了 containerizer，Mesos 也提供其它方法隔离资源。包括 posix/cpu 和 posix/mem（默认），cgroups/cpu 和 cgroups/mem。怎样使用对资源的隔离和监视呢？是通过在 slave 上，给–isolation 配置选项提供参数列表，来实现的。

本篇文章，我们更深入得了解了 Mesos 的资源调度、分配及隔离机制，下面我们使用 Marathon 来实际运行一个任务。

 

http://www.tuicool.com/articles/JzYvi23

Apache Mesos 0.22.1 发布，集群管理器

Apache Mesos 0.22.1 发布，此版本更新内容：

Bug 修复

• [MESOS-1795] - Assertion failure in state abstraction crashes JVM

• [MESOS-2161] - AbstractState JNI check fails for Marathon framework

• [MESOS-2583] - Tasks getting stuck in staging

• [MESOS-2592] - The sandbox directory is not chown''ed if the fetcher doesn''t run

• [MESOS-2601] - Tasks are not removed after recovery from slave and mesos containerizer

• [MESOS-2643] - Python scheduler driver disables implicit acknowledgments by default.

• [MESOS-2668] - Slave fails to recover when there are still processes left in its cgroup

改进

• [MESOS-2461] - Slave should provide details on processes running in its cgroups

更多内容请看发行说明，此版本现已提供下载：mesos-0.22.1.tar.gz。

Apache Mesos是一个集群管理器，提供了有效的、跨分布式应用或框架的资源隔离和共享，可以运行Hadoop、MPI、Hypertable、Spark。

特性：

• Fault-tolerant replicated master using ZooKeeper

• Scalability to 10,000s of nodes

• Isolation between tasks with Linux Containers

• Multi-resource scheduling (memory and CPU aware)

• Java, Python and C++ APIs for developing new parallel applications

• Web UI for viewing cluster state

Apache Mesos 0.23.0 发布，集群管理器

Apache Mesos 0.23.0 发布，此版本主要改进：

• [MESOS-692] - Reservations are not reported in master''s state.json

• [MESOS-994] - Add an Option<string> os::getenv() to stout

• [MESOS-1733] - Change the stout path utility to declare a single, variadic ''join'' function instead of several separate declarations of various discrete arities

• [MESOS-1991] - Remove dynamic allocation from Option

• [MESOS-2023] - mesos-execute should allow setting environment variables

• [MESOS-2050] - Revise Authenticator interface

• [MESOS-2057] - Concurrency control for fetcher cache

• [MESOS-2069] - Basic fetcher cache functionality

• [MESOS-2070] - Implement simple slave recovery behavior for fetcher cache

• [MESOS-2072] - Fetcher cache eviction

• [MESOS-2074] - Fetcher cache test fixture

• [MESOS-2103] - Expose number of processes and threads in a container

• [MESOS-2104] - Correct naming of cgroup memory statistics

• [MESOS-2110] - Configurable Ping Timeouts

• [MESOS-2111] - Add build instructions for OSX in getting started

• [MESOS-2136] - Expose per-cgroup memory pressure

• [MESOS-2155] - Make docker containerizer killing orphan containers optional

• [MESOS-2277] - Document undocumented HTTP endpoints

• [MESOS-2323] - write flags to log at startup

• [MESOS-2332] - Report per-container metrics for network bandwidth throttling

• [MESOS-2333] - Securing Sandboxes via Filebrowser Access Control

• [MESOS-2340] - Add ability to decode JSON serialized MasterInfo from ZK

• [MESOS-2374] - Support relative host paths for container volumes

• [MESOS-2392] - Rate limit slaves removals during master recovery.

• [MESOS-2400] - Improve NsTest.ROOT_setns

• [MESOS-2438] - Improve support for streaming HTTP Responses in libprocess.

• [MESOS-2454] - Add support for /proc/self/mountinfo on Linux

• [MESOS-2461] - Slave should provide details on processes running in its cgroups

• [MESOS-2462] - Add option for Subprocess to set a death signal for the forked child

• [MESOS-2485] - Add ability to distinguish slave removals metrics by reason.

• [MESOS-2507] - Performance issue in the master when a large number of slaves are registering.

• [MESOS-2519] - Log IP addresses from HTTP requests

• [MESOS-2527] - Add default bind to socket

• [MESOS-2528] - Symlink the namespace handle with ContainerID for the port mapping isolator.

• [MESOS-2547] - Cleanup stale bind mounts for port mapping isolator during slave recovery.

• [MESOS-2549] - Remove non-variadic strings::format

• [MESOS-2550] - Mesos doesn''t compile with clang 3.6

• [MESOS-2565] - Clean up style and comments in modules.

• [MESOS-2571] - Expose Memory Pressure in MemIsolator

• [MESOS-2573] - Use Memory Test Helper to improve some test code.

• [MESOS-2595] - Create docker executor

• [MESOS-2604] - Upgrade minimum required compilers for MESOS

• [MESOS-2608] - test-framework should support principal only credential

• [MESOS-2609] - Move StatusUpdateStream implementation to a compilation unit

• [MESOS-2620] - Implement a mechanism which allows access control of endpoints

• [MESOS-2624] - "configure" should fail when "patch" is not available.

• [MESOS-2653] - Slave should act on correction events from QoS controller

• [MESOS-2654] - Update FrameworkInfo to opt in to revocable resources

• [MESOS-2666] - use standard compiler detection macros

• [MESOS-2680] - Update modules doc with hook usage example

• [MESOS-2693] - Printing a resource should show information about reservation, disk etc

• [MESOS-2709] - Design Master discovery functionality for HTTP-only clients

• [MESOS-2716] - Add non-const reference version of Option<T>::get.

• [MESOS-2729] - Update DRF sorter to update total resources

• [MESOS-2743] - Include ExecutorInfos for custom executors in master/state.json

• [MESOS-2745] - Add ''Path'' to stout''s user guide

• [MESOS-2752] - Add HTB queueing discipline wrapper class

• [MESOS-2784] - Added constexpr to C++11 whitelist.

• [MESOS-2793] - Add support for container rootfs to Mesos isolators

• [MESOS-2801] - Remove dynamic allocation from Future<T>

• [MESOS-2804] - Log framework capabilities in the master.

• [MESOS-2805] - Make synchronized as primary form of synchronization.

• [MESOS-2836] - Report per-container metrics for network bandwidth throttling to the slave

• [MESOS-2837] - Decode network statistics from mesos-network-helper

• [MESOS-2870] - Add validation capability to stout Flags

• [MESOS-2888] - Add SSL socket tests

• [MESOS-2928] - Update stout #include headers

• [MESOS-2940] - Reconciliation is expensive for large numbers of tasks.

• [MESOS-2955] - Introduce acceptOffers scheduler driver API for performing operations on Offers

• [MESOS-2957] - Add version to MasterInfo

• [MESOS-2958] - Update Call protobuf to move top level FrameworkInfo inside Subscribe

• [MESOS-2966] - socket::peer() and socket::address() might fail with SSL enabled

详细更新内容请看发行说明，现已提供下载：http://archive.apache.org/dist/mesos/0.23.0/mesos-0.23.0.tar.gz。

Apache Mesos 是一个集群管理器，提供了有效的、跨分布式应用或框架的资源隔离和共享，可以运行Hadoop、MPI、Hypertable、Spark。

特性：

• Fault-tolerant replicated master using ZooKeeper

• Scalability to 10,000s of nodes

• Isolation between tasks with Linux Containers

• Multi-resource scheduling (memory and CPU aware)

• Java, Python and C++ APIs for developing new parallel applications

• Web UI for viewing cluster state

Apache Mesos 0.24.0 发布，集群管理器

Apache Mesos 0.24.0 发布，此版本更新内容如下：

Bug 修复

• [MESOS-2166] -  PerfEventIsolatorTest.ROOT_CGROUPS_Sample requires ''perf'' to be installed

• [MESOS-2337] - __init__.py not getting installed in $PREFIX/lib/pythonX.Y/site-packages/mesos

• [MESOS-2480] - Protobuf jar is required for unbundled protobuf regardless of --disable-java flag.

• [MESOS-2493] - google glog link is incorrect

• [MESOS-2497] - Create synchronous validations for Calls

• [MESOS-2552] - C++ Scheduler library should send HTTP Calls to master

• [MESOS-2559] - Do not use RunTaskMessage.framework_id.

• [MESOS-2660] - ROOT_CGROUPS_Listen and ROOT_IncreaseRSS tests are flaky

• [MESOS-2862] - mesos-fetcher won''t fetch uris which begin with a " "

• [MESOS-2868] - --attributes flag in slave cannot take a value with '':''

• [MESOS-2882] - Duplicate name-value env-vars in ''-e'' option of docker run

• [MESOS-2900] - Display capabilities in state.json

• [MESOS-2989] - Changing to "framework" from "framwork"

• [MESOS-3001] - Create a "demo" HTTP API client

• [MESOS-3002] - Rename Option<T>::get(const T& _t) to getOrElse() broke network isolator

• [MESOS-3027] - Compiler warning in stout subcommand tests

• [MESOS-3053] - Failing DockerContainerizerTest.ROOT_DOCKER_Launch_Executor_Bridged

• [MESOS-3058] - Cgroup tests relies on cgroups::get() returning in a specific order

• [MESOS-3079] - `sudo make distcheck` fails on Ubuntu 14.04 (and possibly other OSes too)

• [MESOS-3121] - Always disable SSLV2

• [MESOS-3124] - Updating persistent volumes after slave restart is problematic.

• [MESOS-3138] - PersistentVolumeTest.SlaveRecovery test fails on OSX

• [MESOS-3141] - Compiler warning when mocking function type has an enum return type.

• [MESOS-3143] - Disable endpoints rule fails to recognize HTTP path delegates

• [MESOS-3148] - Resolve issue with hanging tests with Zookeeper

• [MESOS-3168] - MesosZooKeeperTest fixture can have side effects across tests

• [MESOS-3170] - 0.23 Build fails when compiling against -lsasl2 which has been statically linked

• [MESOS-3175] - subprocess_tests.cpp:598 delete used but allocated with new[]

• [MESOS-3178] - Perform a self bind mount of rootfs itself in fs::chroot::enter.

• [MESOS-3192] - ContainerInfo::Image::AppC::id should be optional

• [MESOS-3195] - Fix master metrics for scheduler calls

• [MESOS-3197] - MemIsolatorTest/{0,1}.MemUsage fails on OS X

• [MESOS-3201] - Libev handle_async can deadlock with run_in_event_loop

• [MESOS-3203] - MasterAuthorizationTest.DuplicateRegistration test is flaky

• [MESOS-3204] - PortMappingIsolatorProcess shell script can silently fail

• [MESOS-3207] - C++ style guide is not rendered correctly (code section syntax disregarded)

• [MESOS-3209] - parameterize allocator benchmark by framework count

• [MESOS-3234] - enable automake maintainer mode

• [MESOS-3237] - HTTP requests with nested path are not properly handled by libprocess

• [MESOS-3238] - Master endpoint help message is incorrect

• [MESOS-3260] - SchedulerTest.* are broken on OSX and CentOS

• [MESOS-3262] - HTTPTest.NestedGet is flaky

• [MESOS-3263] - SchedulerTask.KillTest fails for JSON Requests

• [MESOS-3267] - JSON serialization/deserialization of bytes is incorrect

• [MESOS-3274] - Build error with port mapping isolator

• [MESOS-3275] - ContentType/HttpApiTest.UpdatePidToHttpSchedulerWithoutForce is flaky

• [MESOS-3284] - JSON representation of Protobuf should use base64 encoding for ''bytes'' fields.

• [MESOS-3287] - downloadWithHadoop tries to access Error() for a valid Try<bool>

• [MESOS-3290] - Master should drop HTTP calls when it''s recovering

• [MESOS-3294] - Failing ROOT_ tests on CentOS 7.1 - UserCgroupIsolatorTest

• [MESOS-3296] - Failing ROOT_ tests on CentOS 7.1 - LinuxFilesystemIsolatorTest

• [MESOS-3297] - Failing ROOT_ tests on CentOS 7.1 - MesosContainerizerLaunchTest

• [MESOS-3311] - SlaveTest.HTTPSchedulerSlaveRestart

• [MESOS-3321] - Spurious fetcher message about extracting an archive

文档

• [MESOS-1838] - Add documentation for Authentication

• [MESOS-2555] - Document issue with slave recovery when using systemd.

• [MESOS-3087] - Typos in oversubscription doc

• [MESOS-3167] - Design doc for versioning the HTTP API

• [MESOS-3278] -  Add the revocable metrics information in monitoring doc

• [MESOS-3281] - Create a user doc for Scheduler HTTP API

• [MESOS-3286] - Revocable metrics information are missed for slave node

Epic

• [MESOS-336] - Mesos slave should cache executors

• [MESOS-2288] - Scheduler HTTP API

改进

• [MESOS-809] - External control of the ip that Mesos components publish to zookeeper

• [MESOS-2350] - Add support for MesosContainerizerLaunch to chroot to a specified path

• [MESOS-2736] - Upgrade the design of MasterInfo

• [MESOS-2794] - Implement filesystem isolators

• [MESOS-2795] - Introduce filesystem provisioner abstraction

• [MESOS-2798] - Export statistics on "unevictable" memory

• [MESOS-2800] - Rename Option<T>::get(const T& _t) to getOrElse() and refactor the original function

• [MESOS-2841] - FrameworkInfo should include a Labels field to support arbitrary, lightweight metadata

• [MESOS-2880] - Add Frameworkinfo.capabilities on framework re-registration

• [MESOS-2902] - Enable Mesos to use arbitrary script / module to figure out IP, HOSTNAME

• [MESOS-2946] - Authorizer Module: Interface design

• [MESOS-2947] - Authorizer Module: Implementation, Integration & Tests

• [MESOS-2951] - Inefficient container usage collection

• [MESOS-2965] - Add implicit cast to string operator to Path.

• [MESOS-2967] - Missing doxygen documentation for libprocess socket interface

• [MESOS-3020] - Expose major, minor and patch components from stout Version

• [MESOS-3054] - update gitignore

• [MESOS-3093] - Support HTTPS requests in libprocess

• [MESOS-3112] - Fetcher should perform cache eviction based on cache file usage patterns.

• [MESOS-3118] - Remove pthread specific code from Stout

• [MESOS-3119] - Remove pthread specific code from Libprocess

• [MESOS-3120] - Remove pthread specific code from Mesos

• [MESOS-3127] - Improve task reconciliation documentation.

• [MESOS-3173] - Mark Path::basename, Path::dirname as const functions.

• [MESOS-3182] - Make Master::registerFramework() and Master::reregisterFramework() call into Master::subscribe()

Story

• [MESOS-2293] - Implement the scheduler endpoint on master

• [MESOS-2860] - Create the basic infrastructure to handle /scheduler endpoint

• [MESOS-3135] - Publish MasterInfo to ZK using JSON

• [MESOS-3142] - As a Developer I want a better way to run shell commands

• [MESOS-3154] - Enable Mesos Agent Node to use arbitrary script / module to figure out IP, HOSTNAME

• [MESOS-3211] - As a Python developer I want a simple way to obtain information about Master from ZooKeeper

• [MESOS-3212] - As a Java developer I want a simple way to obtain information about Master from ZooKeeper

任务

• [MESOS-2294] - Implement the Events stream on master for Call endpoint

• [MESOS-2640] - Remove old frameworks and ec2 scripts from core Mesos repository

• [MESOS-2910] - Add an Event message handler to scheduler driver

• [MESOS-2913] - Scheduler driver should send Call messages to the master

• [MESOS-2933] - Pass slave''s total resources to the ResourceEstimator and QoSController via Slave::usage().

• [MESOS-2961] - Add cpuacct subsystem utils to cgroups

• [MESOS-3012] - Support existing message passing optimization with Event/Call.

• [MESOS-3067] - Implement a streaming response decoder for events stream

• [MESOS-3088] - Update scheduler driver to send SUBSCRIBE call

• [MESOS-3089] - Update scheduler library to send REQUEST call

• [MESOS-3101] - Standardize separation of Windows/Linux-specific OS code

• [MESOS-3102] - Separate OS-specific code in the stout library

• [MESOS-3130] - Custom isolators should implement Isolator instead of IsolatorProcess.

• [MESOS-3131] - Master should send heartbeats on the subscription connection

• [MESOS-3132] - Allow slave to forward messages through the master for HTTP schedulers.

• [MESOS-3145] - Using a unresolvable hostname crashes the framework on registration

• [MESOS-3149] - Use setuptools to install python cli package

• [MESOS-3162] - Provide a means to check http connection equality for streaming connections.

• [MESOS-3179] - Create a test abstraction for preparing test rootfs.

• [MESOS-3194] - Implement a ''read-only'' AppC Image Store

下载：http://www.apache.org/dyn/mirrors/mirrors.cgi/mesos/0.24.0/。

Apache Mesos 是一个集群管理器，提供了有效的、跨分布式应用或框架的资源隔离和共享，可以运行 Hadoop、MPI、Hypertable、Spark。

关于集群管理器 Mesos 能在 Windows Server上运行了和集群管理器如何配置的介绍已经告一段落，感谢您的耐心阅读，如果想了解更多关于Apache Mesos (4)-Mesos 的资源调度、分配及隔离机制、Apache Mesos 0.22.1 发布，集群管理器、Apache Mesos 0.23.0 发布，集群管理器、Apache Mesos 0.24.0 发布，集群管理器的相关信息，请在本站寻找。