Set up a Mesos Cluster with Kafka Framework / 02 Dec 2015 / Author: Haim Ari

    Estimated read time: 4 minutes

    Apache Mesos is an open-source cluster manager that was developed at the University of California, Berkeley. It “provides efficient resource isolation and sharing across distributed applications, or frameworks”. The software enables resource sharing in a fine-grained manner, improving cluster utilization.

    Set up Mesos Masters Cluster (3 nodes with ZooKeeper)

    Steps:

    • Install Ubuntu 14.04 (currently the easiest to install Mesos)
    • Install Zookeeper
    • Install Mesos masters cluster
    • Install Mesos Slaves
    • Install Zookeeper on slaves (or on other nodes)
    • Install Kafka Framework

    There is also an option to install Mesosphare instead but i found it less stable for production.

    Mesosphare Project

    Install zookeeper:

    ZooKeeper

    Configure /your-path/zookeeper/conf/zoo.cfg with your zookeeper cluster parameters:

    tickTime=2000
    initLimit=10
    syncLimit=5
    dataDir=/data/zookeeper
    clientPort=12182
    
    maxClientCnxns=500
    
    server.1=master-1:12888:13888
    server.2=master-1:12888:13888
    server.3=master-1:12888:13888
    
    
    autopurge.snapRetainCount=3
    autopurge.purgeInterval=6
    

    Set “myid” file with the id number for each node of the zookeeper cluster

    echo "1" >/data/zookeeper/myid
    

    Start zookeeper, make sure it is running and that you have 1 leader and 2 followers. You can check it using netcat:

    echo "stat" | nc localhost 2181
    
    Clients: 
    /127.0.0.1:38470[0](queued=0,recved=1,sent=0) 
    
    Latency min/avg/max: 0/0/432 
    Received: 28009125 
    Sent: 28022128 
    Connections: 7 
    Outstanding: 0 
    Zxid: 0x200996dc2 
    Mode: leader
    Node count: 3
    

    Install Mesos

    Download and install Mesos (Current stable version is: 0.25.0)

    apt-get update 
    apt-get install -y openjdk-7-jdk 
    apt-get install -y autoconf libtool 
    apt-get -y install build-essential python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev  
    
    cd ~ 
    wget http://www.apache.org/dist/mesos/0.25.0/mesos-0.25.0.tar.gz 
    tar -zxf mesos-0.25.0.tar.gz 
    cd mesos-0.25.0 
    mkdir build 
    cd build 
    ../configure 
    make -j [ core num ] V=0 
    make install
    

    Start Mesos cluster:

    Run this command on all 3 masters

    nohup /usr/local/sbin/mesos-master --zk=zk://master-1:2181,master-2:2181,master-3:2181/mesos \
    --work_dir=/var/lib/mesos --quorum=2 &
    

    Install Mesos Slaves(on your slave hosts)

    apt-get update 
    apt-get install -y openjdk-7-jdk 
    apt-get install -y autoconf libtool 
    apt-get -y install build-essential python-dev python-boto libcurl4-nss-dev libsasl2-dev maven libapr1-dev libsvn-dev  
    
    cd ~ 
    wget http://www.apache.org/dist/mesos/0.25.0/mesos-0.25.0.tar.gz 
    tar -zxf mesos-0.25.0.tar.gz 
    cd mesos-0.25.0 
    mkdir build 
    cd build 
    ../configure 
    make -j [ core num ] V=0 
    make install
    

    Install Kafka Framework on Masters:

    Kafka Framework

    Install OpenJDK 7 (or higher) http://openjdk.java.net/install/ Install gradle http://gradle.org/installation

    Clone and build the project

    git clone https://github.com/mesos/kafka
    cd kafka
    ./gradlew jar
    wget https://archive.apache.org/dist/kafka/0.8.2.2/kafka_2.10-0.8.2.2.tgz
     
    # export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
    
    Update kafka-mesos.properties.

    This is an example. update according to your environment

    # Scheduler options defaults. See `./kafka-mesos.sh help scheduler` for more details
    
    debug=true
    storage=file:kafka-mesos.json
    master=zk://master-1:2181,master-2:2181,master-3:2181/mesos
    zk=zookeeper-1:12181,zookeeper-2:12181,zookeeper-3:12181
    api=http://master-1:7000
    clusterStorage=zk:/kafka-mesos
    

    Start the Scheduler:

    nohup ./kafka-mesos.sh scheduler &
    

    Add 3 brokers (you can add just 1 or any other number of brokers)

    ./kafka-mesos.sh broker add 0..2
    

    Start the brokers

    ./kafka-mesos.sh broker start 0..2
    

    Now check your Kafka zookeeper ensemble (not the masters zookeeper) you should see your kafka clients listed:

    echo "stat" | nc zookeeper-1 12181
    

    You can adjust the options for kafka as you like. run help to see the options:

    ./kafka-mesos.sh help scheduler
    
    Start scheduler 
    Usage: scheduler [options] [config.properties]
    
    Option               Description
    ------               -----------
    --api                Api url. Example: http://master:7000
    --bind-address       Scheduler bind address (master, 0.0.0.0, 192.168.50.*, if:eth1). Default - all
    --debug     Debug mode. Default - false
    --framework-name     Framework name. Default - kafka
    --framework-role     Framework role. Default - *
    --framework-timeout  Framework timeout (30s, 1m, 1h). Default - 30d
    --jre                JRE zip-file (jre-7-openjdk.zip). Default - none.
    --log                Log file to use. Default - stdout.
    --master             Master connection settings. Examples:
                          - master:5050
                          - master:5050,master2:5050
                          - zk://master:2181/mesos
                          - zk://username:[email protected]:2181
                          - zk://master:2181,master2:2181/mesos
    --principal          Principal (username) used to register framework. Default - none
    --secret             Secret (password) used to register framework. Default - none
    --storage            Storage for cluster state. Examples:
                          - file:kafka-mesos.json
                          - zk:/kafka-mesos
                         Default - file:kafka-mesos.json
    --user               Mesos user to run tasks. Default - none
    --zk                 Kafka zookeeper.connect. Examples:
                          - master:2181
                          - master:2181,master2:2181
    

    Now to add more brokers just use “broker add” and “broker start” as above.

    Updating broker configurations

    ./kafka-mesos.sh help broker update
    
    Update broker
    Usage: broker update  [options]
    
    Option                Description
    ------                -----------
    --bind-address        broker bind address (broker0, 192.168.50.*, if:eth1). Default - auto
    --constraints         constraints (hostname=like:master,rack=like:1.*). See below.
    --cpus        cpu amount (0.5, 1, 2)
    --failover-delay      failover delay (10s, 5m, 3h)
    --failover-max-delay  max failover delay. See failoverDelay.
    --failover-max-tries  max failover tries. Default - none
    --heap          heap amount in Mb
    --jvm-options         jvm options string (-Xms128m -XX:PermSize=48m)
    --log4j-options       log4j options or file. Examples:
                           log4j.logger.kafka=DEBUG\, kafkaAppender
                           file:log4j.properties
    --mem           mem amount in Mb
    --options             options or file. Examples:
                           log.dirs=/tmp/kafka/$id,num.io.threads=16
                           file:server.properties
    --port                port or range (31092, 31090..31100). Default - auto
    --stickiness-period   stickiness period to preserve same node for broker (5m, 10m, 1h)
    --volume              pre-reserved persistent volume id
    
    Generic Options
    Option  Description
    ------  -----------
    --api   Api url. Example: http://master:7000
    
    broker-expr examples:
      0      - broker 0
      0,1    - brokers 0,1
      0..2   - brokers 0,1,2
      0,1..2 - brokers 0,1,2
      *      - any broker
    attribute filtering:
      *[rack=r1]           - any broker having rack=r1
      *[hostname=slave*]   - any broker on host with name starting with 'slave'
      0..4[rack=r1,dc=dc1] - any broker having rack=r1 and dc=dc1
    
    constraint examples:
      like:master     - value equals 'master'
      unlike:master   - value not equals 'master'
      like:slave.*    - value starts with 'slave'
      unique          - all values are unique
      cluster         - all values are the same
      cluster:master  - value equals 'master'
      groupBy         - all values are the same
      groupBy:3       - all values are within 3 different groups
    
    Note: use "" arg to unset an option