Starting point for getting started
This commit is contained in:
Renan DelValle
parent
77ae0bc310
commit
8f85c75b94
1 changed files with 240 additions and 0 deletions
240
docs/Getting_Started.md
Normal file
240
docs/Getting_Started.md
Normal file
|
|
@ -0,0 +1,240 @@
|
||||||
|
# Running custom executors on Aurora
|
||||||
|
|
||||||
|
In this instance we will be using the docker-compose executor to demonstrate
|
||||||
|
how Aurora can use multiple executors on a single Scheduler.
|
||||||
|
|
||||||
|
## Configuring Scheduler to use Docker-Compose executor
|
||||||
|
In order use the docker compose executor with Aurora, we must first give the scheduler
|
||||||
|
a configuration file that contains information on how to run the executor.
|
||||||
|
|
||||||
|
### Configuration file
|
||||||
|
The configuration is a JSON file that contains where to find the executor and how to run it.
|
||||||
|
|
||||||
|
A sample config file for the docker-compose executor looks like this:
|
||||||
|
```
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"executor":{
|
||||||
|
"command":{
|
||||||
|
"value":"java -jar docker-compose-executor_0.0.1.jar",
|
||||||
|
"shell":"true",
|
||||||
|
"uris":[
|
||||||
|
{
|
||||||
|
"cache":false,
|
||||||
|
"executable":true,
|
||||||
|
"extract":false,
|
||||||
|
"value":"https://dl.bintray.com/rdelvalle/mesos-compose-executor/docker-compose-executor_0.0.1.jar"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"name":"docker-compose-executor",
|
||||||
|
"resources":[]
|
||||||
|
},
|
||||||
|
"task_prefix":"compose-"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
### Configuring Scheduler to run custom executor
|
||||||
|
#### Setting the proper flags `-custom_executor_config` flag and enabling mesos fetcher on jobs
|
||||||
|
|
||||||
|
The `--custom_executor_config` flag must point to the location of the JSON blob.
|
||||||
|
|
||||||
|
The `--enable_mesos_fetcher` flag must be set to true in order to allow jobs to fetch resources.
|
||||||
|
|
||||||
|
#### On vagrant
|
||||||
|
Modify the file `/etc/init/aurora-scheduler.conf` to include:
|
||||||
|
```
|
||||||
|
--custom_executor_config=<Location of JSON blob> \
|
||||||
|
--enable_mesos_fetcher=true
|
||||||
|
```
|
||||||
|
|
||||||
|
#### On a scheduler installed via package manager
|
||||||
|
Modify the file `/etc/default/aurora-scheduler.conf` to include:
|
||||||
|
```
|
||||||
|
AURORA_EXTRA_ARGS="--custom_executor_config=<Location of JSON blob> \
|
||||||
|
--enable_mesos_fetcher=true"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Using a custom client
|
||||||
|
Pystachio does yet support launching tasks using a custom executors. Therefore, a custom
|
||||||
|
client must be used in order to launch tasks using a custom executor. In this case,
|
||||||
|
we will be using [gorealis](https://github.com/rdelval/gorealis) to launch a task with
|
||||||
|
the compose executor on Aurora.
|
||||||
|
|
||||||
|
# Configuring the system to run the custom client and docker-compose executor
|
||||||
|
|
||||||
|
## Installing Go
|
||||||
|
|
||||||
|
### Linux
|
||||||
|
|
||||||
|
#### Ubuntu
|
||||||
|
|
||||||
|
##### Add PPA and install via apt-get
|
||||||
|
```
|
||||||
|
$ sudo add-apt-repository ppa:ubuntu-lxc/lxd-stable
|
||||||
|
$ sudo apt-get update
|
||||||
|
$ sudo apt-get install golang
|
||||||
|
```
|
||||||
|
|
||||||
|
##### Set GOPATH
|
||||||
|
|
||||||
|
Configure the environment to be able to compile and run Go code.
|
||||||
|
|
||||||
|
```
|
||||||
|
$ mkdir $HOME/go
|
||||||
|
$ echo export GOPATH=$HOME/go >> $HOME/.profile
|
||||||
|
$ echo export PATH=$PATH:$GOPATH/bin >> $HOME/.profile
|
||||||
|
$ echo export PATH=$PATH:$GOROOT/bin >> $HOME/.profile
|
||||||
|
```
|
||||||
|
|
||||||
|
### OS X
|
||||||
|
|
||||||
|
One way to install go on OS X is by using [Homebrew](http://brew.sh/)
|
||||||
|
|
||||||
|
#### Install Homebrew
|
||||||
|
Run the following command from the terminal to install Hombrew:
|
||||||
|
```
|
||||||
|
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Install Go using Hombrew
|
||||||
|
|
||||||
|
Run the following command from the terminal to install Go:
|
||||||
|
```
|
||||||
|
$ brew install go
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Set the GOPATH
|
||||||
|
|
||||||
|
Configure the environment to be able to compile and run Go code.
|
||||||
|
|
||||||
|
```
|
||||||
|
$ mkdir $HOME/go
|
||||||
|
$ echo export GOPATH=$HOME/go >> $HOME/.profile
|
||||||
|
$ echo export GOROOT=/usr/local/opt/go/libexec >> $HOME/.profile
|
||||||
|
$ echo export PATH=$PATH:$GOPATH/bin >> $HOME/.profile
|
||||||
|
$ echo export PATH=$PATH:$GOROOT/bin >> $HOME/.profile
|
||||||
|
```
|
||||||
|
|
||||||
|
### Windows
|
||||||
|
|
||||||
|
Download and run the msi installer from https://golang.org/dl/
|
||||||
|
|
||||||
|
## Installing Docker Compose
|
||||||
|
|
||||||
|
In order to run the docker-compose executor, each agent must have docker-compose
|
||||||
|
installed on it.
|
||||||
|
|
||||||
|
This can be done using pip:
|
||||||
|
```
|
||||||
|
$ sudo pip install docker-compose
|
||||||
|
```
|
||||||
|
|
||||||
|
## Downloading gorealis
|
||||||
|
Finally, we must get `gorealis` using the `go get` command:
|
||||||
|
|
||||||
|
```
|
||||||
|
go get github.com/rdelval/gorealis
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
# Creating Aurora Jobs
|
||||||
|
|
||||||
|
### Creating a thermos job
|
||||||
|
To demonstrate that we are able to run jobs using different executors on the
|
||||||
|
same scheduler, we'll first launch a thermos job using the default Aurora Client.
|
||||||
|
|
||||||
|
We can use a sample job for this:
|
||||||
|
|
||||||
|
hello_world.aurora
|
||||||
|
```
|
||||||
|
hello = Process(
|
||||||
|
name = 'hello',
|
||||||
|
cmdline = """
|
||||||
|
while true; do
|
||||||
|
echo hello world
|
||||||
|
sleep 10
|
||||||
|
done
|
||||||
|
""")
|
||||||
|
|
||||||
|
task = SequentialTask(
|
||||||
|
processes = [hello],
|
||||||
|
resources = Resources(cpu = 1.0, ram = 128*MB, disk = 128*MB))
|
||||||
|
|
||||||
|
jobs = [Service(
|
||||||
|
task = task, cluster = 'devcluster', role = 'www-data', environment = 'prod', name = 'hello')]
|
||||||
|
```
|
||||||
|
|
||||||
|
Now we create the job:
|
||||||
|
```
|
||||||
|
aurora job create devcluster/www-data/prod/hello hello_world.aurora
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
### Creating a docker-compose-executor job
|
||||||
|
Now that we have a thermos job running, it's time to launch a docker-compose job.
|
||||||
|
|
||||||
|
We'll be using the gorealis library sample client to send a create job request
|
||||||
|
to the scheduler, specifiying that we would like to use the docker-compose executor.
|
||||||
|
|
||||||
|
Furthermore, we will be specifying what resources we need to download in order to
|
||||||
|
successfully run a docker compose job.
|
||||||
|
|
||||||
|
The job configuration in the sample client looks like this:
|
||||||
|
```
|
||||||
|
job = realis.NewJob().
|
||||||
|
Environment("prod").
|
||||||
|
Role("vagrant").
|
||||||
|
Name("docker-compose").
|
||||||
|
ExecutorName("docker-compose-executor").
|
||||||
|
ExecutorData("{}").
|
||||||
|
CPU(1).
|
||||||
|
Ram(64).
|
||||||
|
Disk(100).
|
||||||
|
IsService(false).
|
||||||
|
InstanceCount(1).
|
||||||
|
AddPorts(1).
|
||||||
|
AddLabel("fileName", "sample-app/sample-app.yml").
|
||||||
|
AddURI("https://dl.bintray.com/rdelvalle/mesos-compose-executor/sample-app.tar.gz", true, true)
|
||||||
|
```
|
||||||
|
|
||||||
|
Now we run the client sending the create job command to Aurora:
|
||||||
|
```
|
||||||
|
go run $GOPATH/src/github.com/rdelval/gorealis/examples/Client.go -executor=compose -url=http://192.168.33.7:8081 -cmd=create
|
||||||
|
```
|
||||||
|
|
||||||
|
If everything went according to plan, a new job will be shown in the Aurora UI.
|
||||||
|
|
||||||
|
We can further investigate inside the Mesos task sandbox.
|
||||||
|
|
||||||
|
Inside the sandbox, under the sample-app folder, we can find a docker-compose.yml-generated.yml.
|
||||||
|
|
||||||
|
If we inspect this file, we can find the port at which we can find the web server we launched.
|
||||||
|
|
||||||
|
Under Web->Ports, we find the port Mesos allocated. We can then navigate to:
|
||||||
|
`<agent address>:<assigned port>`
|
||||||
|
|
||||||
|
And a message from the executor should greet us.
|
||||||
|
|
||||||
|
## Cleaning up
|
||||||
|
|
||||||
|
### Using the Default Client
|
||||||
|
|
||||||
|
```
|
||||||
|
$ aurora job killall devcluster/www-data/prod/hello
|
||||||
|
$ aurora job killall devcluster/vagrant/prod/docker-compose
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
### Using gorealis
|
||||||
|
|
||||||
|
```
|
||||||
|
$ go run $GOPATH/src/github.com/rdelval/gorealis/examples/Client.go -executor=compose -url=http://192.168.33.7:8081 -cmd=kill
|
||||||
|
$ go run $GOPATH/src/github.com/rdelval/gorealis/examples/Client.go -executor=thermos -url=http://192.168.33.7:8081 -cmd=kill
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue