Commit graph

261 commits

Author SHA1 Message Date
Renan DelValle
1d3854aa5f
Trace level for logger (#94)
* Add trace level to print out response thrift objects. Allows user to control whether these are printed or not to avoid pollution.

* Using named parameters to be more explicit about what is being set for LevelLogger.

* Adding TracePrint and TracePrintln. Inlined library level prefixes.
2019-01-10 16:58:59 -08:00
Renan DelValle
73e7ab2671
Releasing version 1.3.1 2019-01-08 15:57:19 -08:00
Renan DelValle
22b1d82d88
Bug fix for logger interface. Varidic arguments need to be unrolled when passed to print functions. 2019-01-08 15:37:25 -08:00
Renan DelValle
2f7015571c
Adding support for setting GPU as a resource. (#93)
* Adding support for setting GPU as a resource.
* Refactoring pulse update test.
2019-01-08 15:11:52 -08:00
Robert Allen
296af622d1 This adds the following function to the PartitionPolicy configuration to the Job interface (#91)
* Adding Partition Policy API
2018-12-20 14:38:06 -08:00
Renan DelValle
9a835631b2
Running goimports on all repository to conform to newest goimports. 2018-12-19 15:33:35 -08:00
Renan DelValle
b100158080
Updating Travis CI config file to include running CI on master-v2.0 branch 2018-12-19 15:30:22 -08:00
Renan DelValle
45a4416830
Adding .gitattributes to ignore generated files. 2018-12-03 16:09:46 -08:00
Renan DelValle
2eaa60f681
Support Drain SLA API (#88)
* Bringing thrift API up to date with Aurora 0.21.0.

* Adding support for SLA Drain Host API.
2018-11-16 11:41:09 -08:00
Renan DelValle
a09a18ea3b
Stop retrying if we find a permanent url error. (#85)
* Detecting if the transport error was not temporary in which case we stop retrying. Changed bug where get results was being called before we checked for an error.

* Adding exception for EOF error. All EOF errors will be retried.

* Addressing race conditions that may happen when client is closed or connection is re-established.

* Adding documentation about how this particular implemantion of the realis client uses retries in scenarios where a temporary error is found.
2018-11-01 17:00:03 -07:00
Renan DelValle
6762c1784b
Bug fix: get quota and set quota would not retry if an error was hit. (#84) 2018-10-29 14:56:24 -07:00
Renan DelValle
fa5133c13d
Test coverage improvement (#83)
* Adding tests for getPendingReasons and startMaintenance.

* Added tests for ThriftBinary and ThriftJSON.

* Adding test for NOOP Logger.
2018-10-28 19:16:44 -07:00
JC Martin
5de913493c Add Start Maintenance and Get Pending Reason (#82)
* Add startMaintenance

* Add getPendingReason
2018-10-26 11:38:03 -07:00
Renan DelValle
2306d6180f
Adding force Implicit and force Explicit recon to gorealis. (#81) 2018-10-22 16:43:35 -07:00
Renan DelValle
231793df71
Adding a separate function to add dedicated attributes. (#80)
Dedicated wrapper for "dedicated" constraints
2018-10-11 09:43:35 -07:00
Renan DelValle
e0f33ab60e
Bumping up the version number advertised by gorealis to the scheduler. 2018-10-05 08:09:30 -07:00
Renan DelValle
9dcb7a8969
Moving the Codecov badge to right beside the Travis CI badge. 2018-10-05 08:09:05 -07:00
Renan DelValle
4395c2ae1a
Code coverage (#79)
* Turning on codecoverage from Codecov.
2018-10-05 07:57:19 -07:00
Renan DelValle
70252ffacf Updating Aurora compatibility in anticipation of next release. 2018-10-04 18:46:27 -07:00
Renan DelValle
4963bbb922 Sharling layers in docker compose between agent and master. 2018-10-04 18:46:27 -07:00
Renan DelValle
149d03988c
Sample Client cleanup, misc cleanup (#74)
* Changing print + os.exit to log.Fatal. Leaving a TODO to move documentation to interface.
2018-10-04 11:28:32 -07:00
Renan DelValle
037c636d6d
Retry switch fallthrough fix and create multiple tests (#77)
* Bugfix: switch statements were missing fallthrough statement thus making them retry non-retriable errors. Using a list to catch cases now.

* Adding tests for CreateService, createService when the executor doesn't exist, and createJob when the executor doesn't exist. Renamed Pulse test to reflect that it's using CreateService instead of CreateJob.

* Repsonse propagate back up to caller for context for CreateJob, CreateService, and StartJobUpdate.

* Deleting PR template as Travis CI takes care of running tests and formatting tests now.
2018-10-04 10:47:08 -07:00
Renan DelValle
9ebf118e71
Create job bevaviour does not override default batch size. (#75) 2018-09-25 16:37:17 -07:00
Renan DelValle
e85781e6d4
Upgrade Aurora to 0.21.0 and Mesos to 1.5.1 for compose setup. 2018-09-14 16:38:05 -07:00
Renan DelValle
5099d7e6ec
Adding force snapshot and force backup APIs (#73)
* Adding force snapshot and force backup APIs.
2018-09-14 15:04:16 -07:00
Renan DelValle
0f2ece10ac
Ignoring vendor folder when checking for goimports failure. 2018-09-13 17:22:04 -07:00
Renan DelValle
ad0da8c867
Adding goimports check. From here on in, any PR that doesn't pass goimports will fail the CI build. 2018-09-13 17:14:38 -07:00
Renan DelValle
48318e026c
Fixing issues caught by goimports before adding goimports check to CI. 2018-09-13 17:02:15 -07:00
Renan DelValle
98d2fa2dd7
Forking Thrift Go library to use 0.10.0 with THRIFT-4215 and THRIFT-4219 on top of it in hopes of fixing a stray nil buffer error. (#72)
This should fix #65
2018-08-21 08:20:41 -07:00
Renan DelValle
1c2b1c5079
Continous integration through Travis CI (#71)
* Adding Travis CI badge

* Modifying end to end tests to reflect testing against docker-compose setup in Travis CI.

* Adding bash script to run simple container with tests within bridge network for Mac.

* Adding documentation for setting up a developer environment.

* Decreasing amount of CPU needed for CreateJobWithPulse because a higher value causes Travis CI to hang.
2018-08-13 20:09:25 -07:00
PRADYUMNA KAUSHIK
0e4a0d726b Fix JSON client example and Update documentation. (#67)
* Updated the JSON client to be consistent with the library.
The JSON client requires two JSONs,
1. JOB json -- contains job description.
2. Config json -- contains configuration information such as username,
	password, schedulerUrl, zookeeper cluster configuration etc.

* Job json using docker-compose executor.

Used https://github.com/paypal/dce-go/blob/develop/examples/client.go#L50
to create a json file for a job that uses the docker-compose executor.
The current job json file (examples/job.json) uses an outdated version
of docker-compose executor. Once examples/client.go has been modified
to use examples/job_dce.json, it should be okay to get rid of
examples/job.json.

* Run thermos jobs using json client.

Added an extra field to JobJson, ExecutorDataFile, that holds
the path to the json file representing the executor configuration
data.
Added a new example job json file (examples/job_thermos.json) that
is to be passed to the json client along with the config file to
run a thermos job.

* Using scheduler URL instead of leader from zk.

The endpoints returned by ZKEndpoints(...) is not reachable
from outside the vagrant box. Hence, using the scheduler URL
directly.

* Added docs for using dce-go and json client.

* Place json client docs in separate subsection.

* Config now embeds realis.Cluster to be backwards compatible with
the python client cluster.json file.
Changed the type of Transport to string to stay flexible if new
transport types come up. JSON is used as the default transport option
is not transport is provided with the config.
2018-07-13 11:14:11 +02:00
Ezequiel Torres Feyuk
fe567ee966 Task query optional parameters (#69)
* Change TaskQuery struct parameters to optional

* Thrift API is modified to make all the parameters in the
  TaskQuery struct optional

* Autogenerated code is regenerated

* Changes in TaskQuery structs used in the project

* Now that TaskQuery receive optional values, pointers
  instead of values must be passed to the struct
2018-06-28 11:48:28 -07:00
Renan DelValle
6c8ab10b64 Merge develop branch into master (#68)
* Fixing possible race condition when passing backoff around as a pointer.

* Adding a debug logger that is turned off by default.
Info logger is enabled by default but prints out less information.

* Removing OK Aurora acknowledgment.

* Making Mutex a pointer so that there's no chance it can accidentally be copied.

* Changing %v to %+v for composite structs. Removing a repetitive statement for the Aurora return code.

* Removing another superflous debug statement.

* Removing a leftover helper function from before we changed how we configured the client.

* Changing the logging paradigm to only require a single logger. All logging will be disabled by default. If debug is enabled, and a logger has not been set, the library will default to printing all logging (INFO and DEBUG) to the stdout.

* Minor changes to demonstrate how a logger can be used in conjunction to debug mode.

* Removing port override as it is not needed

* Changing code comments to reflect getting rid of port override.

* Adding port override back in.

* Bug fix: Logger was being set to NOOP despite no logger being provided when debug mode is turned on.

* Turn on logging by default.

* Removing option to override schema and ports for information found on Zookeeper.

* Turning off debug mode for tests because it's too verbose. Making sure LevelLogger is initialized correctly under all scenarios.

* Removing override fields for zk config.

* Remove space.

* Removing info that is now incorrect about zk options.
2018-06-22 12:57:21 -07:00
Renan DelValle
8ca953f925
Bug fix: using AND in place of OR or SSL flags. (#64)
* Bug fix: using AND in place of OR or SSL flags.

* Separating CA certificate path and client key and cert addition to options.
2018-05-29 12:46:16 -07:00
kkrishna
800efccb31
Merge pull request #63 from paypal/addSSLToExample
Add ssl to example client, misc doc fixes
2018-05-23 11:43:58 -07:00
Renan DelValle
5d12029227
Update PR template to hide away instructions on submission. 2018-05-22 17:00:30 -07:00
Renan DelValle
4f6a5e9741
Adding SSL flags to sample client. 2018-05-22 16:56:42 -07:00
Renan DelValle
e6b204b9da
Removing unnecessary space. 2018-05-13 18:34:34 -07:00
Renan DelValle
d03a7b61e4
Removing napping from the TODO list as go's native http libraries are good enough. 2018-05-13 18:32:38 -07:00
Renan DelValle
4f5766b443
Misc. bug fixes and addition of debug logging (#61)
* Fixing possible race condition when passing backoff around as a pointer.

* Adding a debug logger that is turned off by default. If debug is turned on, but a logger has not been assigned, a default logger that will print to STDOUT will be created.

* Making Mutex a pointer so that there's no chance it can accidentally be copied.

* Removing a leftover helper function from before we changed how we configured the client.

* Minor changes to demonstrate how a logger can be used in conjunction to debug mode in the sample client.
2018-04-13 11:03:29 -07:00
Robert Allen
c0d2969976 Adding Admin Client calls GetQuota & SetQuota (#59)
* Adding Admin Client calls `GetQuota` & `SetQuota`

This change set adds admin client calls to fetch and
mutate the OwnerRole quota[cpu,ram,disk].
2018-03-07 16:24:27 -08:00
kkrishna
66809c55f7
Merge pull request #58 from paypal/zkPolish
Zookeeper functions and retry functions cleanup
2018-03-05 12:09:18 -08:00
Renan DelValle
acc54c1015
Adding logging when there is a client error. 2018-03-05 11:20:39 -08:00
Renan DelValle
0bb23cec71
Adding unit tests for Zookeeper related functions to prevent regressions. 2018-03-03 14:13:47 -08:00
Renan DelValle
3d62df1684
* Errors have been refactored.
* ZK retries have been cleaned up. We will now retry after every error
EXCEPT when we have a badly formed path.
* ZK library has been reworked with optional arguments pattern to not be
so intertwined with the cluster.json file.
* Timeout error has been re-implemented as RetryError. RetryError
behaves like a Timeout error but is used exclusively to add more context
privately. This allows us to have unit tests that check our retry
mechanism is actually retrying.
* Additional logging has been added to retry mechanisms as well as to
the Zookeeper library we use.
2018-03-03 14:08:04 -08:00
Sivaram Mothiki
dc327bebad change config for certs path (#57)
Bug fix. Changing hardcoded certificate to one provided by configuration object.
2018-03-02 15:21:45 -08:00
Renan DelValle
a43dc81ea8
Simplifying retry mechanism for Thrift Calls (#56)
* Deleting permament error as it doesn't make sense. Just return a plain old error and that will be considered permanent.

* Removing double closure at as it's unmaintainable and can be error prone. Separated back offs into a generic one and a thrift call specific one.

* ZK leader finder now returns a temporary error instead of constantly no leader found and quitting. It could be that the leader info is being propagated so it's worth trying another time.

* Adding more logging to the retry.

* Wrapping lock and unlock in an anonymous function so that we can use defer on unlock such that it is called in the case of a panic.
2018-02-15 15:16:39 -08:00
Renan DelValle
64948c3712
Backoff mechanism fix (#54)
* Fixing logic that can lead to nil error being returned and retry stopping early.

* Fixing possible code path that may lead to an incorrect nil error.
2018-02-06 12:44:27 -08:00
kkrishna
a6b077d1fd Aurora jobupdate functionality -- pause/resume/pulse api (#55)
* Adding GetJobs api

* Adding Aurora pause/resume/pulse api
2018-02-06 12:39:02 -08:00
kkrishna
8bd3957247 GetJobs api (#53)
* GetJobs API added
2018-01-27 10:33:55 -08:00