Akash Kothawale
acf9332a50
Comments: Explain what Variance is used for.
2018-09-23 18:16:27 -04:00
Akash Kothawale
2d3158d99a
pcp: Remove unused functions
2018-09-23 18:16:17 -04:00
Akash Kothawale
c52bfb08c5
Handle case: host -> slaveID may not be available
2018-09-23 18:16:06 -04:00
Akash Kothawale
3d5bb772fb
Fix: Mutex lock handling issues with task running
2018-09-23 18:15:52 -04:00
Akash Kothawale
9c09321d81
Fix incorrect degColLog file location
2018-09-23 18:14:48 -04:00
Akash Kothawale
39d1cf980c
Use existing Mean and Variance functions
...
Available in github.com/montanaflynn/stats
2018-09-23 18:13:47 -04:00
Akash Kothawale
46be28ef8d
pcp: CPU/MEM utilization & task share variance per node
2018-09-23 18:13:28 -04:00
Akash Kothawale
abadb9a328
degCol: Logger for storing degree of colocation
...
degCol: logger
2018-09-23 18:10:59 -04:00
Pradyumna Kaushik
b58ffdbdd6
formatted code
2018-09-23 18:10:31 -04:00
Pradyumna Kaushik
435c4ca1bc
fixed bug in schedWindow resizing strategy, where earlier it was stopping after checking if the first task in the queue can be scheduled in the next offer cycle. Changed baseScheduler to store the schedWindowResizingStrategy instead of the criteria. Retrofitted the scheduling policies to use the scheduling window resizing strategy directly from baseSchedRef.
2018-09-23 18:10:02 -04:00
Pradyumna Kaushik
3ebd7b0c7e
added utility to compute the scheduling window. Right now there's only criteria on which this is determined -- fillNextOfferCycle. So, the schedWindow is the max number of tasks, that aren't yet scheduled, whose aggregate resource requirement is as close as possible to the resource available in the next round of resource offers. To be able to make the most use of the next offer cycle, one would need to perform a non-polynomial search of the TaskQueue and as this is computationally expensive, a linear search is performed on the TaskQueue. Retrofitted scheduling policies to also call utilities.schedUtils#schedWindowResizingStrategy#Apply before switching to a new scheduling policy.
2018-09-23 18:03:14 -04:00
Akash Kothawale
6f0f3788b9
sched: Make baseScheduler publicly accessible
2018-09-23 17:55:12 -04:00
Pradyumna Kaushik
9599588fb5
formatted code.
2018-09-23 17:49:08 -04:00
Pradyumna Kaushik
c31bf8db01
BUG Fix1: taskID creation in taskUtils#initTaskResourceRequirements. BUG Fix2: Converted value in utilities/trackResourceUsage#TrackResourceUsage#perHostResourceAvailability to be a pointer to ResourceCount.
2018-09-23 17:46:38 -04:00
Pradyumna Kaushik
657dc8df93
Track resource usage across the cluster. Created utility in utilities/ to track the total and the unused resources for each host in the cluster. Added utility to def/taskUtils.go to retrieve the resource requirement for a given taskID. Decoupled the code, to launch a list of tasks on a set of offerIDs, to schedulers/helpers.go and updated all the scheduling policies to call this function instead of directly calling mesos.SchedulerDriver#LaunchTasks. The resource availability of the cluster is updated at 2 stages -- 1. When the tasks are about to be launched (in schedulers/helpers.go#LaunchTasks), the scheduling policy switching logic will be able to adhere to the update in the resource availability due to the JUST launched tasks and 2. when a terminal status update is received for a task (in schedulers/base.go#statusUpdate).
2018-09-23 17:44:36 -04:00
Pradyumna Kaushik
6cd61ed18b
comment cleanup.
2018-09-23 17:33:34 -04:00
Akash Kothawale
3b80af6d8f
Fix: mesos-go imports
2018-09-23 17:31:03 -04:00
Pradyumna Kaushik
065705d480
Merged in experimentation/schedPolicySwitcher (pull request #1 )
...
Experimentation/schedPolicySwitcher
1. Initial commit for consolidated loggers using observer pattern.
2. class factory for schedulers.
3. Using the scheduling policy class factory in schedulers/store.go and the scheduler builder helpers in schedulers/helpers.go, feature to be able to be able to plug a scheduling policy of your choice from the command line (right now only first-fit and bin-packing are possible. Will be updating the class factory to include other scheduling policies as well.
4. Removed TODO for using generic task sorters. Modified TODO for a config file input to run electron.
5. Added other schedulers to the factory
6. Partially retrofitted the other scheduling policies to use the logging library.
7. Retrofitted extrema and progressive to use the consolidated logging library. Fixed parameter issue with s.base.Disconnected(). Formatted project
8. Move statusUpdate(...) into base.go to remove redundant code.
9. Converted the baseScheduler into a state machine where the state is a scheduling policy that defines an approach to consume resource offers.
10. Added another command line argument to be used to enable switching of scheduling policies. Retrofitted scheduling policies to switch only if the particular feature has been enabled.
changed argument to coLocated(...) to take base type rather than ElectronScheduler type. Also, prepended the prefix to the directory of the logs so that it would be easier to determine what the files in a directory correspond to without viewing the contents of the directory.
Defined methods in ElectronScheduler. Each of these methods corresponds to a type of log that an ElectronScheduler would make. Each of these methods would need to be implemented by the scheduling policy.
Electron has only one scheduler that implements the mesos scheduler interface. All the scheduling policies are just different implementations of ways to consume mesos resource offers. Retrofitted scheduling policies to now embed SchedPolicyState instead of baseScheduler.
Approved-by: Pradyumna Kaushik <pkaushi1@binghamton.edu>
2018-09-23 17:19:28 -04:00
Pradyumna Kaushik
cb71153362
Sorting the tasks in non-decreasing order of watts for max-greedymins and max-min. Had unintentionally removed the sorting of tasks for max-greedymins and max-mins.
2018-01-18 18:30:27 -05:00
Pradyumna Kaushik
1b9442bf50
Fixed missing default value issue in Mesos master command line argument.
2017-10-03 20:10:44 -04:00
Pradyumna Kaushik
3d07efe15d
removed default value for -master commandline argument.
2017-10-03 01:38:34 -04:00
Pradyumna Kaushik
bb5df1e3ad
Linked to corresponding documentation
2017-09-28 15:56:16 -04:00
Pradyumna Kaushik
ad9f67a2a0
moved documentation of scheduling policies and power-capping strategies into a documentation folder called docs/
2017-09-28 15:53:36 -04:00
Pradyumna Kaushik
4051aa1fb3
changed the named of the scheduling policy documentation to SchedulingPolicies.md. Also, added a documentation for power-capping strategies called PowerCappingStrategies.md
2017-09-28 15:51:22 -04:00
Pradyumna Kaushik
b807625b78
Fixed the comments to be capitalized at the start and also terminate with a period.
2017-09-28 15:36:47 -04:00
Pradyumna Kaushik
577120ae7c
removed .gitignore from the .gitignore file so as to include the gitignore file and start tracking.
2017-09-28 15:14:34 -04:00
Pradyumna Kaushik
c2b42c2662
Now retrieving path to rapl throttle script by reading value of an environment variable. Also, stored the RAPL_PKG_Throttle.py script in a directory called scripts/. The location of this needs to be stored in an environment variable named RAPL_PKG_THROTTLE_LOCATION.
2017-09-28 15:01:06 -04:00
Pradyumna Kaushik
dcf67661e9
removed the inlining of ssh password for username=rapl. Now we read value of environment variable named RAPL_PSSWD
2017-09-27 19:52:17 -04:00
Pradyumna Kaushik
971f6833c3
modified schedulers/README.md to now use Elektron instead of Electron. Also, changed the content to just display the contained scheduling policies with an explaination for each of them.
2017-09-26 13:43:23 -04:00
Pradyumna Kaushik
04f24beac5
scheduling policies pluggable from commandline
2017-09-26 13:17:47 -04:00
Pradyumna Kaushik
051aca4d10
updated pcp config file to not include stratos information
2017-09-26 01:40:17 -04:00
Pradyumna Kaushik
978788a5aa
updated example workload definition
2017-09-26 01:37:28 -04:00
Pradyumna Kaushik
60e5e0ecc1
added max-min scheduling policy
2017-09-26 01:14:39 -04:00
Pradyumna Kaushik
3fa4f3d0e6
changed commandline argument descriptions. Changed the names of the scheduling policies and the structs.
2017-09-26 01:05:18 -04:00
Pradyumna Kaushik
59266d207d
retrofitted to use the generic task sorter.
2017-09-26 00:37:12 -04:00
Pradyumna Kaushik
180e85ddfa
removed unnecessary constants
2017-09-26 00:31:45 -04:00
Pradyumna Kaushik
2cb09ece1f
fixed importing issues in power-capping strategies. Changed names of power-capping strategies to extrema.go and progressive-extrema.go
2017-09-26 00:26:01 -04:00
Pradyumna Kaushik
9897c983fe
removed proactive-clusterwide power-capper. Retrofitted package names to use elektron and not electron
2017-09-26 00:18:35 -04:00
Pradyumna Kaushik
1c4b6f0f01
refined README and removed unwanted scheduling policies.
2017-09-26 00:05:19 -04:00
Pradyumna Kaushik
dc279801b7
Merged in genericTaskSorter (pull request #18 )
...
GenericTaskSorter
Approved-by: Renan DelValle <rdelval1@binghamton.edu>
Approved-by: ajain13 <ajain13@binghamton.edu>
2017-09-05 18:29:13 +00:00
Pradyumna Kaushik
3a9db01fa0
formatted files
2017-08-26 22:33:06 -04:00
Pradyumna Kaushik
8bc408dea1
fixed comments. Removed an old comment that no longer made sense.
2017-08-26 15:43:09 -04:00
Pradyumna Kaushik
9e620eaede
changed def.TaskSorter(...) to def.SortTasks(...) for semantics. Now, def.SortTasks(...) also sorts the tasks instead of returning just a func (i, j int) bool {}. Removed TODO from the README associated with this PR
2017-08-26 15:39:45 -04:00
Pradyumna Kaushik
0869bea2d8
Used closures instead of strings for the generic sorter. This removed the need for reflection.
2017-08-26 15:19:30 -04:00
Pradyumna Kaushik
3c65bdf02e
changed the scope of TaskResourceNames and SortCriteria to be package private.
2017-08-23 20:10:33 -04:00
Pradyumna Kaushik
c2c7a2c090
Added TODO to use the generic task sorters and get rid of the existing sorters present in def/task.go
2017-08-23 20:06:44 -04:00
Pradyumna Kaushik
f29b7f51a9
Added a generic task sorter utility. This allows for an array of tasks to be sorted based on any resource. The possible resources by which an array of tasks can be sorted currently are listed in def/sortingCriteria.go. One can add more to this if required.
2017-08-23 19:35:19 -04:00
Pradyumna Kaushik
9b9dc73269
Merged in kmeansTaskClassification (pull request #17 )
...
KmeansTaskClassification
Approved-by: ajain13 <ajain13@binghamton.edu>
Approved-by: Renan DelValle <rdelval1@binghamton.edu>
2017-08-23 02:57:33 +00:00
Pradyumna Kaushik
235ed189d5
Moved the TODO, for the refinement of the cluster sorting algorithm, to the main README file.
2017-08-22 13:09:05 -04:00
Pradyumna Kaushik
e1350838f3
Merge branch 'kmeansTaskClassification' of https://bitbucket.org/sunybingcloud/electron into kmeansTaskClassification
2017-08-22 13:03:17 -04:00
