Akash Kothawale
46be28ef8d
pcp: CPU/MEM utilization & task share variance per node
2018-09-23 18:13:28 -04:00
Pradyumna Kaushik
3ebd7b0c7e
added utility to compute the scheduling window. Right now there's only criteria on which this is determined -- fillNextOfferCycle. So, the schedWindow is the max number of tasks, that aren't yet scheduled, whose aggregate resource requirement is as close as possible to the resource available in the next round of resource offers. To be able to make the most use of the next offer cycle, one would need to perform a non-polynomial search of the TaskQueue and as this is computationally expensive, a linear search is performed on the TaskQueue. Retrofitted scheduling policies to also call utilities.schedUtils#schedWindowResizingStrategy#Apply before switching to a new scheduling policy.
2018-09-23 18:03:14 -04:00
Akash Kothawale
3b80af6d8f
Fix: mesos-go imports
2018-09-23 17:31:03 -04:00
Pradyumna Kaushik
065705d480
Merged in experimentation/schedPolicySwitcher (pull request #1 )
...
Experimentation/schedPolicySwitcher
1. Initial commit for consolidated loggers using observer pattern.
2. class factory for schedulers.
3. Using the scheduling policy class factory in schedulers/store.go and the scheduler builder helpers in schedulers/helpers.go, feature to be able to be able to plug a scheduling policy of your choice from the command line (right now only first-fit and bin-packing are possible. Will be updating the class factory to include other scheduling policies as well.
4. Removed TODO for using generic task sorters. Modified TODO for a config file input to run electron.
5. Added other schedulers to the factory
6. Partially retrofitted the other scheduling policies to use the logging library.
7. Retrofitted extrema and progressive to use the consolidated logging library. Fixed parameter issue with s.base.Disconnected(). Formatted project
8. Move statusUpdate(...) into base.go to remove redundant code.
9. Converted the baseScheduler into a state machine where the state is a scheduling policy that defines an approach to consume resource offers.
10. Added another command line argument to be used to enable switching of scheduling policies. Retrofitted scheduling policies to switch only if the particular feature has been enabled.
changed argument to coLocated(...) to take base type rather than ElectronScheduler type. Also, prepended the prefix to the directory of the logs so that it would be easier to determine what the files in a directory correspond to without viewing the contents of the directory.
Defined methods in ElectronScheduler. Each of these methods corresponds to a type of log that an ElectronScheduler would make. Each of these methods would need to be implemented by the scheduling policy.
Electron has only one scheduler that implements the mesos scheduler interface. All the scheduling policies are just different implementations of ways to consume mesos resource offers. Retrofitted scheduling policies to now embed SchedPolicyState instead of baseScheduler.
Approved-by: Pradyumna Kaushik <pkaushi1@binghamton.edu>
2018-09-23 17:19:28 -04:00
Pradyumna Kaushik
1b9442bf50
Fixed missing default value issue in Mesos master command line argument.
2017-10-03 20:10:44 -04:00
Pradyumna Kaushik
3d07efe15d
removed default value for -master commandline argument.
2017-10-03 01:38:34 -04:00
Pradyumna Kaushik
b807625b78
Fixed the comments to be capitalized at the start and also terminate with a period.
2017-09-28 15:36:47 -04:00
Pradyumna Kaushik
04f24beac5
scheduling policies pluggable from commandline
2017-09-26 13:17:47 -04:00
Pradyumna Kaushik
3fa4f3d0e6
changed commandline argument descriptions. Changed the names of the scheduling policies and the structs.
2017-09-26 01:05:18 -04:00
Pradyumna Kaushik
9897c983fe
removed proactive-clusterwide power-capper. Retrofitted package names to use elektron and not electron
2017-09-26 00:18:35 -04:00
Pradyumna Kaushik
41ef269c62
fixed indentation
2017-08-22 12:56:36 -04:00
Abhishek Jain
9546529f80
Corrected a spelling mistake in a print statement
2017-03-24 17:13:21 -04:00
Pradyumna Kaushik
531c1a120b
Fixed messages for commandline arguments.
2017-02-20 23:49:27 -05:00
Pradyumna Kaushik
726c0555ed
Fixed corner cases in progressive extrema -- When a node is capped and the new cap value is above a threshold then that node can be capped or uncapped in the next cycle. If the new cap value is equal to the threshold then the node cannot be capped further and can only be uncapped. When the node is uncapped and the newUncapValue is below 100 then the node can be capped or uncapped in the next cycle. If the newUncapValue is 100 then the node can only be capped.
2017-02-20 20:55:06 -05:00
Pradyumna Kaushik
57512ac2dd
Changed ignoreWatts to wattsAsAResource. This resulted in flipping of the condition checks that use these variablees
2017-02-09 18:41:40 -05:00
Pradyumna Kaushik
fdcb401447
Made classMapWatts a commandLine option where one can enable and disable mapping of watts to powerclasses when accepting offers from Mesos. Removed the schedulers that were created solely for the classMapWatts feature. Retrofitted all schedulers to use the powerClass mapped watts attribute for a task, if classMapWatts has been enabled. Removed unnecessary functions and variables from constants.go. Removed unnecessary functions from utilities/utils.go. Fixed operator precendence issue with takeOffer(...) in some of the schedulers. Added TODO to decouple capping strategies from the schedulers completely. Added TODO to move all the common struct attributes in the schedulers into base.go.
2017-02-09 18:05:38 -05:00
Pradyumna Kaushik
0d79079bf6
changed scheduler to binPacked
2017-02-05 14:59:34 -05:00
Pradyumna Kaushik
395917a97e
changed pcplogger to pcplog and extrema
2017-02-05 01:08:48 -05:00
Pradyumna Kaushik
eea0db0b3d
retrofitted schedulers by renaming 'taken' to 'offerTaken' for the boolean to indicate whether an offer has been consumed.
2017-02-04 16:59:25 -05:00
Pradyumna Kaushik
04d722d20f
removed the new commandline argument for powerClass. This will be part of another patch to the code.
2017-01-31 15:13:55 -05:00
Pradyumna Kaushik
92e432c312
formatted code
2017-01-28 21:37:01 -05:00
Pradyumna Kaushik
3a7c0dc33b
changed message for command-line argument 'powerClass'
2017-01-28 21:36:33 -05:00
Pradyumna Kaushik
ee0cada747
changed name of command-line argument 'classMapWatts' to 'powerClass'
2017-01-28 21:33:36 -05:00
Pradyumna Kaushik
3119724154
Added another command-line argument to indicate whether to use the Watts attribute or the ClassToWatts attribute in the workload JSON.
2017-01-28 21:30:15 -05:00
Pradyumna Kaushik
6b0f4e3729
removed commented lines. Changed the scheduler to BinPackedSortedWattsSortedOffers.
2017-01-28 17:48:15 -05:00
Pradyumna Kaushik
8c6ad36b5e
changed name of StartLogAndDynamicCap(...) to StartPCPLogAndExtremaDynamicCap(...).]
2017-01-15 15:23:57 -05:00
Pradyumna Kaushik
15c4f04d75
changed the scheduler to ffswClassMapWatts
2017-01-14 19:57:01 -05:00
Pradyumna Kaushik
118e0a571a
changed scheduler to BPSWClassMapWatts. Also, capping is extrema.
2017-01-06 16:46:31 -08:00
Pradyumna Kaushik
d5860a91ae
fixed merge conflicts
2017-01-06 15:58:50 -08:00
Pradyumna Kaushik
58bc24cb72
Changed the scheduler to run
2017-01-06 15:57:08 -08:00
Renan DelValle
776bffb100
Finished bin packed max min.
2017-01-04 20:25:54 -05:00
Renan DelValle
2678032c2c
Go fmt on the entire project
2017-01-03 20:57:25 -05:00
Renan DelValle
b73c30a8bf
Implemented a scheduling tracer. Logs time date and host:task-ID. Made log prefix more global since it needs to be constant for all log files generated.
2017-01-03 20:30:21 -05:00
Pradyumna Kaushik
de4df8ba7b
switched scheduler to now be an instance of PistonCapper.
2016-12-22 22:59:00 -05:00
Pradyumna Kaushik
b0140a8b93
Using ranked proactive cluster wide capper with clever recapping
2016-12-22 22:59:00 -05:00
Pradyumna Kaushik
a1c8319b81
changed extrema to non-extrema. This was done so that proactive cluster wide capping scheme doesn't conflict with the extrema capping scheme.
2016-12-22 22:58:59 -05:00
Pradyumna Kaushik
a35e43da51
scheduler is now an instance of ProactiveClusterwideCapFCFS
2016-12-22 22:58:59 -05:00
Renan DelValle
bf9a5a6005
Finished integrating dynamic capping policy that uses a high and a low threshold to start capping and uncapping node via an ssh commands that triggers a python script. Commit has sensitive data, scrub before releasing to public.
2016-12-22 22:58:59 -05:00
Renan DelValle
fce62981da
Moved schedulers from the main programs to schedulers package. Can now choose different scheduelrs to use. Work on code sharing between schedulers remains to be done.
2016-12-22 22:58:58 -05:00
Renan DelValle
6a406cad69
SIGINT is now captured. Linux/MacOS shutdown for pcpdumptext child
...
processes added to clean up after we finish running. Made all tasks
run on bridged network mode so that benchmarks that heavily utilize
network are protected from eachother. I.e: Tradebeans, tradesoap
2016-12-22 22:58:58 -05:00
Renan DelValle
086b06256d
Added Error state. Fixed bug with tasks returning error because 0.0 watts was requested as a resource. Changed name to be more readable by eliminating electron. PCP logs can now have a prefix.
2016-12-22 22:58:58 -05:00
Renan DelValle
6878bf55ec
Added the ability to prefix a pcplog
2016-12-22 22:58:58 -05:00
Renan DelValle
9fab9d3260
Adding shorthands
2016-12-22 22:58:58 -05:00
Renan DelValle
dd7a42f582
Added ability to tie benchmark to single node. Uses offer hostname for this, maybe a better soluition would be to look at offer attributes. Added shorthand for workloads flag -w
2016-12-22 22:58:58 -05:00
Renan DelValle
e1e8a33e40
Added option to run without taking watts
2016-12-22 22:58:58 -05:00
Renan DelValle
2575c2a20b
PCP is now controlled by main scheduler. It will start recording upon the acceptance of the very first offer. pcp is now it's own package. README has been updated with instructions on how to create workloads and running instructions.
2016-12-22 22:58:58 -05:00
Renan DelValle
3cb60d0ca2
Detection of co-scheduled benchmarks is complete
2016-12-22 22:58:58 -05:00
Renan DelValle
e0dc0c7368
Fixed deadlock issue while running a large number of benchmarks. Changed names of tasks to indicate what they are running. Added name to task schema to append it to the name more easily.
2016-12-22 22:58:58 -05:00
Renan DelValle
8ddfe8e585
Bug fix for running multiple different benchmarks. Loop did not exit upon being able to take an offer.
2016-12-22 22:58:58 -05:00
Renan DelValle
3679510909
Electron now launches a series of benchmarks and then shuts down when everything has been sucessfully scheduled
2016-12-22 22:58:58 -05:00
