Commit graph

55 commits

Author SHA1 Message Date
Bhargavi Hanumant Alandikar
3543960689 Elektron Logging library (#16)
switch to logrus for logging.

replaced old logging library with a wrapper around logrus.
We now just need to use the exported Log(...) and Logf(...) from the logging/
package that wraps around a set of loggers constituting a chain (following COR).
Loggers are configured using a YAML file that specifies the following.
1. enabled/disabled
2. whether the message should be logged on console.
3. filename extension.
4. minimum log level.

Retrofitted source code to now use the updated logging library.
Updated the documentation with information regarding the specification
of the log config file.

Currently, the log format in the config file is not adhered to. This is going to be
addressed in a future commit.
2019-12-09 20:15:33 -05:00
PRADYUMNA KAUSHIK
e3caa90c31
Refactor imports, Add vendor/ as submodule (#5)
refact imports to github.xxx. update go.mod + go.sum

* removed vendor/ folder

* Added vendor/ submodule.

Moved dependencies that need to be vendored into separate repository.
	- https://github.com/spdfg/elektron-vendor
Added vendor/ as a submodule using the below command.
	- git submodule add https://github.com/spdfg/elektron-vendor vendor
If wanting to use vendor, run the following commands after cloning
elektron.
1. git submodule init
2. git submodule update

* added instructions to clone vendor/ submodule.

* updated module to spdfg. Refactored imports
2019-10-31 14:32:46 -04:00
Pradyumna Kaushik
91e06fab4f Unit testing for pcp module.
Added unit tests to test code in def/ module.
Currently, AverageNodePowerHistory(...) assumes that the DRAM power
readings are being monitored. In addition, it also assumes that the
host servers are dual socket machines. However, this assumption might
not always hold true. Thus, added a TODO to fix this.
2019-10-24 20:07:17 +00:00
Renan DelValle
291f4f3979
Adding GPLv3 Licensing. 2018-10-06 20:03:14 -07:00
Pradyumna Kaushik
bcc2b4e4cc Removed Degree of Collocation logger. 2018-10-05 14:11:49 -04:00
Pradyumna Kaushik
aca9dcd185 Added degcol logger to extrema and prog-extrema.
Degree of collocation logger (or task share variance logger) was
only present in pcp.go. Copied that code into extrema and prog-extrema.
Also, exposed visibility of pcp/utils.go#cpuUtilsPerNode(...) and
pcp/utils.go#memUtilsPerNode(...) to public.
2018-10-05 00:38:24 -04:00
Pradyumna Kaushik
c9d4e66236 Powercap policy and PCP config from commandline.
Added command line arguments corresponding to powercapping policy
and pcp config file path.
2018-10-04 19:21:45 -04:00
Pradyumna Kaushik
410a6ef1fa Added pcp config file to Command args. 2018-10-04 14:10:41 -04:00
Pradyumna Kaushik
1e9828c35b Provide pcp config file from commandline.
Taking pcp config file name (or path) from commandline.
In scheduler.go, also checking to see if provided pcp config file
exists. If not, then exitting.
2018-10-04 13:57:35 -04:00
Pradyumna Kaushik
8e87bcb439 Fixed import alias prefix.
Changed the prefix to import aliases to be 'elek' instead of 'elec'.
2018-10-04 13:45:31 -04:00
Renan DelValle
9554bf2666
Runing goimports on project as well as fixing README so that headers render correctly. 2018-09-30 18:23:38 -07:00
Renan DelValle
45f9efa578
Changing imports from bitbucket to gitlab. 2018-09-30 17:56:14 -07:00
Pradyumna Kaushik
5a28f8539a formatted code. 2018-09-23 18:23:37 -04:00
Akash Kothawale
acf9332a50 Comments: Explain what Variance is used for. 2018-09-23 18:16:27 -04:00
Akash Kothawale
2d3158d99a pcp: Remove unused functions 2018-09-23 18:16:17 -04:00
Akash Kothawale
c52bfb08c5 Handle case: host -> slaveID may not be available 2018-09-23 18:16:06 -04:00
Akash Kothawale
9c09321d81 Fix incorrect degColLog file location 2018-09-23 18:14:48 -04:00
Akash Kothawale
39d1cf980c Use existing Mean and Variance functions
Available in github.com/montanaflynn/stats
2018-09-23 18:13:47 -04:00
Akash Kothawale
46be28ef8d pcp: CPU/MEM utilization & task share variance per node 2018-09-23 18:13:28 -04:00
Pradyumna Kaushik
065705d480 Merged in experimentation/schedPolicySwitcher (pull request #1)
Experimentation/schedPolicySwitcher
1. Initial commit for consolidated loggers using observer pattern.
2. class factory for schedulers.
3. Using the scheduling policy class factory in schedulers/store.go and the scheduler builder helpers in schedulers/helpers.go, feature to be able to be able to plug a scheduling policy of your choice from the command line (right now only first-fit and bin-packing are possible. Will be updating the class factory to include other scheduling policies as well.
4. Removed TODO for using generic task sorters. Modified TODO for a config file input to run electron.
5. Added other schedulers to the factory
6. Partially retrofitted the other scheduling policies to use the logging library.
7. Retrofitted extrema and progressive to use the consolidated logging library. Fixed parameter issue with s.base.Disconnected(). Formatted project
8. Move statusUpdate(...) into base.go to remove redundant code.
9. Converted the baseScheduler into a state machine where the state is a scheduling policy that defines an approach to consume resource offers.
10. Added another command line argument to be used to enable switching of scheduling policies. Retrofitted scheduling policies to switch only if the particular feature has been enabled.

changed argument to coLocated(...) to take base type rather than ElectronScheduler type. Also, prepended the prefix to the directory of the logs so that it would be easier to determine what the files in a directory correspond to without viewing the contents of the directory.
Defined methods in ElectronScheduler. Each of these methods corresponds to a type of log that an ElectronScheduler would make. Each of these methods would need to be implemented by the scheduling policy.

Electron has only one scheduler that implements the mesos scheduler interface. All the scheduling policies are just different implementations of ways to consume mesos resource offers. Retrofitted scheduling policies to now embed SchedPolicyState instead of baseScheduler.

Approved-by: Pradyumna Kaushik <pkaushi1@binghamton.edu>
2018-09-23 17:19:28 -04:00
Pradyumna Kaushik
b807625b78 Fixed the comments to be capitalized at the start and also terminate with a period. 2017-09-28 15:36:47 -04:00
Pradyumna Kaushik
2cb09ece1f fixed importing issues in power-capping strategies. Changed names of power-capping strategies to extrema.go and progressive-extrema.go 2017-09-26 00:26:01 -04:00
Pradyumna Kaushik
9897c983fe removed proactive-clusterwide power-capper. Retrofitted package names to use elektron and not electron 2017-09-26 00:18:35 -04:00
Abhishek Jain
85b14e125d Got rid of commented code 2017-03-24 17:28:53 -04:00
Abhishek Jain
f85ed944f4 Changed the loop as now we are looping over a map and not a slice. 2017-03-23 22:13:29 -04:00
Pradyumna Kaushik
41206dd82e refactored name of CapThreshold to LowerCapLimit. Added comment to mention that floating point operations can lead to precision loss. 2017-03-09 19:20:13 -05:00
Pradyumna Kaushik
87bd8d7cf0 Added logging when there is no victim left to cap. 2017-03-09 19:17:06 -05:00
Pradyumna Kaushik
2939943afd Made sure that the victim that is capped the most is the one picked to be uncapped. This is to reduce the chances of benchmark starvation. Refined comments and logging. 2017-02-22 20:09:04 -05:00
Pradyumna Kaushik
373ba63933 fixed bugs. Made sure that we don't cap a victim below the threshold. Made sure the victims the can be capped and uncapped are maintained in both cappedVictims and orderCappedVictims. 2017-02-21 21:05:47 -05:00
Pradyumna Kaushik
726c0555ed Fixed corner cases in progressive extrema -- When a node is capped and the new cap value is above a threshold then that node can be capped or uncapped in the next cycle. If the new cap value is equal to the threshold then the node cannot be capped further and can only be uncapped. When the node is uncapped and the newUncapValue is below 100 then the node can be capped or uncapped in the next cycle. If the newUncapValue is 100 then the node can only be capped. 2017-02-20 20:55:06 -05:00
Pradyumna Kaushik
d42b7a3a3b changed the type of percentage in rapl.Cap(...) from int to float64. Retrofitted power-capping strategies to cap using a float64 value instead of an int. Moved common functions in loganddynamiccap.go and logAndProgressiveExtrema.go into pcp/utils.go. New power-capping strategy that builds on top of extrema, where it caps the victims at different until it can't cap further, in which case it starts uncapping them in the reverse order of capping. 2017-02-15 19:22:56 -05:00
Pradyumna Kaushik
a0a3e78041 Merged in hybridSchedulerAndSortedOffers (pull request #8)
Hybrid Scheduler -- TopHeavy, BottomHeavy, FirstFit and BinPacked schedulers with sortedOffers.
2017-02-10 20:21:04 +00:00
Renan DelValle
12efaedaa9 Style and naming fixes. 2017-02-05 14:54:15 -05:00
Renan DelValle
610c006834 Adding DRAM monitoring to Extrema. 2017-02-03 23:12:49 -05:00
Pradyumna Kaushik
0413d2abd3 moved proactiveclusterwidecappers.go to powerCapping/ from pcp/ 2017-01-28 21:42:23 -05:00
Pradyumna Kaushik
2cd77a7ba8 Changed Window to ConsiderationWindow. 2017-01-28 18:29:00 -05:00
Pradyumna Kaushik
8c6ad36b5e changed name of StartLogAndDynamicCap(...) to StartPCPLogAndExtremaDynamicCap(...).] 2017-01-15 15:23:57 -05:00
Pradyumna Kaushik
8c0cde939a renamed Recap to NaiveRecap 2017-01-08 19:13:44 -08:00
Renan DelValle
2678032c2c Go fmt on the entire project 2017-01-03 20:57:25 -05:00
Renan DelValle
b73c30a8bf Implemented a scheduling tracer. Logs time date and host:task-ID. Made log prefix more global since it needs to be constant for all log files generated. 2017-01-03 20:30:21 -05:00
Renan DelValle
147bbc8b5a Fixing commit history 2016-12-22 23:27:18 -05:00
Pradyumna Kaushik
f829cfea66 Moved it to pcp/ as it is not a scheduler and is a capping strategy 2016-12-22 23:20:18 -05:00
Pradyumna Kaushik
79a5dca76b Added another log message to log the name of the file to which the pcplogs are getting written to. 2016-12-22 22:58:59 -05:00
Pradyumna Kaushik
383f088b6a changed bingcloud to sunybingcloud in the import statements. 2016-12-22 22:58:59 -05:00
Renan DelValle
d2fc72885d Go FMT run on entire project 2016-12-22 22:58:59 -05:00
Renan DelValle
bf9a5a6005 Finished integrating dynamic capping policy that uses a high and a low threshold to start capping and uncapping node via an ssh commands that triggers a python script. Commit has sensitive data, scrub before releasing to public. 2016-12-22 22:58:59 -05:00
Renan DelValle
f7e38285a4 rapl capping via ssh command in rapl package 2016-12-22 22:58:59 -05:00
Renan DelValle
fbc05a42df Initial version of command which will cap system 2016-12-22 22:58:59 -05:00
Renan DelValle
7d57957524 Averages are calculated correctly, PCP is able to determine a victim to be capped 2016-12-22 22:58:59 -05:00
Renan DelValle
fce62981da Moved schedulers from the main programs to schedulers package. Can now choose different scheduelrs to use. Work on code sharing between schedulers remains to be done. 2016-12-22 22:58:58 -05:00