Simplifying retry mechanism for Thrift Calls (#56)

* Deleting permament error as it doesn't make sense. Just return a plain old error and that will be considered permanent.

* Removing double closure at as it's unmaintainable and can be error prone. Separated back offs into a generic one and a thrift call specific one.

* ZK leader finder now returns a temporary error instead of constantly no leader found and quitting. It could be that the leader info is being propagated so it's worth trying another time.

* Adding more logging to the retry.

* Wrapping lock and unlock in an anonymous function so that we can use defer on unlock such that it is called in the case of a panic.

errors.go

@@ -16,6 +16,8 @@ package realis
 
 // Using a pattern described by Dave Cheney to differentiate errors
 // https://dave.cheney.net/2016/04/27/dont-just-check-errors-handle-them-gracefully
+
+// Timeout errors are returned when a function has unsuccessfully retried.
 type timeout interface {
 	Timeout() bool
 }
@@ -38,6 +40,7 @@ func NewTimeoutError(err error) *TimeoutErr {
 	return &TimeoutErr{error: err, timeout: true}
 }
 
+// Temporary errors indicate that the action may and should be retried.
 type temporary interface {
 	Temporary() bool
 }
@@ -60,8 +63,3 @@ func (t *TemporaryErr) Temporary() bool {
 func NewTemporaryError(err error) *TemporaryErr {
 	return &TemporaryErr{error: err, temporary: true}
 }
-
-// Nothing can be done about this error
-func NewPermamentError(err error) TemporaryErr {
-	return TemporaryErr{error: err, temporary: false}
-}