
Design of the CounterService
============================

Author: Bela Ban, Sept 2011


The CounterService is a cluster wide atomic counter. It talks to a protocol (COUNTER) via events, similar to how the
LockService works. Operations are get, set, increment, decrement and compare-and-set, see
org.jgroups.blocks.atomic.Counter for details.

Counters are named, and a client always sends requests to a named counter. This allows for different counters to exist
at the same time. Counters can be created with an initial value and can also be deleted again.

Operations are always sent to the coordinator which applies the change atomically and returns a value to the client.
Each operation also increments a version associated with a counter.

When the coordinator leaves or fails, the new coordinator runs a reconciliation round, which gathers value and version
pairs for each named counter (also in a merge case). It creates a (server side) counter from all responses,
based on the highest version number.

(Note: if we only provide a unique ID generator, then we can set the new counter to be the max of all
received values, and don't need the version).

After the new coordinator has received and incorporated all current values and versions, it accepts new requests and
sends a message to all members to resend their pending requests, which are requests that were sent,
but never got a response.

We may maintain a number of backup coordinators, to which the coordinator sends counter changes, to prevent the case
where the coordinator and client update a value, then both leave and the cluster doesn't know about it and hands out
the same ID again.

The goal of COUNTER is to handle regular requests in 1 round-trip. It will be a bit more costly to handle coordinator
crashes.


Details
=======

- Server and client are roles, a cluster member can be both (in the case of a coordinator, for instance)

- Server state:
  - Server Counters: map of names and counters, their values and versions

- Client state:
  - Client counters: map of counters, their values and versions
  - Pending requests: map of owners (thread-IDs) and requests, e.g. "owner-1": incr("x")



On getting a request from the CounterService (via COUNTER.down() event):
------------------------------------------------------------------------
- Add the request to the pending requests table
- Send the request to the coordinator
- Block on the newly created entry in the pending requests table



On getting a client request:
----------------------------
- Discard if we're not the coordinator
- Find the named counter
- Update the value and increment the version
- Send an update request (incl. value and version) to the backup(s). This can be done synchronously
  or asynchronously (configurable).
- Send the response to the client including the value and version



On getting a response from the coordinator:
-------------------------------------------
- Find the entry corresponding to the owner in the pending requests table, set the value and unblock the thread
  - This makes the caller of the request return with the value
- Remove the entry from the pending requests map



On getting an update request:
-----------------------------
- Update the server counters map with the new value and version



On getting a view change (including a merge view):
--------------------------------------------------
- If we're the new coordinator and *don't* have the server counters map populated:
  - Block all requests
  - Run the reconciliation phase
  - Unblock the requests


Reconciliation phase:
---------------------
- Send a reconciliation request to all members to return all of their counters (names, values and versions)
- Wait for all responses or a timeout
- For each named counter:
  - Create a counter in the server counters map if it doesn't exist yet
  - Set the value to the highest version number received (or the highest value if we don't use versions)
    (If the named counter already exists, but the version of the response is higher -> update the value and version.
     Else, leave it)


On reception of a reconciliation request from P:
------------------------------------------------
- For all counterd in the client counters map:
  - Add the named counter, its value and version to a list
- Send the list with the response back to the new coordinator P



Issues:
-------
- Int or long counters? Do we let the users choose?


