EC optimizations pools do not update every shard on every I/O. The primary
must have a complete log and requires objects to have up to date object
attributes, so the choice of primary has to be restricted. Shards that
cannot become a primary are listed in the nonprimary_shards set.

For a K+M EC pool with optimizations enabled the 1st data shard and all
M coding parity shards are always updated and can become a primary, the
other shards will be marked as nonprimary.

The new set nonprimary_shards stores shards that cannot become the primary,
by default it is an empty set which retains existing behavior. When
optimisations are enabled on an EC pool this set will be filled in to
restrict the choice of primary.

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

Add partial_writes_last_complete map to pg_info_t and pg_fast_info_t.
For optimized EC pools not all shards receive every log entry. As
log entries are marked completed the partial writeis last complete
map is updated to track shards that did not receive the log entry.

Each map entry stores an eversion range. The first version is the last
completion the shard participated in, the second version tracks subsequent
updates where the shard was not updated. For example the range 88'10-88'12
means a shard completed update 10 and that updates 11 and 12 intentionally
did not update the shard. This information is used during peering to
distinguish a shard that is missing updates from a shard that intentionally
did not participate in an update to work out what recovery is required.

By default this map is empty indicating that every shard is expected to
participate in an update and have a copy of the log entry.

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

…nty_t

Add two new sets to the pg_log_entry for use by EC optimization pools.
The written shards set tracks which shards were written to, the
present shards set tracks which shards were in the acting set at the
time of the write.

An empty set (default) is used to indicate all shards. For pools without
allow_ec_optimizations the written set is empty (indicating all shards are
written) and the present set is empty and unused.

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>

EC optimized pools do not always update every shard for every write I/O,
this includes not updating the object_info_t (OI attribute). This means
different shards can have OI indicaiting the object is at different
versions. When an I/O updates a subset of the shards, the OI for the
updated shards will record the old version number for the unmodified
shards in the shard_versions map. The latest OI therefore has a record
of the expected version number for all the shards which can be used to
work out what needs to be backfilled.

An empty shard_versions map imples that the OI attribute should be the
same on all shards.

Signed-off-by: Bill Scales <bill_scales@uk.ibm.com>