A Look at Swift EC Pieces from Bottom-Up

gistfile1.txt

A Look at Swift EC Pieces from Bottup-Up  (December 8, 2016)
============================================================

Document Purpose
================
- Convey understanding of what the pieces are and how they fit together
- NOT intended to teach anything about EC


liberasurecode
==============
- https://github.com/openstack/liberasurecode
- C library that provides a consistent API to various EC backends
- also provides a couple of built-in/native implementations

                                      |----> native liberasurecode
                                      |
                                      |----> Jerasure (http://jerasure.org)
                                      |
                    liberasurecode ----
                    ==============    |
                                      |----> ISA-L (Intel Storage Acceleration Library)
                                      |
                                      |----> SHSS (NTT Lab Japan)


EC Types
--------
jerasure_rs_vand        = 1
jerasure_rs_cauchy      = 2
flat_xor_hd             = 3
isa_l_rs_vand           = 4
shss                    = 5
liberasurecode_rs_vand  = 6
isa_l_rs_cauchy         = 7


Backend Implementations by EC Type
----------------------------------
flat_xor:   native (built-in)
rs_vand:    jerasure, isa-l, native (built-in)
rs_cauchy:  jerasure, isa-l

* Don't know about shss - not open source?


API - base
----------
liberasurecode_instance_create
liberasurecode_instance_destroy
liberasurecode_encode
liberasurecode_encode_cleanup
liberasurecode_decode
liberasurecode_decode_cleanup
liberasurecode_reconstruct_fragment
liberasurecode_fragments_needed


API - fragment checksums
------------------------
liberasurecode_get_fragment_metadata
is_invalid_fragment
liberasurecode_verify_stripe_metadata


Build/Install
-------------
[TODO]: add build/install documentation for liberasurecode


===============================================================================

pyeclib
=======
- https://github.com/openstack/pyeclib
- python bindings for liberasurecode (no more, no less)

                                      |----> native liberasurecode
                                      |
                                      |----> Jerasure (http://jerasure.org)
                                      |
         pyeclib -> liberasurecode ----
         =======                      |
                                      |----> ISA-L (Intel Storage Acceleration Library)
                                      |
                                      |----> SHSS (NTT Lab Japan)


Relationship of pyeclib and liberasurecode
------------------------------------------
- currently each is in a separate github repo
- same set of folks maintain both
  -- Kota, Thiago, Tushar, Kevin
- currently some discussion (#openstack-swift) of putting pyeclib into liberasurecode repo
- same set of defined EC types


bindep.txt
----------
liberasurecode-dev [platform:dpkg]
liberasurecode-devel [platform:rpm]


Build/Install
-------------
[TODO]: add build/install documentation for pyeclib


===============================================================================

Swift
=====

Overview
--------
- EC supported in Swift
- A cluster need not use EC
- 'storage policy' mechanism allows for data storage details
- 'erasure_coding' (EC) is a type of storage policy (other type is 'replication')


                                      |----> native liberasurecode
                                      |
                                      |----> Jerasure (http://jerasure.org)
                                      |
swift -> pyeclib -> liberasurecode ----
=====                                 |
                                      |----> ISA-L (Intel Storage Acceleration Library)
                                      |
                                      |----> SHSS (NTT Lab Japan)


requirements.txt (transitive dependency on liberasurecode)
----------------
PyECLib>=1.3.1


pyeclib calls
-------------
Construction:  ECDriver(k=self._ec_ndata, m=self._ec_nparity, ec_type=self._ec_type)
encode
decode
get_segment_info
get_metadata
reconstruct

Configure EC Components
-----------------------
[TODO]: add build/install documentation for swift EC components


===============================================================================

Swift EC Touchpoints / API Calls
================================

proxy/controllers/obj.py
------------------------
from swift.common.storage_policy import (POLICIES, REPL_POLICY, EC_POLICY,
                                         ECDriverError, PolicyError)

pyeclib_driver.decode(fragments)
pyeclib_driver.encode(chunk_to_encode)
pyeclib_driver.get_segment_info(last_segment_size, policy.ec_segment_size)


obj/diskfile.py
---------------
from pyeclib.ec_iface import ECDriverError, ECInvalidFragmentMetadata, \
    ECBadFragmentChecksum, ECInvalidParameter
from swift.common.storage_policy import (
    get_policy_string, split_policy_string, PolicyError, POLICIES,
    REPL_POLICY, EC_POLICY)

pyeclib_driver.get_metadata(frag)

NOTE: also some EC-specific derived classes


obj/reconstructor.py
--------------------
from swift.common.storage_policy import POLICIES, EC_POLICY

pyeclib_driver.reconstruct(fragment_payload,[frag_index])[0]


common/storage_policy.py
------------------------
from pyeclib.ec_iface import ECDriver, ECDriverError, VALID_EC_TYPES
DEFAULT_EC_OBJECT_SEGMENT_SIZE = 1048576

# Initialize PyECLib EC backend
ECDriver(k=self._ec_ndata, m=self._ec_nparity, ec_type=self._ec_type)