Skip to content

Instantly share code, notes, and snippets.

@gWOLF3

gWOLF3/config.md

Forked from sykesm/config.md
Created October 11, 2019 19:40
Show Gist options
  • Save gWOLF3/08e6aaae7e34bdcbba4e4d67fa451c77 to your computer and use it in GitHub Desktop.
Save gWOLF3/08e6aaae7e34bdcbba4e4d67fa451c77 to your computer and use it in GitHub Desktop.
Download ZIP
Config Discussion
Raw
config.md

Fabric Configuration

Fabric configuration is currently implemented around a configuration library called viper. Viper reads configuration from files, environment variables, and flags, and exposes an API that uses dot qualified keys to reference the configuration values (think System Properties on steroids).

When configuration is read from files, the segments of the configuration key are used to walk config file stanzas to the data. Values read from the configuration file can be overridden by setting an environment variable that maps to the configuration key. Config values can also be sourced from flags. Flags take precedence over environment variables and values source from files.

General Issues with Fabric Configuration

Most of the issues we have with our configuration aren't problems with viper as much as how we use it.

Viper Everywhere

Viper provides a function based API that makes it easy for Fabric components to retrieve configuration values. Unfortunately, the API is so easy to use that it has proliferated throughout the code base like a virus. What was initially a light touch, easy to use pattern has become a core component with several issues:

  • Config is a global singleton that impacts concurrent testing of multiple config values
  • Easy access to global configuration resulted in code that could not be explicitly configured without using the viper API
  • Creation of "utility" layers on top of viper resulted in multiple entry points to the configuration data
  • Config information is spread throughout the system without good documentation or a single source for the config schema

Default Configuration

While viper enables programs to set default configuration values by calling SetDefault, Fabric has chosen to use "sample configuration" documents as configuration defaults. This results in poor configuration defaults and less than ideal sample config documents.

In many cases, default values were chosen based on assumptions in test cases rather than utility in production scenarios.

Inconsistencies

Even though the orderer and the peer use viper for configuration, are documented together, and live in the same source tree, they use similar, but different patterns to obtain configuration information.

For example, core.yaml uses camelCase for configuration keys while the orderer's orderer.yaml uses PascalCase. While the difference isn't particularly significant1, the arbitrary case difference makes it harder to template shared configuration values between the two.

In TLS configuration, the orderer allows users to specify a PEM encoded certificate block or a path to a file. While this is very flexible, using "magic" instead of separate keys requires non-standard config processing and error handling.

Fabric-CA also uses viper but, instead of using camelCase or UpperCamelCase, it uses lowercase for configuration keys. It's default configuration is also handled differently. If a config file does not exist, it will write one to the working directory for future customization.

There are also places where the names of required configuration values are different for no really good reason. An example of this is the MSP configuration path. In the peer it's called mspConfigPath while in the orderer it's called LocalMSPDir.

Reflect Based Extensions

The orderer configuration attempted to avoid some of the viper proliferation problems that the peer suffers from. As part of that, a new viperutil package with an EnhancedExactUnmarshal function was created. This package uses a combination of reflection, viper, and mapstructure to discover viper configuration keys.

Unfortunately, the road to hell is paved with good intentions. In addition to violating the "clear is better than clever" proverb, the config parsing implementation of the orderer ended up relying on case-preserving behavior in viper that was deemed to be a bug. This bug-as-feature behavior pinned us to an ancient version of viper.

Environment Variable Overrides

Historically, fabric has primarily been distributed as docker images. Since these images only contain the sample, default configuration values, environment variables were used to override the defaults. This is a perfectly reasonable mechanism, however, these overrides quickly got out of hand. Take, for example, a command we document in our "bring your first network" sample:

CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/users/[email protected]/msp
CORE_PEER_ADDRESS=peer0.org1.example.com:7051
CORE_PEER_LOCALMSPID="Org1MSP"
CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt

In addition to the odd gopath references, the reliance on explicit, repeated, configuration with differing values while working across peers is error prone and confusing - especially when using command line tools.

Our configuration model and viper further complicate things when mapping configuration values to environment variables. For example, the list of client root CA certificates for the operations service looks like this in config documents:

operations:
  tls:
    # paths to PEM encoded ca certificates to trust for client authentication
    clientRootCAs:
      files: []

The variable associated with this config is prefix_OPERATIONS_TLS_CLIENTROOTCAS_FILES. Since the key is associated with a list, how should the list be encoded? Well, in the peer, it looks like this:

CORE_OPERATIONS_TLS_CLIENTROOTCAS_FILES: '/certs/tls/cacerts/cacert.pem /certs/msp/operationscerts/operationscert-1.pem'

and in the orderer it looks like this:

ORDERER_OPERATIONS_TLS_CLIENTROOTCAS: '[/certs/tls/cacerts/cacert.pem,/certs/msp/operationscerts/operationscert-1.pem]'

Ignoring the slight variation in environment key name, the encoding of the values are quite different. The peer requires a space separated list of files while the orderer requires a comma separated list of values enclosed in square brackets.

Another problem area is where we use variable values as keys in a map. The peer system chaincode element is a good example:

    # system chaincodes whitelist. To add system chaincode "myscc" to the
    # whitelist, add "myscc: enable" to the list below, and register in
    # chaincode/importsysccs.go
    system:
        _lifecycle: enable
        cscc: enable
        lscc: enable
        escc: enable
        vscc: enable
        qscc: enable

When the keys are variable, it may not be possible to map it to a valid environment variable. This was highlighted when we called the new lifecycle chaincode +lifecycle because + is not part of the POSIX portable character set definition for environment variables.

Goals

These are the goals of the configuration work:

  • Provide a consistent, YAML serialization of configuration for the peer, orderer, and CA.
    • Consistent use of camelCase (or snake_case1) for configuration keys.
    • Identical configuration schema for TLS and MSP for all Fabric programs.
  • Remove automatic mapping of environment variable overrides for config elements and move to explicitly named overrides where appropriate.
  • Provide a simple mechanism to reference environment variables as values in the configuration files.
  • Enable a portable configuration and runtime tree
    • Use the current working directory as the default configuration directory.
    • Ensure all file references within configuration are relative to the configuration document.
  • Enable automated generation of documentation and default configuration documents directly from code.
  • Rename core.yaml to peer.yaml

Stretch Goals

  • Extract peer subcommands to a new CLI (called fabric or fabcli) and associated with a correspondingly named file for configuration. Config and data for this tool should be sourced from XDG compliant locations by default. (e.g. $HOME/.config/fabcli.yaml)
  • Implement an alternative local MSP that is sourced from a single YAML document instead of a configuration tree.
  • Stop using mapstructure when decoding config
  • Remove viper from Fabric

Out of Scope

These are explicitly out of scope:

  • Dynamic reloading of configuration
  • Updates to the channel config transaction tooling
  • General UX improvements to the command line

Rules of the Road

YAML Configuration Documents

We continue to use YAML for our configuration documents. In addition to allowing JSON elements, it supports comments and multi-line block values.

Default Config Location

Long running processes will no longer use /etc/hyperledger/fabric as the default config location; instead, the current working directory will be used. An explicit configuration directory can still be specified by setting FABRIC_CFG_PATH in the environment.

Command line flags should be provided to specify a specific configuration file. The flag will override any default location or environment variable value.

Relative Paths are Context Sensitive

When relative paths are used in a configuration file, the paths are to be interpreted relative to the directory containing the configuration file.

When relative paths are used on command line flags, the paths are to be interpreted relative to the working directory.

Consistent Naming Convention

Configuration keys and command line flags will use a consistent naming convention. The choice of convention matters much less than ensuring it's used consistently.

The leading candidates for config file keys are camelCase, snake_case, and PascalCase.

The leading candidates for long command line flags are nodelimeter, hyphen-delimited, and camelCase.

Config Keys are Constants

When mapping a set of configuraiton values to some entity, avoid patterns where an input value is treated as a key:

  mapping:
    1.2.3.4: 5.6.7.8
    9.8.7.6: 4.3.2.1

Instead, use lists of objects with explicit field names:

  mappings:
  - from: 1.2.3.4
    to: 5.6.7.8
  - from 9.8.7.6
    to: 4.3.2.1

The latter form promotes type safety and extensibility.

The major exception to this rule is when the config model is carrying generic or opaque configuration elements. In these cases, an opaque map[string]interface{} is appropriate.

Certificates and Keys

Certificate and key values in configuration will be PEM encoded blocks. If certificate chains are used, a multi-block value should be used. The blocks must be concatenated such that each certificate certifies the preceding it; the root CA shall be the last certificate in the list.

Separate keys must be used for certificates and key values and for files that contain certificate and key values. For example:

  tls:
    cert: |
      -----BEGIN CERTIFICATE-----
      Base64–encoded certificate
      -----END CERTIFICATE-----
    certfile: ~
    key: ~
    keyfile: tls/private.key

Notice cert and certfile are elements of the tls configuration. When an inline certificate is used, it should be the value associated with cert; when a file reference is used, the path should be associated with certfile.

It is an error to provide values for inline and file references for the same configuration element. In the example above, certfile is explicitly nil.

Certificate pools should always support multiple PEM encoded blocks.

Referencing Environment Variables

Instead of exposing all configuration as environment variables, we will allow configuration values to be sourced from environment variables by using a value of ${env.ENVIRONMENT_NAME} as a value in the configuration file.

  id: ${env.MY_ENV_VAR}

The ${env.NAME} format is will not interpreted if used within a value. For example, whem MESSAGE_ENV_VAR is set to "msg":

  msg: ${env.MESSAGE_ENV_VAR}
  message: my message is ${env.MESSAGE_ENV_VAR}.

will result in message="my message is ${env.MESSAGE_ENV_VAR}" and msg="msg".

Referencing environment variables is only supported for the basic types used in leaf nodes of the configuration.

Durations

Durations must always include units compatible with go's time.ParseDuration function. A value without a suffix is expressed in nanoseconds and is rarely appropriate.

This means, by extension, that configuration keys should not contain units for the duration. (e.g. timeoutInSeconds should not be used)

Structure Defines Schema

To support automatic generation of documentation, configuration will be expressed as a set of structures in code.

A config package will be provided decode configuration documents. The package will expose types and functions to resolve relative paths and apply configuration defaults to values omitted from the config.

The config package will also expose functions to write example configuration documents with comments. Field level godoc will be used to document the config keys and tags will be used to provide the default value and, when required, the name of the environment variable that can be used to override what is read from configuration.

// MyConfig is my configuration.
type MyConfig struct {
    // ID provides a unique identifier.
    ID string `yaml:"id" example:"example-id" env:"MY_CONFIG_ID"`

    // Timeout is the maximum time the service will wait for a response.
    Timeout time.Duration `yaml:"timeout" default:"30s"`

    // Subsystem configures the subsystem.
    Subsystem *SubsystemConfig `yaml:"subsystem"`
}

// SubystemConfig is the configuration structure for subsystem.
type SubsystemConfig struct {
    // Timeout is the maximum amount of time the subysstem will wait for a
    // response.
    Timeout time.Duration `yaml:"timeout" default:"10s"`

    // WorkingStoragePath points to the directory to store temporary data.
    WorkingStoragePath config.Path `yaml:"workingStoragePath" default:"subsys"`
}

The generated documentation will replace occurrences of the struct field name with the appropriate YAML config element name.

The default configuration for MyConfig would look like this:

---
# id provides a unique identifier
id: example-id

# timeout is the maximum time the service will wait for a response.
timeout: 30s

# subsystem configures the subsystem
subsystem:
  # timeout is the maximum amount of time the subsystem will wait for a
  # response.
  timeout: 10s

  # workingStoragePath points the directory to store temporary data.
  workingStoragePath: subsys

Footnotes

  1. In the case of the orderer, yaml tags were not specified so the case doesn't matter when decoding but does matter when using the viper API. ↩

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment