From 9fd33a265fa403e0890ff28448001558fe287827 Mon Sep 17 00:00:00 2001 From: "Mark T. B. Carroll" Date: Mon, 7 Aug 2023 13:19:18 +0100 Subject: [PATCH 1/3] Mention Raft less in v1.2 documentation, warn of removal. (anticipates Raft's removal) Signed-off-by: Mark T. B. Carroll --- docs/1.2/glossary.md | 7 ++- docs/1.2/index.md | 4 -- .../sysadmin_guide/about_dynamic_consensus.md | 58 ++----------------- 3 files changed, 10 insertions(+), 59 deletions(-) diff --git a/docs/1.2/glossary.md b/docs/1.2/glossary.md index 6125e4c5..a00c6697 100644 --- a/docs/1.2/glossary.md +++ b/docs/1.2/glossary.md @@ -177,9 +177,10 @@ Raft consensus with a restricted membership. Raft is crash fault tolerant, not Byzantine fault tolerant, and has finality (does not fork). For more information, see [Raft (computer science) on - Wikipedia](https://en.wikipedia.org/wiki/Raft_(computer_science)) - and the [Sawtooth Raft - documentation](https://sawtooth.hyperledger.org/docs/raft/nightly/master/introduction.html). + Wikipedia](https://en.wikipedia.org/wiki/Raft_(computer_science)). + **Sawtooth Raft is due for removal** because the prototype was never ready + for production, and applications requiring finality can use PBFT consensus + instead. REST API diff --git a/docs/1.2/index.md b/docs/1.2/index.md index 4a4441ab..48ab35f4 100644 --- a/docs/1.2/index.md +++ b/docs/1.2/index.md @@ -181,10 +181,6 @@ for these algorithms: > simulator is also called \"PoET/CFT\" because it is crash > fault tolerant, not Byzantine fault tolerant. > -> - [Sawtooth Raft]({% link docs/1.2/raft/index.md %}) is a -> leader-based consensus algorithm that provides crash fault -> tolerance for a small network with restricted membership. -> > - Devmode (short for \"developer mode\") is a simplified > random-leader algorithm that is useful for developing and testing > a transaction processor. Devmode is not recommended for multi-node diff --git a/docs/1.2/sysadmin_guide/about_dynamic_consensus.md b/docs/1.2/sysadmin_guide/about_dynamic_consensus.md index d8fedb79..889127fa 100644 --- a/docs/1.2/sysadmin_guide/about_dynamic_consensus.md +++ b/docs/1.2/sysadmin_guide/about_dynamic_consensus.md @@ -234,59 +234,13 @@ Requirements: ## Raft Consensus {#raft-consensus-label} -[Sawtooth Raft]({% link docs/1.2/glossary.md %}#term-sawtooth-raft) provides a -simple consensus algorithm that is easy to understand. It is leader-based, crash -fault tolerant, and does not fork. For more information, see the [Raft -documentation]({% link docs/1.2/raft/index.md %}). +Sawtooth Raft was intended to provide a simple consensus algorithm that is +easy to understand. Raft is leader-based, crash fault tolerant, and does not +fork. -> **Note** -> -> Sawtooth Raft is currently a prototype (not yet released). For more -> information, see the -> [sawtooth-raft](https://github.com/hyperledger/sawtooth-raft) -> repository. - -Requirements: - -- Each node must install the Raft consensus engine package, - `sawtooth-raft-engine`. - -- Each node must run the Raft consensus engine: - - - Service: `sawtooth-raft-engine.service` - - Executable: `raft-engine` - - For more information, see - [Running Sawtooth as a Service]({% link - docs/1.2/sysadmin_guide/setting_up_sawtooth_network.md - %}#running-sawtooth-as-a-service). (To start the consensus engine on the - command line, see [Step 5. Start Sawtooth on the First Node]({% link - docs/1.2/app_developers_guide/ubuntu_test_network.md %})). - -- Specify static peering when starting each validator. Use the - `--peering` option when starting the validator (see [sawtooth-validator]({% - link docs/1.2/cli/sawtooth-validator.md %})) or set the off-chain `peers` - setting in the `validator.toml` configuration file (see - [About Sawtooth Configuration Files]({% link - docs/1.2/sysadmin_guide/configuring_sawtooth.md %})). - -- Use these on-chain settings for Raft consensus: - - ``` none - sawtooth.consensus.algorithm.name=raft - sawtooth.consensus.algorithm.version=[VERSION] - sawtooth.consensus.raft.peers=[VAL1KEY,VAL2KEY,...,VALnKEY] - ``` - - For the version number, see the `Cargo.toml` file. Use only the - first two digits (for example, `[0.1]`). - - For `VALxKEY`, specify the validator public key of each node in the - network. - -- For other on-chain settings for Raft, see [Optional - Settings]({% link docs/1.2/raft/configuring_deploying.md %}) - in the Raft documentation. +**Sawtooth Raft is due for removal** because the prototype was never ready +for production, and applications requiring finality can use PBFT consensus +instead. ## Devmode Consensus {#devmode-consensus-label} From 24cd0e6415190e48484975fbefe20c61818c11e4 Mon Sep 17 00:00:00 2001 From: "Mark T. B. Carroll" Date: Mon, 7 Aug 2023 13:20:59 +0100 Subject: [PATCH 2/3] Mention Raft less in FAQ. (anticipates Raft's removal) Signed-off-by: Mark T. B. Carroll --- faq/consensus.md | 36 ++++++++++++------------------------ faq/glossary.md | 8 +------- faq/settings.md | 30 +++--------------------------- faq/validator.md | 10 +--------- 4 files changed, 17 insertions(+), 67 deletions(-) diff --git a/faq/consensus.md b/faq/consensus.md index dca0cd40..136cea75 100644 --- a/faq/consensus.md +++ b/faq/consensus.md @@ -36,30 +36,16 @@ PoET SGX (name \"PoET\", version 0.1) (a processor supporting Intel SGX). Currently supported in Sawtooth 1.0 only. -Raft (name \"raft\", version 0.1) - -: Consensus algorithm that elects a leader for a term of arbitrary - time. Leader replaced if it times-out. Raft is faster than PoET, but - is CFT, not BFT. Also Raft does not fork. For Sawtooth Raft is new - and still being stabilized. - ## Will Sawtooth support more consensus algorithms in the future? Yes. With pluggable consensus, the idea is to have a meaningful set of -consensus algorithms so the \"best fit\" can be applied to an -application\'s use case. Raft is a recent addition\--still being -stabilized. Others are being planned. +consensus algorithms so the "best fit" can be applied to an +application's use case. Someday, the previous Raft prototype may be completed +or replaced. Others algorithms are being planned. REMME.io has independently implemented Algorand Byzantine Agreement on Sawtooth. -## Where is Raft documented? - - To use, -basically set `sawtooth.consensus.algorithm` to `raft` and -`sawtooth.consensus.raft.peers` to a list of peer nodes (network public -keys). - ## Does the PBFT implementation follow the original paper? Yes, it follows the original 1999 Castro and Liskov paper with some @@ -238,14 +224,17 @@ running, and this message was quite common. The initial default consensus algorithm is `devmode`, which is not for production use. -Here is an example that changes the consensus to Raft: +Here is an example that changes the consensus to PBFT: ``` sawset proposal create \ ---url http://localhost:8008 --key /etc/sawtooth/keys/validator.priv \ - sawtooth.consensus.algorithm=raft sawtooth.consensus.raft.peers=\ +--url http://localhost:8008 --key /etc/sawtooth/keys/validator.priv \ + sawtooth.consensus.algorithm=pbft \ + sawtooth.consensus.algorithm.version=1.0 \ + sawtooth.consensus.pbft.members=\ '["0276f8fed116837eb7646f800e2dad6d13ad707055923e49df08f47a963547b631", \ - "035d8d519a200cdb8085c62d6fb9f2678cf71cbde738101d61c4c8c2e9f2919aa"]'` + "c4e026c55b62abce8a38c431203cae2c0fcce259ddab5b5ddc6d5242b379afc270", \ + "035d8d519a200cdb8085c62d6fb9f2678cf71cbde738101d61c4c8c2e9f2919aa3"]'` ``` ## How do I change the consensus algorithm for a network that has forked? @@ -293,9 +282,8 @@ production, have 2 nodes per identity. ## Can PoET be configured for small networks? Yes, for development purposes. For production purposes, consider using -another consensus algorithm. For example, Raft or PBFT handles a small -number of nodes nicely. We recommend PBFT for small networks. Raft is -less interesting being CFT and not BFT, and having overall less testing. +another consensus algorithm. For example, we recommend PBFT for small +networks of a few nodes. For PoET in a small blockchain network, disable defense-in-depth tests for small test networks (say, \< \~12 nodes) with: diff --git a/faq/glossary.md b/faq/glossary.md index 2568f094..8123621f 100644 --- a/faq/glossary.md +++ b/faq/glossary.md @@ -111,7 +111,7 @@ Context Classical Consensus : Uses an agreement or voting mechanism to select a leader (vs. - Nakamoto-style consensus) E.g., PBFT and Raft + Nakamoto-style consensus), e.g., PBFT Crypto @@ -407,12 +407,6 @@ r : Rate, measurement of performance in transactions per second -Raft - -: Consensus algorithm that elects a leader for a term of arbitrary - time. Leader replaced if it times-out. Raft is faster than PoET, but - is not BFT (Raft is CFT). Also Raft does not fork. - Remix : A popular web-based IDE for Solidity diff --git a/faq/settings.md b/faq/settings.md index 8fa5ed52..c44ed46a 100644 --- a/faq/settings.md +++ b/faq/settings.md @@ -44,13 +44,12 @@ sawtooth.config.authorization_type sawtooth.consensus.algorithm.name : Pluggable consensus algorithm name. These include `PoET`, `Devmode`, - `raft`, and `pbft`. The default is `devmode` for Sawtooth 1.1 or - earlier, with no default for the (unreleased) nightly build. + and `pbft`. The default is `devmode` for Sawtooth 1.1 or earlier, with no + default for the (unreleased) nightly build. sawtooth.consensus.algorithm.version -: Consensus algorithm version. Currently 0.1 for PoET, Devmode, PBFT, - and Raft. +: Consensus algorithm version. Currently 0.1 for PoET, Devmode, and PBFT. sawtooth.consensus.block_validation_rules @@ -106,29 +105,6 @@ sawtooth.consensus.pbft.forced_view_change_interval : How many blocks to commit before forcing a view change for fairness Optional, default 100 blocks -sawtooth.consensus.raft.election_tick - -: RAFT consensus election tick, in seconds. E.g., 1500 - -sawtooth.consensus.raft.heartbeat_tick - -: RAFT consensus heartbeat tick, in seconds. E.g., 150 - -sawtooth.consensus.raft.peers - -: JSON list of each peer node\'s public key. Only required RAFT - setting. Key is from `/etc/sawtooth/keys/validator.pub` . Example: - ```json - ["0276f8fed116837eb7646f800e2dad6d13ad707055923e49df08f47a963547b631",\ - "035d8d519a200cdb8085c62d6fb9f2678cf71cbde738101d61c4c8c2e9f2919aa3"] - ``` - -sawtooth.consensus.raft.period - -: RAFT consensus period, in seconds. E.g., 3. Higher settings cause - larger blocks, small settings have faster performance with smaller, - quicker block publication, but causes more network traffic. - sawtooth.consensus.valid_block_publishers : List of public keys for allowed block publishers. For devmode diff --git a/faq/validator.md b/faq/validator.md index 860f418f..fd4e4379 100644 --- a/faq/validator.md +++ b/faq/validator.md @@ -135,11 +135,6 @@ specified by at least one other node (and preferably multiple nodes). : contains policy settings, if any -`/var/lib/sawtooth-raft/` - -: Optional Raft consensus-specific files. Present only if you use - Raft. Directory can be changed with `$SAWTOOTH_RAFT_HOME` - If set these files are placed under directory `$SAWTOOTH_HOME` (except files under your home directory, `~` ). Detailed configuration information and examples for Sawtooth directories is at @@ -198,7 +193,7 @@ peers. firewall configuration to communicate with peer validators. Uses ZMQ. -- 5050 is used by the consensus engine (such as PoET or Raft). This +- 5050 is used by the consensus engine (such as PBFT or PoET). This port should be closed to external hosts in a firewall configuration. Uses ZMQ. @@ -311,9 +306,6 @@ to be scheduled irrespective of the failed transaction. language and therefore slower. It also suffers from the Global Interpreter Lock (GIL), which locks executing multiple threads to one thread at-a-time -- When fully stabilized, substitute PoET consensus with Raft - consensus. Raft is CFT instead of BFT, but it should perform better - in exchange for lower fault tolerance - As you make changes, measure the impact with a performance tool such as Hyperledger Caliper From ffc01751cdfe525e9984dc032f3bcecd1b371959 Mon Sep 17 00:00:00 2001 From: "Mark T. B. Carroll" Date: Mon, 7 Aug 2023 13:23:09 +0100 Subject: [PATCH 3/3] Remove Raft documentation. These pages are not currently deployed on a website. Signed-off-by: Mark T. B. Carroll --- docs/raft/0.1/configuring_deploying.rst | 404 ------------------------ docs/raft/0.1/introduction.rst | 68 ---- docs/raft/0.1/troubleshooting.rst | 120 ------- 3 files changed, 592 deletions(-) delete mode 100644 docs/raft/0.1/configuring_deploying.rst delete mode 100644 docs/raft/0.1/introduction.rst delete mode 100644 docs/raft/0.1/troubleshooting.rst diff --git a/docs/raft/0.1/configuring_deploying.rst b/docs/raft/0.1/configuring_deploying.rst deleted file mode 100644 index ec568bfd..00000000 --- a/docs/raft/0.1/configuring_deploying.rst +++ /dev/null @@ -1,404 +0,0 @@ -*************************************** -Configuring and Deploying Sawtooth Raft -*************************************** - -This guide assumes previous experience with creating and deploying Sawtooth -networks; if you are unfamiliar with Sawtooth, please see the `Application -Developer's Guide`_ and `System Administrator's Guide`_ in the Sawtooth Core -documentation. - -.. _Application Developer's Guide: https://sawtooth.hyperledger.org/docs/core/nightly/master/app_developers_guide.html -.. _System Administrator's Guide: https://sawtooth.hyperledger.org/docs/core/nightly/master/sysadmin_guide.html - -This guide also requires a basic understanding of the Raft consensus algorithm; -if you are not familiar with Raft, please see the `Raft`_ webpage. - -.. _Raft: https://raft.github.io/ - - -.. _validator-peering-requirements-label: - -Validator Peering Requirements -============================== - -Sawtooth Raft requires a fully-peered network where all nodes are able to -communicate with all other nodes. This is necessary to establish a leader -through the election process, as well as to maintain leadership through regular -heartbeat messages. If a Raft network is not fully-peered, leadership election -may fail due to some nodes being unable to communicate with one another. - -In order to achieve a fully-peered network, the list of peers that comprise the -network will need to be known and maintained at all times. This means that Raft -networks do not support open membership; an administrator must explicitly add or -remove nodes to change membership (see `Changing Network Membership`_ for more -details). - -To setup a fully-peered network, all validators must be started with the -following two options: - -* ``--peering static`` - Informs the validator to connect with all of the - specified peers. -* ``--peers {peers_list}`` - Provides the list of peers for the validator to - connect to, where `{peers_list}` is the list of public endpoints for all other - previously started validators. - -For example, if there are two validators already running with public endpoints -``203.0.113.1:8800`` and ``203.0.113.2:8800``, a new validator would be started -with the options ``--peering static --peers -tcp://203.0.113.1:8800,tcp://203.0.113.2:8800``. - -Since peering is bi-directional, when a new validator connects with existing -validators, the existing validators will be fully peered as well. For example, -to startup a 3 node network where the nodes have endpoints ``alpha``, ``beta``, -and ``gamma``, you would do the following: - -1. Startup the alpha validator with no peers specified (but ``--peering static`` - should still be used) -2. Startup the beta validator with ``--peers tcp://alpha:8800`` -3. Startup the gamma validator with ``--peers tcp://alpha:8800,tcp://beta:8800`` - - -On-Chain Settings -================= - -When starting a fresh network with the Sawtooth Raft consensus engine, the -required settings below (and the desired optional settings) should be set in the -genesis block. - - -Required Settings ------------------ - -* ``sawtooth.consensus.algorithm.name`` - Tells the network the name of the - consensus engine to use. Must be set to ``raft``. -* ``sawtooth.consensus.algorithm.version`` - Tells the network the version of - the consensus engine to use. Must match the version of Raft being used. -* ``sawtooth.consensus.raft.peers`` - A JSON list of public keys of all - validators in the network, formatted as hex strings. - -The ``sawtooth.consensus.raft.peers`` setting is extremely important, since the -Raft engine uses it as its source-of-truth for which nodes to communicate with. -When a Raft engine starts initially, it checks the setting to determine which -nodes are in the network. This setting is also used to add or remove nodes from -the Raft network as covered in the `Changing Network Membership`_ section. As an -example, the value of ``sawtooth.consensus.raft.peers`` would be formatted like -this: - -.. code-block:: json - - [ - "dc26a7099e81bb02869cc8ae57da030fbe4cf276b38ab37d2cc815fec63a14ab", - "df8e8388ced559bd35c2b05199ca9f8fbebb420979715003355dcb7363016c1d" - ] - - -Optional Settings ------------------ - -* ``sawtooth.consensus.raft.heartbeat_tick`` - Determines the interval at which - the leader sends heartbeat messages to all other nodes. Default is ``2``. -* ``sawtooth.consensus.raft.election_tick`` - Determines the timeout after which - nodes can start new elections if no messages have been received from the - leader. Default is ``20``. -* ``sawtooth.consensus.raft.period`` - Determines the length of time (in - milliseconds) that the consensus engine will wait between initializing and - finalizing a block. Default is ``3000``. - -These optional settings should only be set `before` any Raft engines have -started up; if they are changed on a running network and a new node is started, -these values will be different on the new engine than the rest of the network, -which could result in unpredictable behavior. - - -Starting a Raft Engine -====================== - - -Configuring the State Storage Directory ---------------------------------------- - -The Raft engine stores its state on the file system; by default, it uses the -``/var/lib/sawtooth-raft`` directory, but this can be changed by setting the -``SAWTOOTH_RAFT_HOME`` environment variable to the desired path. - - -Logging Configuration ---------------------- - -Sawtooth Raft logs to the console by default, but supports -configurable logging through the `log4rs`_ and `log4rs-syslog`_ libraries. The -`sawtooth-raft repository`_ provides an example logging configuration file that -demonstrates how to configure Raft to use syslog (``logging/syslog.yaml``). For -more examples of logging configuration files, please see the -`log4rs documentation`_. - -.. _log4rs: https://github.com/sfackler/log4rs -.. _log4rs-syslog: https://github.com/im-0/log4rs-syslog -.. _sawtooth-raft repository: https://github.com/hyperledger/sawtooth-raft -.. _log4rs documentation: https://docs.rs/log4rs/0.8.0/log4rs/ - -The Raft engine provides the ``-L`` option for specifying a logging -configuration YAML file. For example, to start Raft with the provided syslog -configuration, you would provide the option ``-L logging/syslog.yaml``. - -Sawtooth Raft also supports configurable logging verbosity at run-time. By -default, it logs with the ``WARN`` logging level (or the logging level specified -in the configuration file), but this can be changed by providing one or more -verbosity flags on startup: - -- ``-v`` - ``INFO`` logging level -- ``-vv`` - ``DEBUG`` logging level -- ``-vvv`` - ``TRACE`` logging level - - -Connecting to the Validator ---------------------------- - -When a Sawtooth Raft engine is started, it must be connected to the validator, -which can be done using the ``--connect`` command line option. For example, if -the validator is running on host ``203.0.113.0`` and is using the default -consensus port ``5050``, then the Raft engine should be started with ``--connect -tcp://127.0.0.1:5050``. If this option is not specified, Sawtooth Raft will -attempt to connect with the default validator address: ``tcp://localhost:5050``. - - -Verifying Raft Is running -------------------------- - -This section assumes that Raft is being run with logging level ``INFO``. - -When the Sawtooth Raft process starts initially, you will see a message that -indicates the version of the Raft engine and the validator endpoint that it is -attempting to connect to: - -.. code-block:: console - - INFO | sawtooth_raft:84 | Sawtooth Raft Engine (X.Y.Z) - INFO | sawtooth_raft:90 | Raft Node connecting to 'tcp://validator:5050' - -This indicates that the Raft process has started; however, it does not indicate -that the Raft engine itself is running. The Raft engine waits until the genesis -block has been received and committed by the validator before running. Once the -genesis block has been committed, you will see a message in the validator's logs -indicating that the Raft engine has been registered: - -.. code-block:: console - - Consensus engine registered: sawtooth-raft-engine X.Y.Z - -If you do not see the message above in the validator logs, make sure that the -Raft engine is properly connecting to the validator (see `Connecting to the -Validator`_) and that the validator has committed the genesis block with the -required Raft settings (see `Creating the Genesis Block`_ in the Sawtooth Core -documentation). - -.. _Creating the Genesis Block: https://sawtooth.hyperledger.org/docs/core/nightly/master/sysadmin_guide/creating_genesis_block.html - -Once the consensus engine is running and connected to the validator, you will -see a message in the Raft log that displays the configuration values that are -being used by the Raft engine, similar to this one: - -.. code-block:: console - - INFO | sawtooth_raft::engin | Raft Engine Config Loaded: RaftEngineConfig { peers: [026c49c05b153ca92e2fae01fea85663ae397eb435cc7744907edfe839e84fb288], period: 3s, raft: { election_tick: 20, heartbeat_tick: 2, applied: 0 }, storage: cached storage: file-system backed persistent storage } - -You should also see a group of messages that indicate if the node has been -elected leader, similar to this: - -.. code-block:: console - - INFO | raft::raft:833 | [15456778813275318575] became follower at term 0 - INFO | raft::raft:433 | [15456778813275318575] newRaft [peers: [15456778813275318575], term: 0, commit: 0, applied: 0, last_index: 0, last_term: 0] - INFO | raft::raft:833 | [15456778813275318575] became follower at term 1 - INFO | raft::raft:1120 | [15456778813275318575] is starting a new election at term 1 - INFO | raft::raft:848 | [15456778813275318575] became candidate at term 2 - INFO | raft::raft:950 | [15456778813275318575] received MsgRequestVoteResponse from 15456778813275318575 at term 2 - INFO | raft::raft:891 | [15456778813275318575] became leader at term 2 - -At this point, the Raft engine is running and ready to handle blocks. - - -Starting a Multi-Node Raft Network -================================== - -The `sawtooth-raft repository`_ provides a set of Docker Compose files in the -``adhoc`` directory that allow one to quickly and easily setup a Raft network -using Docker. The compose files in this directory are designed to make manual, -ad-hoc deployments and testing of Sawtooth Raft networks simpler. - - -Starting the Network --------------------- - -Make sure you have all the Docker images you need: - -- sawtooth-validator (>= v1.1) -- sawtooth-rest-api -- sawtooth-intkey-tp-python -- sawtooth-intkey-workload -- sawtooth-settings-tp - -To build these from `sawtooth-core`_, run the following: - -.. _sawtooth-core: https://github.com/hyperledger/sawtooth-core -.. code-block:: console - - $ docker-compose -f docker-compose-installed.yaml build \ - validator \ - rest-api \ - intkey-tp-python \ - intkey-workload \ - settings-tp - - -Starting the Admin Service -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -This compose file creates the shared network and volume required for the -validator network to communicate. - -.. code-block:: console - - $ docker-compose -f admin.yaml up -d - - -Starting the Nodes -~~~~~~~~~~~~~~~~~~ - -If N is the number of nodes you want to create, startup N-1 nodes with the -``node.yaml`` compose file. For each node, you will need to select a unique name -and provide this with the ``-p`` option. Example: - -.. code-block:: console - - $ docker-compose -p alpha -f node.yaml up - -Note that you can include the ``-d`` flag to create all nodes from a single -terminal. - -For the last node, do the same as above, but also set the ``GENESIS`` environment -variable. Example: - -.. code-block:: console - - $ GENESIS=1 docker-compose -p genesis -f node.yaml up - -Note that each time a node is started, it adds itself to the list of nodes that -all new nodes will connect to. This means that if you mess something up starting -a node, you may need to start over. To do this, use ``docker-compose down`` for -each node you started, passing the appropriate ``-p`` value; then do -``docker-compose -f admin.yaml down``. - - -Stopping and Restarting Nodes On a Network -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -To stop a node on a network, you need to stop the containers for that node -**without removing them**. If you remove the containers, then that node is gone -for good. To avoid for this, do **NOT** use ``docker-compose down``. - -If you are attached to the containers, you can press CTRL+C to stop the node's -containers. - -If you are not attached to the containers, use the command ``docker-compose -stop`` with the appropriate ``-p`` flag. - -After you have stopped a node, you can use ``docker-compose start`` with the -appropriate ``-p`` flag to restart the stopped node. - - -Verifying the Network Is Ready -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When the network is setup correctly, Raft will elect a node as leader. If all -nodes are peered properly, you should also see a group of messages in the logs -of each Raft engine that indicate if the node has been elected leader or if it -has voted for another node as leader. On a leader node, the logs will look -similar to this: - -.. code-block:: console - - INFO | raft::raft:833 | [12797589408118497989] became follower at term 0 - INFO | raft::raft:433 | [12797589408118497989] newRaft [peers: [9142281103993713288, 12797589408118497989], term: 0, commit: 0, applied: 0, last_index: 0, last_term: 0] - INFO | raft::raft:833 | [12797589408118497989] became follower at term 1 - INFO | raft::raft:1120 | [12797589408118497989] is starting a new election at term 1 - INFO | raft::raft:848 | [12797589408118497989] became candidate at term 2 - INFO | raft::raft:950 | [12797589408118497989] received MsgRequestVoteResponse from 12797589408118497989 at term 2 - INFO | raft::raft:927 | [12797589408118497989] [logterm: 1, index: 2] sent MsgRequestVote request to 9142281103993713288 at term 2 - INFO | raft::raft:950 | [12797589408118497989] received MsgRequestVoteResponse from 9142281103993713288 at term 2 - INFO | raft::raft:1648 | [12797589408118497989] [quorum:2] has received 2 MsgRequestVoteResponse votes and 0 vote rejections - INFO | raft::raft:891 | [12797589408118497989] became leader at term 2 - -On a node that becomes a follower and votes for another node, the logs will look -like this: - -.. code-block:: console - - INFO | raft::raft:833 | [9142281103993713288] became follower at term 0 - INFO | raft::raft:433 | [9142281103993713288] newRaft [peers: [9142281103993713288, 12797589408118497989], term: 0, commit: 0, applied: 0, last_index: 0, last_term: 0] - INFO | raft::raft:833 | [9142281103993713288] became follower at term 1 - INFO | raft::raft:1014 | [9142281103993713288] [term: 1] received a MsgRequestVote message with higher term from 12797589408118497989 [term: 2] - INFO | raft::raft:833 | [9142281103993713288] became follower at term 2 - INFO | raft::raft:1181 | [9142281103993713288] [logterm: 1, index: 2, vote: 0] cast MsgRequestVote for 12797589408118497989 [logterm: 1, index: 2] at term 2 - -This indicates that a leader has been elected and that the network is ready. - - -Applying Workload On a Network -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A workload generator can be started against the network using the -``workload.yaml`` compose file. To change the rate from the default, set the -``RATE`` environment variable prior to running the compose command. - -For example, to start a workload of 2 TPS against the network, do: - -.. code-block:: console - - $ RATE=2 docker-compose -f workload.yaml up - -To stop the workload, simply do: - -.. code-block:: console - - $ docker-compose -f workload.yaml down - - -Changing Network Membership -=========================== - -Even though membership of a Sawtooth Raft network is not open, it does support -changes. Membership is changed by updating the ``sawtooth.consensus.raft.peers`` -on-chain setting to add a new node or remove an existing node. The leader -monitors this setting; when it detects that a new key is present (adding a node) -or that a key has been deleted (removing a node), it stops publishing blocks -and proposes the configuration change to the network. - -Due to limitations in the raft-rs library, only one node may be added or removed -at a time. This means that the updated value of -``sawtooth.consensus.raft.peers`` may have a single key removed **OR** it may -have a single key added, but not both. If the leader detects that more than one -node's membership has changed, it will report an error and shutdown. - - -Adding a Node -------------- - -The new Sawtooth Raft engine should be started before the update to the on-chain -setting is made. Once the setting is updated to include the public key of the -new node and the network has been made aware of the change, the leader will -begin to send messages to the new node and bring it into consensus with the rest -of the network. - - -Removing a Node ---------------- - -The on-chain setting should be updated before the node is shutdown. When the -setting has been updated to no longer include the old node's public key, the -leader will stop sending messages to the node and ignore any messages from it. -Once this happens, the node can be shutdown safely since it is no longer -participating in the network. - -.. Licensed under Creative Commons Attribution 4.0 International License -.. https://creativecommons.org/licenses/by/4.0/ diff --git a/docs/raft/0.1/introduction.rst b/docs/raft/0.1/introduction.rst deleted file mode 100644 index b39a0364..00000000 --- a/docs/raft/0.1/introduction.rst +++ /dev/null @@ -1,68 +0,0 @@ -************ -Introduction -************ - -Sawtooth Raft is a consensus engine for Hyperledger Sawtooth based on the crash -fault tolerant consensus algorithm `Raft`_. -Specifically, it is built on top of the Rust implementation of Raft used by -TiKV, `raft-rs`_. - -.. _Raft: http://raft.github.io/ -.. _raft-rs: https://github.com/pingcap/raft-rs - -Sawtooth Raft differs from the PoET SGX and devmode consensus engines in a few -major ways: - -* **Raft is a leader-based consensus algorithm** - The Raft algorithm defines - an election process whereby a leader is established and recognized by all - followers. This means that only one node—the leader—publishes blocks, which - are then validated and agreed on by the other nodes in the network—the - followers. -* **Raft is non-forking** - No forks arise on a Raft network because only one - node can publish blocks. This means that no fork resolution needs to be done, - which reduces the overhead associated with block management and lends itself - to increased performance. -* **Raft membership is fixed** - Raft requires that a majority of nodes agree on - all progress that is made. To ensure safety by preventing invalid node - configurations, Sawtooth Raft does not support open membership. The population - of nodes is fixed unless changed by an administrator. -* **Raft networks should be small** - Raft requires exchanging messages between - all nodes to reach consensus for each block, and the leader must wait for a - majority of nodes to agree on a new entry before it can be committed. The - number of messages that need to be exchanged increases exponentially with the - size of the network, which make large networks impractical. -* **Raft is not Byzantine fault tolerant** - The Raft algorithm was designed to - be crash fault tolerant—it can continue to make progress as long as a majority - of its nodes are available. However, Raft only guarantees safety and - availability under non-Byzantine conditions, which makes it ill-suited for - networks that require Byzantine fault tolerance. - -The Sawtooth Raft consensus engine may be right for you if your Sawtooth -deployment will: - -1. Consist of a small number of nodes (roughly 1 to 10) -2. Have a mostly fixed membership -3. Not require Byzantine fault tolerance - - -Join the Sawtooth Community -=========================== - -Sawtooth is an open source project under the Hyperledger umbrella. We welcome -working with individuals and companies interested in advancing distributed -ledger technology. Please see :doc:`/community` for ways to become a part of -the Sawtooth community. - - -Acknowledgements -================ - -This project uses software developed by the OpenSSL Project for use in the -OpenSSL Toolkit (http://www.openssl.org/). - -This project relies on other third-party components. For details, see the -LICENSE and NOTICES files in the `sawtooth-raft repository -`_. - -.. Licensed under Creative Commons Attribution 4.0 International License -.. https://creativecommons.org/licenses/by/4.0/ diff --git a/docs/raft/0.1/troubleshooting.rst b/docs/raft/0.1/troubleshooting.rst deleted file mode 100644 index 5016d882..00000000 --- a/docs/raft/0.1/troubleshooting.rst +++ /dev/null @@ -1,120 +0,0 @@ -********************************* -Troubleshooting and Common Issues -********************************* - -Below is a list of common issues that are seen when attempting to start and run -a Sawtooth Raft network. If you encounter any problems, please consult this list -to see if your issue is discussed. If you are still having problems, find out -how to get help in the :doc:`/community/join_the_discussion` section. - - -No Leader Is Elected -==================== - -When a Sawtooth Raft network is unable to elect a leader, one or more nodes will -continuously start new elections. The logs will probably look similar to this: - -.. code-block:: console - - INFO | raft::raft:833 | [10283357333226787554] became follower at term 1 - INFO | raft::raft:1120 | [10283357333226787554] is starting a new election at term 1 - INFO | raft::raft:848 | [10283357333226787554] became candidate at term 2 - INFO | raft::raft:950 | [10283357333226787554] received MsgRequestVoteResponse from 10283357333226787554 at term 2 - INFO | raft::raft:927 | [10283357333226787554] [logterm: 1, index: 2] sent MsgRequestVote request to 6831727538046783822 at term 2 - INFO | raft::raft:1120 | [10283357333226787554] is starting a new election at term 2 - INFO | raft::raft:848 | [10283357333226787554] became candidate at term 3 - INFO | raft::raft:950 | [10283357333226787554] received MsgRequestVoteResponse from 10283357333226787554 at term 3 - INFO | raft::raft:927 | [10283357333226787554] [logterm: 1, index: 2] sent MsgRequestVote request to 6831727538046783822 at term 3 - INFO | raft::raft:1120 | [10283357333226787554] is starting a new election at term 3 - INFO | raft::raft:848 | [10283357333226787554] became candidate at term 4 - INFO | raft::raft:950 | [10283357333226787554] received MsgRequestVoteResponse from 10283357333226787554 at term 4 - INFO | raft::raft:927 | [10283357333226787554] [logterm: 1, index: 2] sent MsgRequestVote request to 6831727538046783822 at term 4 - INFO | raft::raft:1120 | [10283357333226787554] is starting a new election at term 4 - INFO | raft::raft:848 | [10283357333226787554] became candidate at term 5 - INFO | raft::raft:950 | [10283357333226787554] received MsgRequestVoteResponse from 10283357333226787554 at term 5 - INFO | raft::raft:927 | [10283357333226787554] [logterm: 1, index: 2] sent MsgRequestVote request to 6831727538046783822 at term 5 - ... - -This is usually a sign that the nodes are not peering properly. If you encounter -this problem, see `Troubleshooting Connectivity`_ for tips on how to resolve it. - - -Troubleshooting Connectivity -============================ - -If a Raft network is not setup correctly, some nodes may not be able to -communicate; this is detrimental to the functionality of the Raft algorithm, so -it is usually quite obvious when there is a connectivity issue. Common signs -that some nodes are unable to communicate with each other include: - -- Failing to elect or keep a leader -- One or more nodes not making progress with the rest of the network -- Nodes repeatedly sending a message without a response - -Below are a few common causes for node connectivity issues and how to resolve -them. - - -Validator Peering Configuration -------------------------------- - -If you are experiencing network peering problems, the first thing to do is to -verify that the validators are configured to peer properly. If the validators -are not started up with static peering or the list of all existing validator -endpoints is not supplied to a validator, some nodes will not be able to -communicate. See the section on :ref:`validator-peering-requirements-label` to -ensure that you are providing the correct parameters for the validators to peer -properly. - - -Max Peers Exceeded ------------------- - -If you are running a large Raft network (around 10 nodes), you might see an -error in one or more of the validators' logs indicating that a peering request -was rejected because the validator has reached its maximum number of peers. The -message will look like this: - -.. code-block:: console - - Unable to successfully peer with connection_id: 208f8143e9f50aeacf7583bc599e2e4dcd206acfbd0499de6f9b05ac49ad44fcc8a993dea62c43fdbdfe78e19e41c4efec6f7b6e0c2a9a1722b8d008bf97ad91, due to At maximum configured number of peers: 10 Rejecting peering request from tcp://03a56e9f368a:8800. - -To resolve this issue, you will need to increase the peer limit for all -validators. If you are starting the validator from the command line, you can -specify the maximum number of peer connections with the -``--maximum_peer_connectivity`` option (see the `sawtooth-validator -documentation`_ for more details). - -.. _sawtooth-validator documentation: https://sawtooth.hyperledger.org/docs/core/nightly/master/cli/sawtooth-validator.html - - -Troubleshooting ``sawtooth.consensus.raft.peers`` Setting -========================================================= - -When the ``sawtooth.consensus.raft.peers`` on-chain setting is not configured -properly, the Raft network will fail to work. This may result in nodes not -communicating properly, or it may cause one or more nodes to crash with a -message like: - -.. code-block:: console - - raft_1 | thread 'main' panicked at 'Peer id not valid hex: OddLength', libcore/result.rs:945:5 - raft_1 | note: Run with `RUST_BACKTRACE=1` for a backtrace. - -Here are some tips for troubleshooting this setting. - -List Format ------------ - -The list of peers must be a valid JSON formatted list, similar to this: - -.. code-block:: console - - sawtooth.consensus.raft.peers=["dc26a7099e81bb02869cc8ae57da030fbe4cf276b38ab37d2cc815fec63a14ab","df8e8388ced559bd35c2b05199ca9f8fbebb420979715003355dcb7363016c1d"] - -If your list looks like the above but you are still encountering errors, try -escaping the quotes in the JSON string: - -.. code-block:: console - - sawtooth.consensus.raft.peers=[\"dc26a7099e81bb02869cc8ae57da030fbe4cf276b38ab37d2cc815fec63a14ab\",\"df8e8388ced559bd35c2b05199ca9f8fbebb420979715003355dcb7363016c1d\"]