Skip to main content

Production Network Deployment Guide

This page illustrates the process of deploying a new production network comprising multiple independent nodes.

Overview​

The Tutorial page illustrates the process of creating a new network of nodes by running each one on a single development machine. While that is a quick and concise demonstration of how nodes interact and the validator management process, a real production network requires the deployment of nodes on different host machines that typically operate in remote locations. This page demonstrates the deployment of such a network, with attention to the relevant settings and considerations in such a setting.

Concepts​

There are several important concepts to understand when creating a network of independent remote nodes.

Peers and Node IDs​

Every Kwil node has a "node ID" that is includes the node's public key and network address. This plays an important role creating secure authenticated connections between nodes over the internet.

A peer is always specified in the format <public key>#<key type>@<IP address>:<port>. For example, given a node with public key 02977e1f9bcec8c1b1905cf45cc5cfcfbabf9ca579d42d1b1a228e9b0f1c6adecb and key type secp256k1 that is reachable at IP address 10.1.2.3 on TCP port 6600, the notation used to specify this node as a peer is 02977e1f9bcec8c1b1905cf45cc5cfcfbabf9ca579d42d1b1a228e9b0f1c6adecb#[email protected]:6600.

note

Unlike node IDs, which are used to secure P2P node communication, Kwil DB user addresses are in their own formats, such as checksummed Ethereum addresses, to support native integration with web3 wallets for database access control and user identification. The set of supported P2P key types is also different from the types of user addresses supported, so an explicit public key format is used for peers.

An operator can determine their node ID in different ways:

  1. Use the kwild key info command, which derives and displays the node ID given the private key.
  2. Use the kwild admin status command, which retrieves the public key and its type from the running node's admin RPC service.
  3. Locate the node ID in the node logs. An info-level log entry with the message "NODE: This node is" contains the node identity (public key and key type) and all listening addresses

The node's public key is also displayed when the private key is initially generated with the kwild key gen subcommand.

Peer Exchange​

In a healthy network, nodes have many peers to prevent single points of failure or unintended network partitions, and to support efficient network gossip. The feature that facilitates this is called peer exchanges (PEX). This is also enabled by default with the p2p.pex = true setting. Disable this setting only if a node must not share its peers with other nodes.

Network Bootstrapping​

The p2p.bootnodes setting specifies a list of peers to connect to on startup. This is useful for bootstrapping a new node, especially when used in conjunction with PEX. To specify two remote peers to always connect to on startup, list them as follows:

[p2p]
bootnodes = "02ae12856a766c277842e1319fd7e450711b706ef90942e8ad5be9369e5d22817c#[email protected]:6600,02cace485e891cb5e5bd22b008986980791677f9f553fd3bb3f864e6b84e011cdb#[email protected]:6600"

This may also be specified on the command line with the --p2p.bootnodes option, which may be specified multiple times for each peer rather than a comma-separated list. For example:

kwild start \
  --p2p.bootnodes 02ae12856a766c277842e1319fd7e450711b706ef90942e8ad5be9369e5d22817c#[email protected]:6600 \
  --p2p.bootnodes 02cace485e891cb5e5bd22b008986980791677f9f553fd3bb3f864e6b84e011cdb#[email protected]:6600

Validators​

While not a networking concept, it is important to understand the role of validators as it pertains to block production. The consensus engine used by Kwil requires greater than 1/2 of validators to approve proposed blocks. Thus, if the network is not well connected and validators are unable to communicate efficiently, block production may be slowed or even stopped in the case of a network partition.

Not all nodes in a network must be validators. Non-validator nodes called "sentry" nodes are typically created to function as RPC service providers or just to support network health by relaying transactions, blocks, and misbehavior evidence.

Compatible Versions​

A final important concept is version compatibility. On a production network, it is important to only run nodes built from the same release branch, such as the release-v0.8 branch, or a release tag with the same MAJOR.MINOR version. Different patch versions are compatible, but large releases that change the application's state machine logic cannot be run on the same network. For example, it is not possible to run v0.8.4 and v0.9.1 on the same network. To update a network between such major releases, it is necessary to perform a migration.

Walkthrough​

This section illustrates the process for launching a small network of independent remote nodes. It will begin with two validators at genesis, followed by the addition of a sentry node using block sync, and finally the new node will become a validator.

Deploying Genesis Nodes and Validators​

Starting a network with a single validator is possible, however, we will demonstrate starting with two validators from genesis since this also illustrates the consensus considerations described above in Validators.

Generating Node Private Keys​

danger

Securely generate and store private keys. Anyone in possession of a private key can imitate and act on behalf of a node. In the case of a validator, this impacts the security of the network.

Every node has a unique identity, which is determined by its private key. We will generate a private key using the kwild key gen command on the node0 host machine:

[node0]:~$ kwild key gen --key-file ~/nodekey.json

This will write a file named nodekey.json that contains the ed25519 private key for the new node into the user's home directory. Each node must have a different private key. This command should be run independently by each validator operator.

We can inspect the key using the kwild key info command as follows:

[node0]:~$ kwild key info --key-file ~/nodekey.json
Key type: secp256k1
Private key (hex): da1ad3ad3e80d200ff66beb6515360337f6e61f14ce958e5bda717fef5aad9f5
Public key (plain hex): 02d3575f74ffcdb17f781f003d42284a4d1adf271558d0417423b8372ec3f34750

The "Public key (plain hex)" and "Key type" value is used in the genesis.json file to specify the initial validator set as described in the next section.

Creating the Genesis Document​

Before starting the first nodes, it is necessary is to create a genesis file for the new network. The file specifies the validator set at genesis, initial account funding allocations, and other network parameters. All nodes must use the exact same genesis.json file since its parameters affect he consensus rules for the network.

Unlike the node quick start guide where all genesis validator keys are generated with the kwild setup testnet command run once on one machine, we will create a genesis file with the kwild setup genesis command. This takes as input:

  1. the plain hex public keys, key types, and power of each genesis validator (formatted as name:pubkey:power)
  2. the "chain ID" string that is the new network's name
  3. any other default parameter overrides

Consider we want to start a new network called "prod-chain" with two genesis validators:

[organizer]:~$ kwild setup genesis \
  --validator "0277fb40d1b745c8f96f411ab57915e2df3bc86efde70f0e687f4273beb6708b82#secp256k1:1" \
  --validator "03b0eedfb09ac1e19fa6f5eb1d5ade53c120ef32d988e60bb40ab08133b573ba30#secp256k1:1" \
  --leader "0277fb40d1b745c8f96f411ab57915e2df3bc86efde70f0e687f4273beb6708b82#secp256k1" \
  --chain-id "prod-chain" --out ~/prod-chain
Created genesis.json file at /home/user/prod-chain/genesis.json

The resulting genesis file has the chain ID and validators customized, with default values for the rest of the document:

genesis.json
{
  "chain_id": "prod-chain",
  "initial_height": 0,
  "validators": [
    {
      "pubkey": "0277fb40d1b745c8f96f411ab57915e2df3bc86efde70f0e687f4273beb6708b82",
      "type": "secp256k1",
      "power": 1
    },
    {
      "pubkey": "03b0eedfb09ac1e19fa6f5eb1d5ade53c120ef32d988e60bb40ab08133b573ba30",
      "type": "secp256k1",
      "power": 1
    }
  ],
  "migration": {
    "start_height": 0,
    "end_height": 0
  },
  "leader": {
    "type": "secp256k1",
    "key": "0277fb40d1b745c8f96f411ab57915e2df3bc86efde70f0e687f4273beb6708b82"
  },
  "db_owner": "0277fb40d1b745c8f96f411ab57915e2df3bc86efde70f0e687f4273beb6708b82",
  "max_block_size": 6291456,
  "join_expiry": "24h0m0s",
  "disabled_gas_costs": true,
  "max_votes_per_tx": 200,
  "migration_status": "NoActiveMigration"
}

Provide this same genesis.json file to all node operators. The genesis file with the initial validators will remain the same for the entire lifetime of the network, even as the network adds new validators.

Node Configuration​

Unlike genesis.json, which contains network-wide settings, the config.toml file is used to configure a Kwil node for its own purpose and network configuration. Use the kwild setup init command to create the config.toml file for each node separately, setting the listen addresses, external address, persistent peers, and seeds as appropriate for the node.

The kwild setup init command initializes a node's "root" directory, which by default is ~/.kwild. We will use the agreed upon genesis file using the --genesis flag. Since these are genesis validators, we also use the --key-file flag to specify the nodekey.json file that was generated above on each node. Other nodes that are not genesis validators may allow setup init to generate a new key by omitting this flag.

Consider we have prepared the keys and addresses for the two nodes described above as follows:

IP AddressNode ID
node023.45.67.890277fb40d1b745c8f96f411ab57915e2df3bc86efde70f0e687f4273beb6708b82#secp256k1
node198.76.54.3203b0eedfb09ac1e19fa6f5eb1d5ade53c120ef32d988e60bb40ab08133b573ba30#secp256k1

To have them connect to each other, we will set their p2p.bootnodes settings appropriately.

On node0, we can initialize the root directory and the config.toml there with the following command, specifying the path to the key file that was generated on this machine:

node0
[node0]:~$ kwild setup init --genesis prod-chain/genesis.json \
  --root /data/kwil-node0 --key-file ~/nodekey.json
Kwil node configuration generated successfully at: /data/kwil-node0

On node1, do the same, but with node0 as a bootnode

node1
[node1]:~$ kwild setup init --genesis prod-chain/genesis.json \
  --p2p.bootnodes "0277fb40d1b745c8f96f411ab57915e2df3bc86efde70f0e687f4273beb6708b82#[email protected]:6600" \
  --root /data/kwil-node1  --key-file ~/nodekey.json
Kwil node configuration generated successfully at: /data/kwil-node1

Initial Startup and Block Production​

Now that both of the validators have been prepared with the same genesis.json file, and with node1's boot peer configuration pointing to node0, it is time to start the nodes.

tip

Depending on how the nodes are deployed, kwild would be run as a service, container, or in a terminal multiplexer.

PostgreSQL Database​

On each node, install PostgreSQL and create a kwild database. Refer to the Postgres setup guide for more information and quick start instructions.

Start Kwild​

Start the nodes on each of the remote machines, specifying the path to the root directory if it is not at the default location (~/.kwild):

node0
[node0]:~$ kwild start -r /data/kwil-node0
node1
[node1]:~$ kwild start -r /data/kwil-node1

Once the nodes connect, the first block will be produced shortly after:

2025-01-31 17:21:29.992 [INF] CONS: Committed Block {height=1 hash=6dc4f373a39fba30db5c167a328320d735ba62e9e8ca5ef878e31142de50a1eb appHash=2d8f3ceeff2c836527da823d7b654d33d3e44b6159b172235c160001e0c9b4db updates=null}

Recall that both of the validators need to be started and able to communicate before blocks can be created, as discussed in the Validators section. If the nodes are unable to connect, look for peer-related errors in the logs, and check your persistent peers settings and firewall rules.

Adding a New Node​

Now that our network of two validators is running and producing blocks, we will create and synchronize a new node to the network, and once it is synchronized, promote the node to a validator.

Starting as a Sentry Node​

Now we will add a third note to the network as a sentry node. Since this node is not a genesis validator, its public key is not included in the genesis file, which is now immutable. The kwild setup init command can be used to simultaneously initialize a new node's root directory and generate a new private key.

info

When adding a node to the network, start it as a non-validator / sentry node. Always completely synchronize the node to the network before attempting to promote it to a validator.

Like before, we will specify the location of the network's genesis.json file, the host's external IP address, and one or more persistent peers. However, we will use the generated private key:

node2
[node2]:~$ kwild setup init --genesis prod-chain/genesis.json \
  --p2p.bootnodes "0277fb40d1b745c8f96f411ab57915e2df3bc86efde70f0e687f4273beb6708b82#[email protected]:6600" \
  --root /data/kwil-node2
Kwil node configuration generated at /data/kwil-node2

We can see the ID of node2 before starting it:

node2
[node2]:~$ kwild key info --key-file /data/kwil-node2/nodekey.json | grep "Node ID"
Node ID: 03dbe22b9922b5c0f8f60c230446feaa1c132a93caa9dae83b5d4fab16c3404a22#secp256k1

Start node2 using the kwild start -r /data/kwil-node2 command.

Verifying Successful Node Synchronization​

Before adding the node as a validator, ensure that node2 has completely synchronized with the network. You can do this with either the kwil-admin tool, or with the public user service using kwil-cli.

Using kwild admin status, inspect the .sync.best_block_height and .sync.syncing values:

node2
[node2]:~$ kwild admin status
{
  "node": {
    "chain_id": "kwil-test-chain",
    "node_id": "03dbe22b9922b5c0f8f60c230446feaa1c132a93caa9dae83b5d4fab16c3404a22#secp256k1",
    "app_ver": 1,
    "listen_addr": "127.0.0.1:6600",
    "role": "validator",
    "rpc_addr": "/tmp/admin.sock"
  },
  "sync": {
    "app_hash": "537935ed434a25f23bfb18324b36ba1936e8be5d0df1a330eab413d848304d96",
    "best_block_hash": "9a38b3c78d2a4361c7615db779f8beb073b46819bb275295716283d2c700a6ac",
    "best_block_height": 337,
    "best_block_time": "2025-01-27T18:45:00.199-06:00",
    "syncing": false
  },
  "validator": {
    "pubkey": "03dbe22b9922b5c0f8f60c230446feaa1c132a93caa9dae83b5d4fab16c3404a22",
    "type": "secp256k1",
    "power": 1
  }
}

Ensure that the node is no longer syncing (catching up) with the network and that the best block height is within a couple blocks of the network's known height.

Use kwil-cli utils chain-info to query the height of any node with its RPC service exposed:

$ kwil-cli utils chain-info --provider "http://98.76.54.32:8484"
Chain ID: prod-chain
Height: 109
Hash: 1d5a4ef85927dce3c90168e89fdfb32219c5c95bb5ed3332eb69e20bbe7346da

Compare the output of that command with the result from node2 to ensure the new node stays in sync with the network as new blocks are produced.

Alternatively, check the new node's health check REST endpoint, which will return an HTTP status code of 200 and a JSON body with "healthy":true if the node is ready:

node2
[node2]:~$ curl -s --fail http://127.0.0.1:8484/api/v1/health
{"kwil_alive":true,"healthy":true, ...}

Joining the Validator Set​

Unlike node0 and node1, consensus does not require participation from node2, which just executes the blocks that are produced and approved by the validators. In this section we will add node2 to the validator set.

Start by submitting a validator "join" request from node2:

node2
[node2]:~$ kwil-admin validators join 
TxHash: 0722630c4296ea27d77f4ee8db6129c1ca042372811187f7ba787c75464400ed

We have now submitted a request to become a validator, which the existing validators must approve for this node to join the validator set. The validators list-join-requests subcommand from any node can view this request:

$ kwil-admin validators list-join-requests
Pending join requests (1 approval needed):
 Candidate                                                                    | Power | Approvals | Expiration
------------------------------------------------------------------------------+-------+-----------+------------
 03dbe22b9922b5c0f8f60c230446feaa1c132a93caa9dae83b5d4fab16c3404a22#secp256k1 |     1 |         0 | 14627

Both validators are required to approve this request since 1/2 does not meet the threshold of greater than 2/3 of the validators. This is done by running the following on each validator:

$ kwil-admin validators approve 03dbe22b9922b5c0f8f60c230446feaa1c132a93caa9dae83b5d4fab16c3404a22#secp256k1
TxHash: cab4b66d12cd6ecee256ba73dd9500a3ea7b21ba03bd38d5db7701268a0f8ecb

Once the approval transactions are mined, node2 will become part of the validator set:

$ kwil-admin validators list
Current validator set:
  0. {pubkey = 0277fb40d1b745c8f96f411ab57915e2df3bc86efde70f0e687f4273beb6708b82#secp256k1, power = 1}
  1. {pubkey = 03b0eedfb09ac1e19fa6f5eb1d5ade53c120ef32d988e60bb40ab08133b573ba30#secp256k1, power = 1}
  2. {pubkey = 03dbe22b9922b5c0f8f60c230446feaa1c132a93caa9dae83b5d4fab16c3404a22#secp256k1, power = 1}

The node is now a validator. Keep it running and monitor its health.