Skip to main content

FTSOv2 Data Provider

This guide will help you set up an FTSOv2 data provider. FTSO data providers are responsible for sourcing offchain data and bringing it to Flare. FTSOv2 consists of two main types of feeds:

  • Block-latency feeds, protocol code name Fast Updates or fast-updates. Updates with every new block on Flare, every ≈1.8 seconds.

  • Anchor feeds, protocol code name Scaling or ftso-scaling. Updates with every new voting round, which each round lasting exactly 90 seconds.

This guide will help you set up both components, note that only the code names will be used in this guide and any repositories referenced.

Overview​

A data provider system for FTSOv2 consists of five components:

  • Feed Value Provider. Provides the latest values for a given set of feeds.

  • Data Provider. Provides commit, reveal, and median result data to Flare System Client for submission.

  • Flare System Client. Responsible for all interactions with the FTSOv2 smart contracts, including data collection, submission, voter registration, and other system tasks.

  • Fast Updates Client. Responsible for interaction with Fast Updates contracts, including data collection, submission and other system tasks.

  • Indexer. Monitors the blockchain and records all FTSOv2 related transactions and events.

Reference implementations are provided for all five components.

warning

The reference implementation of Feed Value Provider is only provided for testing purposes and should not be relied on. Data providers are expected to reimplement this with their own data sources.

FTSO Provider ArchitectureFTSO Provider Architecture

Operation​

A simplified description of a single voting round operation:

Flare System Client runs a scheduler which triggers voting actions every round (90s):

  • On voting round start, obtain reveal data for the previous round from Data Provider and send to Submission smart contract.

  • Once the reveal deadline passes (45s), obtain median result Merkle root from Data Provider, sign, and send to Submission smart contract.

  • Before the end of the voting round, obtain a feed value commit hash for the current round from the Data Provider (which will get processed in the following round).

  • There is a finalizer process which monitors the indexer database for signature transactions, and once enough signature weight for a voting round is gathered, submits the set of signatures to the Relay smart contract. If signature verification passes, the voting round is considered finalized. The Relay contract is the authoritative storage of confirmed voting round result Merkle roots.

Additionally, once in a reward epoch the System Client triggers voter registration, which allows participating in the following reward epoch.

Data Provider obtains all commit and reveal data straight from encoded transaction calldata recorded in the indexer database. All calls to Submission contract are simply empty function invocations, with the actual submission data provided as additional calldata in the transaction.

Prerequisites​

Ensure you have the following tools installed:

Setup​

Generate sortition key​

Generate a sortition key to use in Fast Updates, read more about this in fast-updates/go-client. You can use the provided docker image to generate a key:

docker run --rm ghcr.io/flare-foundation/fast-updates/go-client:latest keygen

Accounts for submission need to be generated and funded with gas fees. These can be any accounts not used for the five accounts in Registration. We suggest using three separate accounts to avoid nonce conflicts. You will need to pass their private keys in the .env file defined ahead in Configure environment.

Registration​

info

Registration only needs to be performed once.

Each data provider in the FTSOv2 system must set up and register the following five accounts:

  • Identity. Main identity account of the voter. On mainnet this should be held in cold storage. This account is required for initial data provider setup, but is not used during each voting round.

  • Submit. Used for sending commit and reveal transactions.

  • SubmitSignatures. Used for sending voting round result signature transactions. (To avoid nonce conflicts, Flare System Client uses multiple accounts for submitting transactions).

  • SigningPolicy. Used for signature generation during the voting round, and reward epoch signing policy signing (it's a system protocol ran once during reward epoch to establish reward epoch settings, including valid voters and their weights).

  • Delegation. Account to which community should delegate funds (using WNat contract) to increase the vote power of the voter (identity/entity) - and also to later get the rewards.

Accounts need to be funded for gas fees. The delegation account is used for establishing voter power, which can be achieved by wrapping funds directly or by delegation from other accounts. Wrapping can be done via:

warning

The protocol operation uses normalized weights, and the delegation account should have at least 150 Wrapped Native Tokens to obtain a non-zero vote power.

NetworkNative TokenWrapped Native Token
Flare MainnetFLRWFLR
Flare Testnet Coston2C2FLRWC2FLR
Songbird Canary-NetworkSGBWSGB
Songbird Testnet CostonCFLRWCFLR

The contract addresses of the Wrapped Native Tokens (WNat) are in the Solidity Reference.

Registration can be performed in two ways, either:

Register using Hardhat​

To register using Hardhat, start by cloning and building flare-smart-contracts-v2.

git clone https://github.com/flare-foundation/flare-smart-contracts-v2/
cd flare-smart-contracts-v2
yarn
yarn c

Create an entities.json file with the following account addresses and private keys:

entities.json
[
  {
    "identity": {
      "address": "0xca84d6086c5b32212a0cf1638803355d7be31482",
      "privateKey": "<private key hex>"
    },
    "submit": {
      "address": "0x7961de7ad159106a79187379a22d21c1e5a924db",
      "privateKey": "<private key hex>"
    },
    "submitSignatures": {
      "address": "0x7570c09c17f79aa50bab7ba385c0d5ca12c5b4d3",
      "privateKey": "<private key hex>"
    },
    "signingPolicy": {
      "address": "0x9ffa9cf5f677e925b6ecacbf66caefd7e1b9883a",
      "privateKey": "<private key hex>"
    },
    "delegation": {
      "address": "0x95288e962ff1893ef6c32ad4143fffb12e1eb15f",
      "privateKey": "<private key hex>"
    },
    "sortitionPrivateKey": "<private key hex>"
  }
]

Setup the following variables in .env:

danger

Register accounts on mainnet using the explorer to avoid exposing private keys.

warning
  • Do not use .env.template, instead just create a .env using the above example, otherwise the running tasks will fail.
  • Do not use public RPCs, you will get rate limited.

Run the registration tasks:

danger

Register accounts on mainnet using the explorer to avoid exposing private keys.

Register using the explorer​

To register manually you can directly call the EntityManager contract. For the contract address, see the Solidity Reference. The relevant functions on the EntityManager contract are:

The required contract invocation steps can also be found in the register-entities.ts and register-public-keys.ts deployment tasks.

info

When registering the sortition key using the explorer, you will need to manually create a signature and input this into the _verificationData parameter, as described here.

Create the signature using the command below:

docker run --rm ghcr.io/flare-foundation/fast-updates/go-client:latest keygen --key <sortitionKeyPrivateKey> --address <entityAddress>

Install dependencies​

brew install jq gettext bash

Clone the repository​

Clone ftso-v2-provider-deployment:

git clone https://github.com/flare-foundation/ftso-v2-provider-deployment.git
cd ftso-v2-provider-deployment

Configure environment​

Use .env.example to create a .env file:

cp .env.example .env

Set the following variables in .env:

VariableDetails
NODE_RPC_URLURL of your RPC endpoint.
NODE_API_KEYAPI key of your RPC endpoint.
VALUE_PROVIDER_URLEndpoint of your feed value provider, leave default if using the test provider.
IDENTITYAddress of main identity account of the voter (see Registration).
SUBMIT_PKPrivate key of account for sending commit and reveal transactions (see Registration).
SIGNATURES_PKPrivate key of account for sending voting round result signature transactions (see Registration).
SIGNING_PKPrivate key of account used for signature generation during the voting round (see Registration).
FAST_UPDATES_ACCOUNTSComma separated private keys of fast updates submission accounts (see Generate sortition key).
FAST_UPDATES_SORTITION_PRIVATE_KEYPrivate key of the sortition address that was registered with the network (see Registration).
FAST_UPDATES_VALUE_PROVIDER_URLFeed value provider endpoint for fast updates, by default it uses the same endpoint as Scaling, see Obtaining Feed Values.

Populate the provider stack config:

chmod +x populate_config.sh
./populate_config.sh
note

You will need to re-run this command if you modify your .env file.

Run​

Provider stack​

The docker-compose.yaml is already provided, so you can start the provider stack with:

docker compose up -d

To stop everything use docker compose down. The database is persisted in a named docker volume, if you need to wipe the database you will need to remove the volume manually. When the codebase is changed new images can be pulled with docker compose pull.

Feed Value Provider​

warning

The reference implementation of Feed Value Provider is only provided for testing purposes and should not be relied on. Data providers are expected to reimplement this with their own data sources.

Start your own feed value provider, or (for testing only) use the reference feed value provider:

docker run --rm -it --publish "0.0.0.0:3101:3101" --network "ftso-v2-deployment_default" ghcr.io/flare-foundation/ftso-v2-example-value-provider

Once the container is running, you can find the API spec at: http://localhost:3101/api-doc

Issues with starting the reference feed value provider?

Some data providers reported issues with getting the reference feed value provider to start.

For initial testing a fixed value provider can be used that simply returns a constant instead of reading data from external data sources. It can be started by setting the variable VALUE_PROVIDER_IMPL=fixed:

docker run --rm -it --env VALUE_PROVIDER_IMPL=fixed --publish "0.0.0.0:3101:3101" --network "ftso-v2-deployment_default" ghcr.io/flare-foundation/ftso-v2-example-value-provider

You should see the following line in the logs:

WARN [FixedFeed] Initializing FixedFeed, will return 0.01 for all feeds.

Is it working?​

You will see various errors initially on ftso-scaling and flare-system-client, since the data provider will not be registered as a voter for the current reward epoch. There is a time window for voter registration on every reward epoch, and if you leave things running you should eventually see RegisterVoter success logged by flare-system-client. It should then start submitting data successfully in the following reward epoch.

flare-system-client log samples indicating successful operation:

[03-04|06:58:20.000]	DEBUG	protocol/submitter.go:142	submitter submit1 running epoch 567838
[03-04|06:58:20.000]	DEBUG	protocol/submitter.go:143	  epoch is [2024-03-04 06:57:00 +0000 UTC, 2024-03-04 06:58:30 +0000 UTC], now is 2024-03-04 06:58:20.000923016 +0000 UTC m=+234317.457953448
[03-04|06:58:20.001]	DEBUG	protocol/protocol_utils.go:55	Calling protocol client API: http://ftso-scaling-00:3100/submit1/567838/0x5579C824e5550ae24ceFe41B129472c3EC70be5c
[03-04|06:58:20.081]	DEBUG	chain/tx_utils.go:120	Sending signed tx: 0xd4e677859190afca0c5f287e735b959ced12d8f5107bcb14cc31add09cbc92ec
[03-04|06:58:20.212]	DEBUG	chain/tx_utils.go:128	Waiting for tx to be mined...
[03-04|06:58:22.244]	DEBUG	chain/tx_utils.go:134	Tx mined, getting receipt 0xd4e677859190afca0c5f287e735b959ced12d8f5107bcb14cc31add09cbc92ec
[03-04|06:58:22.271]	DEBUG	chain/tx_utils.go:139	Receipt status: 1
[03-04|06:58:22.272]	INFO	protocol/submitter.go:78	submitter submit1 submitted tx
[03-04|06:58:35.001]	DEBUG	protocol/submitter.go:142	submitter submit2 running epoch 567839
[03-04|06:58:35.001]	DEBUG	protocol/submitter.go:143	  epoch is [2024-03-04 06:58:30 +0000 UTC, 2024-03-04 07:00:00 +0000 UTC], now is 2024-03-04 06:58:35.001392224 +0000 UTC m=+234332.458422666
[03-04|06:58:35.001]	DEBUG	protocol/protocol_utils.go:55	Calling protocol client API: http://ftso-scaling-00:3100/submit2/567838/0x5579C824e5550ae24ceFe41B129472c3EC70be5c
[03-04|06:58:35.074]	DEBUG	chain/tx_utils.go:120	Sending signed tx: 0x56fe0f0d60b876122ea13d2ae902c4ad777f26d3e2d44c19742d7fb0a1ae25af
[03-04|06:58:35.197]	DEBUG	chain/tx_utils.go:128	Waiting for tx to be mined...
[03-04|06:58:37.241]	DEBUG	chain/tx_utils.go:134	Tx mined, getting receipt 0x56fe0f0d60b876122ea13d2ae902c4ad777f26d3e2d44c19742d7fb0a1ae25af
[03-04|06:58:37.274]	DEBUG	chain/tx_utils.go:139	Receipt status: 1
[03-04|06:58:37.274]	INFO	protocol/submitter.go:78	submitter submit2 submitted tx
[03-04|06:59:20.000]	DEBUG	protocol/submitter.go:208	signatureSubmitter submitSignatures running epoch 567839
[03-04|06:59:20.003]	DEBUG	protocol/submitter.go:209	  epoch is [2024-03-04 06:58:30 +0000 UTC, 2024-03-04 07:00:00 +0000 UTC], now is 2024-03-04 06:59:20.003337094 +0000 UTC m=+234377.460367537
[03-04|06:59:20.003]	DEBUG	protocol/protocol_utils.go:55	Calling protocol client API: http://ftso-scaling-00:3100/submitSignatures/567838/0x6Bc692221B1feff64218eF6Fb3D1D2cE077a64D3
[03-04|06:59:20.502]	DEBUG	chain/tx_utils.go:120	Sending signed tx: 0xb371cae856bf1f37f52f9db556a346d0de35e2b02b73f87ec2dad63f044d7e8a
[03-04|06:59:20.532]	DEBUG	chain/tx_utils.go:128	Waiting for tx to be mined...
[03-04|06:59:21.564]	DEBUG	chain/tx_utils.go:134	Tx mined, getting receipt 0xb371cae856bf1f37f52f9db556a346d0de35e2b02b73f87ec2dad63f044d7e8a
[03-04|06:59:21.604]	DEBUG	chain/tx_utils.go:139	Receipt status: 1

fast-updates log samples indicating successful operation:

info

Depending on your weight it may take some time until you are selected for the fast-updates. During this waiting time, you may see log messages like:

Public key not registered

No intervention is required here, your provider client is simply waiting until the next reward epoch starts to register its keys.

ftso-v2-deployment-fast-updates  | [04-25|09:00:32.456] INFO    provider/feed_provider.go:65    deltas: +++++++++++++++++++++-++++0+
ftso-v2-deployment-fast-updates  | [04-25|09:00:32.456] INFO    client/client_requests.go:205   submitting update for block 14266248 replicate 0: +++++++++++++++++++++-++++0+
ftso-v2-deployment-fast-updates  | [04-25|09:00:33.496] INFO    client/client_requests.go:239   successful update for block 14266248 replicate 0 in block 14266250

If you made it this far, congratulations! You have successfully set up an FTSOv2 data provider.