Technical Overview of the Chainlink Testnet Implementation

Chainlink Node: The Chainlink node is written in Golang. Golang has unique features that are not seen in most programming languages, such as Goroutines and channels. If you don’t have experience with them, it would be a good idea to understand these two features. The Chainlink node exposes a command line interface that allows users to start the Chainlink node, show all/single job run(s), create a JobSpec, begin a job run, backup the database, import a keyfile, etc. To start the Chainlink node, a user will put the name of the Chainlink executable file followed by the word node or n. In addition, a password is specified on the command line. This command line call ends up calling the RunNode function. The main objective of the RunNode function is to connect to the Ethereum blockchain, create the ChainlinkApplication object, authenticate with the KeyStore, call the ChainlinkApplication start method, and setup the the Chainlink REST API(allows the creation of JobSpecs, bridge adapters,etc). The ChainlinkApplication object is important because it holds references to the objects that control the flow of how the node works. Upon instantiation, the ChainlinkApplication will hold these references: HeadTracker “holds and stores the latest block number experienced by this particular node in a thread safe manner.” EthereumListener “manages push notifications from the ethereum node’s websocket to listen for new heads and log events.” EthereumListener controls the logic for RunLog and Ethlog JobSpecs.   Scheduler controls the logic for Cron and RunAt JobSpecs. Store “contains fields for the database, Config, KeyStore, and TxManager for keeping the application state in sync with the database.” Upon instantiation, the store will set up a socket connection to the Ethereum blockchain by connecting to the configurable EthereumURL field in config.go. Exiter exits the Chainlink node with a specific exit code. Keystore authentication: After the ChainlinkApplication object is instantiated, authentication with the Keystore occurs. An ethereum keystore is an encrypted version of your Ethereum private key that can be decrypted by entering a password. Once, a keystore is decrypted it will allow you to sign transactions and move funds from your account. If a password was specified on the command line when the Chainlink node was started, Chainlink will check if it can unlock all of the keystore files located in a configurable directory on your filesystem. If the password cannot unlock all of the keystore files, then Chainlink will not start. If a password is not presented when the node is started, the user will be prompted for a password. If a keystore file is not present, then a keystore file will be created. To stake Link tokens a keystore must be imported into the Chainlink node. ChainlinkApplication Start method: Once authentication completes, the ChainlinkApplication Start method is called — this will start the Chainlink node. The method creates a channel that is notified when the SIGINT or SIGTERM system calls are observed(common termination signals such as pressing control C or calling the kill command on a process will trigger this system call). A Goroutine is then started that listens on the channel — if a termination signal is observed the ChainLinkApplication will call a method that will allow it to exit gracefully. Afterwards, the method calls the Store, HeadTracker, and Scheduler Start methods. Store Start method: The Store Start method sets the first unlocked keystore account found as the current active account in the TxManager. The TxManager is responsible for interfacing with the Ethereum blockchain (sending signed transactions, bumping the amount of gas on a current transaction, etc). In addition to setting the current active account, the Store Start method sets the current active account nonce(the number of transactions from an account). The nonce is necessary to send transactions from the current active account. HeadTracker Start method: Headtracker subscribes to new blocks on the Ethereum blockchain. When there is a new block on the Ethereum blockchain, the BlockHeader object(data structure that represents a block header on the Ethereum blockchain) will be piped into a channel called headers. The Headtracker connect method will query all log initiated JobSpecs(RunLog and Ethlog) from the local database — for each JobSpec the NewRPCLogSubscription method will be called. This method will create a subscription to new logs — new logs will be piped to the sub.logs channel. The Goroutine listenToLogs will backfill logs and handle new log entries by listening on the sub.logs channel. Backfilling logs, is an important feature of Chainlink because this means that if the Chainlink node goes down/exits, it will be able to handle logs/data requests that happened while the Chainlink node was down. “Handling logs” means validating that the log is of the correct type — in RunLog  validation that would be that the log has the same signature as the one emitted in Oracle.sol, the job id in the log matches the job id of the Runlog instance processing the log, and that the request meets a minimum amount of Link. If validation passes, a job run starts. What happens when a job run starts? Example JobSpecs can be found here. Chainlink runs each task in a JobSpec sequentially, saving the result after every successful task run. Each task run maps to the run of a specific adapter.Chainlink defines the RunResult data struct which holds the result of the current TaskRun. The result of the each TaskRun is set in the Data field of the RunResult, and is sent to the current TaskRun. This is done so that the current TaskRun can use the result of previous TaskRun run. Chainlink defines a minimum number of confirmations SLA that each task must meet before running (for log initiated JobSpecs). After x amount of confirmations have passed, Chainlink can be fairly sure that the block which started the job run is valid (if the block still exists). If a run does not meet the minimum number of confirmations then it is paused and put into the pending confirmations state. One of the most important adapters is called EthTx. This is the adapter that is responsible for sending off chain data to an on chain smart contract. In a RunLog scenario, EthTx(if specified in JobSpec) will send the unique id of the request (read from the RunRequest log emitted in requestData function in Oracle.sol — internalId in example below), and the result of the Data variable to the fulfillData function located in Oracle.sol. Once this data is sent to the fulfillDataFunction, the function will place the unique id of the request into the callback map. The callback map will return the Callback data structure that contains the necessary information to send the data to the correct smart contract address and function. After getting the necessary info from the Callback data structure, the data is sent to the requesting smart contract! This is how requests are fulfilled! How are jobs stuck in pending confirmations state re-started? Going back to the HeadTracker Start method, the final thing the method will do is start a Goroutine called listenToNewHeads which will listen on the headers channel. Chainlink needs to listen on new heads because it needs to save new heads into its database so that it knows the last processed head. In case the node goes down, and then starts up, it will know the last processed head. In addition, this Goroutine will have Chainlink process all of the pending job runs that did not meet the minimum block confirmation SLA when it previously ran. The Chainlink node initiates another job run if the job run meets its minimum block confirmation SLA. Scheduler Start method The Scheduler Start method will make sure that Cron and RunAt JobSpecs run at a specified schedule. Cron : “The Cron initiator is a simple way to schedule recurring job runs, using standard cron syntax with an extra field for specifying second level granularity.” RunAt: “The RunAt initiator triggers a one off job run at the time specified.” For Cron jobs, Chainlink uses this library. The main functionality of the library can be seen in the AddFunc and run methods. In the Scheduler Start method, one of the first actions is calling the Cron library Start method, which will start the Cron library run method in a GoRoutine. Every Cron JobSpec will be added to the Cron library using the AddJob method, which will call the Cron library AddFunc method. The AddFunc method takes a function and its recurring schedule as inputs. The Cron library will call the function that is passed into the AddFunc method at each recurring scheduled interval— in this case, the passed in function invokes the BeginRun method (begins a new job run). The AddFunc method call will end up encapsulating its input (function and its recurring schedule) inside an object called Entry. The Entry object will then be added to the Cron library Entries collection. After getting added to the Entries collection, the Entry object will get piped to the add channel. The cron add channel is used in the Cron run method. The run method iterates over all of the entries in the entries collection and finds the next scheduled time of each function. From these times, the run method finds the earliest scheduled time (if there is one). From the earliest scheduled time, the method will create a timer object (called timer) that will pipe into a channel(called C) when the timer expires. The function then enters a select statement that allows it to wait on multiple channels.The first channel will be the the timer channel of the next scheduled function run(discussed above). Another channel will be the the add channel that I talked about in the AddFunc method(piped when a new cron is added). The next channel is the snapshot channel, which will be piped when someone requests a “snapshot” of all Entries currently present in the library. The last channel will be the close channel, which will be called when the Scheduler/Cron library is requested to stop. When the timer object channel is piped, all of the entries that have their “next” run time less than the current time, will have their functions executed. After executing, the entry/entries will be given a new next run time, sorted based on run time, and the select statement will begin listening on all channels again. When the add channel is piped(from the AddFunc method) the method will stop the current timer (since the added function could have an earlier run time than the current earliest run time). Afterwards, a new timer object is created from the earliest run time, and the select statement begins listening on all of the channels once again. The snapshot channel is piped when there is a request for all cron Entries. The close channel will be piped will be called when the Scheduler/Cron library is requested to stop — it ends the timer object and returns. RunAt functions are very similar to Cron jobs, except that they run only once. For each RunAt JobSpec , Chainlink will start a new Gorutine that will wait on two channels using the select statement(just like in the Cron library). The first channel will be piped when Chainlink wants to gracefully exit. The second channel will be piped when the RunAt JobSpec is slated to run. When the second channel is piped, Chainlink will execute the run that is formatted in the JobSpec by invoking the BeginRun method.