So You Think You Can Validate? NEAR Stake Wars III with AWS

Have you ever wanted to get into blockchain validation but never got the chance? Well, now’s a great time to dive in with NEAR and Stake Wars III. Stake Wars focuses on helping to educate users who otherwise would have never had the resources to get into validating. This comes along right as they’re about to release their Chunk-Only Producers which will drastically increase the accessibility of running a validator by reducing resourcing and upfront staking costs.

I’ve always wanted to try out validating but never had the chance until today! Stake Wars seemed like the right balance of free form and training-wheel-assisted programming that piqued my interest. You can find a full list of the challenges here, but today we’re just going to be focusing on #1-4. But before we get started, let’s get our server setup first.

Setup

I chose to use Amazon Web Services (AWS) as my cloud provider of choice. I haven’t used it in quite some time so it felt like a good time to get my hands dirty and re-learn a bit about the platform too. First, let’s remember that our producer will need to satisfy the following tech specifications:

And this will quickly guide us toward the type of EC2 instance that we’d want to use. In this case, a t3.xlarge with a 500 GB volume should do the trick.

From here make sure to create the correct security groups for your instance and that should be good enough to spin it up. And you should be able to SSH to it with your credentials on your local terminal. If you didn’t set a username, you can find the default one online for your corresponding OS and the host address can be found on the EC2 instance tab.

Create Your Shardnet Wallet

Challenge notes can be found here.

Install node:

curl -sL https://deb.nodesource.com/setup_18.x | sudo -E bash -
sudo apt install build-essential nodejs
PATH="$PATH"

Install Near CLI:

sudo npm install -g near-cli

echo 'export NEAR_ENV=shardnet' >> ~/.bashrc

Setup your validator

Install dependencies on your instance:

sudo apt install -y git binutils-dev libcurl4-openssl-dev zlib1g-dev libdw-dev libiberty-dev cmake gcc g++ python3-pip docker.io protobuf-compiler libssl-dev pkg-config clang llvm cargo

USER_BASE_BIN=$(python3 -m site --user-base)/bin
export PATH="$USER_BASE_BIN:$PATH"

Install build dependencies:

sudo apt install clang build-essential make

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

git clone https://github.com/near/nearcore
cd nearcore
git fetch

git checkout <commit>

Build the project

cargo build -p neard --release --features shardnet

./target/release/neard --home ~/.near init --chain-id shardnet --download-genesis

rm ~/.near/config.json
wget -O ~/.near/config.json https://s3-us-west-1.amazonaws.com/build.nearprotocol.com/nearcore-deploy/shardnet/config.json

Run your node:

cd nearcore
./target/release/neard --home ~/.near run

Your producer will start syncing and downloading blocks. This can take a bit of time so it might be a good time to set this up as a service in systemctrl.

First, we’ll start by setting up your near login on NEAR CLI:

near login

Go through the web flow, and eventually, you’ll hit this page and you can go back to your terminal:

You’ll need to create a validator_key.json file. In order to do so, first, you will:

near generate-key <pool_id>

<pool_id> —> xx.factory.shardnet.near WHERE xx is your pool name

Copy the file generated to shardnet folder: Make sure to replace <pool_id> by your accountId:

cp ~/.near-credentials/shardnet/YOUR_WALLET.json ~/.near/validator_key.json

Edit “account_id” => xx.factory.shardnet.near, where xx is your PoolName and Change private_key to secret_key

This will leave you with something like this:

{
"account_id": "xx.factory.shardnet.near",
"public_key": "ed25519:HeaBJ3xLgvZacQWmEctTeUqyfSU4SDEnEwckWxd92W2G",
"secret_key": "ed25519:****"
}

Now we’re ready to actually setup your systemctrl config:

sudo vi /etc/systemd/system/neard.service

Paste:

sudo systemctl enable neard

sudo systemctl start neard

View logs:

journalctl -n 100 -f -u neard

View logs in pretty print by installing this package:

sudo apt install ccze

View logs with colour:

journalctl -n 100 -f -u neard | ccze -A

Deploy your staking pool

Now we’ll need to actually create your staking pool and this can be done by running the following command:

near call factory.shardnet.near create_staking_pool '{"staking_pool_id": "<pool id>", "owner_id": "<accountId>", "stake_public_key": "<public key>", "reward_fee_fraction": {"numerator": 5, "denominator": 100}, "code_hash":"DD428g9eqLL8fWUxv8QSpVFzyHi1Qd16P8ephYCTmMSZ"}' --accountId="<accountId>" --amount=30 --gas=300000000000000

Such that the following parameters are replaced in the above line:

• Pool ID: Staking pool name, the factory automatically adds its name to this parameter, creating {pool_id}.{staking_pool_factory} If pool id is stakewars will create : stakewars.factory.shardnet.near

• Owner ID: The SHARDNET account (i.e. stakewares.shardnet.near) that will manage the staking pool.
• Public Key: The public key in your validator_key.json file.
• 5: The fee the pool will charge (e.g. in this case 5 over 100 is 5% of fees).
• Account Id: The SHARDNET account deploying and signing the mount tx. Usually the same as the Owner ID.

Lastly, you’ll want to ping your validator periodically, but we’ll handle this in the section below by utilizing a bit of automation.

Add Pool Monitoring And Automation

The two challenges I was trying to solve through automation are:

1. Ping the validator at least once per epoch in order to ensure that it’s online
2. Notify me when the validator is down so I can go fix it

Both of these can be solved by using crontab. Here’s a site I found that gave a lot of great info on the utility and would be a good read for users before diving in.

You can create/edit your config file with crontab -e and when you save the file, it’ll update immediately so it’s pretty easy to test. Here’s an example of my file:

NEAR_ENV=shardnet
0 */4 * * * ~/ping.sh
*/10 * * * * ~/heartbeat.sh

It’s pretty straightforward and only does 3 things:

1. Setup an env variable within the scope of the execution
2. Run ping.sh every 4 hours
3. Run heartbeat.sh every 10 min

More on the scripts themselves below but ultimately this is what triggers each execution. Technically I should probably only run the ping every 24 hours (epoch length) but since the Near is free there’s no real downside to more calls right now (You can also change the frequency to * * * * * to run every minute for easy testing).

Ping.sh

#!/bin/bash
echo "$(TZ=America/Los_Angeles date)" >> ping.log
near call ynot.factory.shardnet.near ping '{}' --accountId ynot.shardnet.near --gas=300000000000000 >> ping.log

1. Pings via cli to let the network know that your validator is up and running and then logs the results to disc so that I can see what’s going on at a later date
2. Also make sure to change the parameters to your account/pool names accordingly

Heartbeat.sh

#!/bin/bash
ACCOUNT_ID="ynot.factory.shardnet.near"
EMAIL_BODY="You should probably go fix it!"
EMAIL_SUBJECT="Your Near validator is BYEBYE"
EMAIL_ADDRESS="SOMEEMAILADDRESS@example.com"
IS_ALIVE=$(near validators current | grep $ACCOUNT_ID)
CURRENT_DATE=$(TZ=America/Los_Angeles date)
#SCRIPT
if [ "$IS_ALIVE" ]
then
echo "$CURRENT_DATE: Validator is alive" >> heartbeat.log
else
echo "$CURRENT_DATE: Validator is down" >> heartbeat.log
echo "$CURRENT_DATE: $EMAIL_BODY" | mail -s "$EMAIL_SUBJECT" $EMAIL_ADDRESS
fi

1. Checks to see if my node is in the list of validators and if it isn’t it’ll try to send me an email so I can hopefully fix it
2. Make sure to update your info/email accordingly
3. My “action” was to send an email but you can easily change it to a discord webhook or whatever you want. It actually ended up taking a lot of extra work to get emails working on EC2 so I’ll leave some comments below

AWS EC2 Email Setup Shenanigans

1. Your Gmail account will filter out anything sent from an EC2 instance since a lot of phishing/spam emails used to originate from these servers. I ended up using a Yahoo email address to circumvent this
2. By default, mail wasn’t available so I had to sudo apt install mailutils
3. You’ll also need to configure postfix as your mail server and this is pretty straightforward, just run it once and select internet as the type and it should auto-populate most things
4. You could just do sudo postfix start and call it a day, but I wanted a little bit more robustness so I wrapped it in a systemd service. If you went through the SW tutorials you did the same thing for the near validator already so this should be very familiar to you. I just wanted the system to restart the email server in the event that it died/rebooted/etc
5. Here’s my config at /etc/systemd/system/postfix.service
6. And you’ll want to systemctl enable and start the service just like you did for the neard one

[Unit]
Description=Postfix Daemon Service

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/home/ubuntu
ExecStart=sudo postfix start
Restart=on-failure
RestartSec=30
KillSignal=SIGINT
TimeoutStopSec=45
KillMode=mixed

[Install]
WantedBy=multi-user.target

To conclude this will mean that I’ll have the following 4 things running on my EC2 box:

1. systemctl wrapped service that is running the validator code
2. systemctl wrapped service that is running postfix (email server optional)
3. crontab that is running a ping every 4 hours
4. crontab that is running a health check and will email me if my validator is broken every 10 min

Costs

I haven’t used AWS in quite some time and knew that it could be quite pricy. But I wanted SW to be as close to a real-life validator tryout as possible and I knew that I’d want to go with one of the more trusted providers in the space. Costs are broken down in 3 ways:

1. EC2 run time in the form of $/instance hour
   • $0.1664 per
2. EBS storage in the form of $/GB-month
   • $0.08 per
3. Data transfer rate per GB above free tier
   • $0.090 per

Here’s an example of what my billing says so far:

Extrapolating on that we end up with:

24 * 30 * 0.1664 = $120 per month for instance hours
500 * 0.08 = $40 per month for EBS storage (guesstimate)
500 * 0.09 = $45 per month for data bandwidth (guesstimate)
Grand total = $205/month or $2460/year


Conclusion

There have definitely been some bumpy moments when it came to the team setting everything up and handling the influx of new users, but this has been quite a fun experience working on a validator as part of SW. The jury’s still out on whether or not this will be profitable long-term, but I think there are some improvements that can be made on my end to trim recurring costs. This will be easier to manage once I know how long I need the instances for so I guess we’ll see if I’m accepted as a mainnet validator before going down that path :).

Special thanks to the Stake Wars team for providing support and the marmaj DAO for showing me this awesome opportunity!