How to Monitor Your Avail Validator Node
Telemetry
Maintaining a healthy, stable, and secure ecosystem involves closely monitoring network operations and performance. In the case of Avail nodes, they come equipped with built-in low-level telemetry components that automatically gather and transmit detailed information about each node's operation within the network. This backend telemetry data is streamed in real-time to a designated endpoint, allowing you to observe information regarding your nodes behavior.
Avail project has made available a public telemetry (opens in a new tab) for anyone to view the status of their nodes. This should not replace your own monitoring and alerts. However it is a key tool that anyone can use to view the status of their nodes at anytime.
Avail telemetry has been added to the chain specification file and should automatically send telemetry data to Avail public telemetry (opens in a new tab). You can however also
add the telemetry --telemetry-url 'ws://telemetry.avail.so:8001/submit/ 0'
to your avail node start command if your node is not visible on the public telemetry.
You can also use --name <Validator Node Name>
to set the name that will display on telemetry. Without this it will display a random node name on the telemetry.
For example:
./data-avail --validator \
--port 30333 \
--base-path `pwd`/data \
--chain `pwd`/chainspec.raw.json \
--name AvailNode \
--telemetry-url 'ws://telemetry.avail.tools:8001/submit/ 0'
Monitoring with Prometheus & Grafana
Prometheus serves as a monitoring platform that gathers metrics from monitored targets through scraping their metrics HTTP endpoints. In contrast, Grafana acts as a visualization dashboard specifically designed to present the collected data in an easily understandable and visually appealing manner.
The below guide is for setting up a local Prometheus & Grafana instance on the same server as your Avail node. If you're already experienced with Prometheus & Grafana and want to connect your Avail node to an existing Prometheus installation on a different server, include the --prometheus-external
flag in your Avail node's startup command.
Install Prometheus
Install prometheus and prometheus node exporter.
sudo apt-get install -y prometheus prometheus-node-exporter
Create the prometheus.yml config file
cat > $HOME/prometheus.yml << EOF
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
# - "first.rules"
# - "second.rules"
scrape_configs:
- job_name: "prometheus"
scrape_interval: 5s
static_configs:
- targets: ["localhost:9090"]
- job_name: "avail_node"
scrape_interval: 5s
static_configs:
- targets: ["localhost:9615"]
- job_name: node
static_configs:
- targets: ['localhost:9100']
EOF
Move prometheus.yml
to the correct location
sudo mv $HOME/prometheus.yml /etc/prometheus/prometheus.yml
Update the file permissions
sudo chmod 644 /etc/prometheus/prometheus.yml
Ensure Prometheus starts automatically
sudo systemctl enable prometheus.service prometheus-node-exporter.service
Restart Prometheus to activate latest settings
sudo systemctl restart prometheus.service prometheus-node-exporter.service
Check the status, ensure Prometheus has started without errors
sudo systemctl status prometheus.service prometheus-node-exporter.service
Install Grafana
Install Grafana
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" > grafana.list
sudo mv grafana.list /etc/apt/sources.list.d/grafana.list
sudo apt-get update && sudo apt-get install -y grafana
Ensure Grafana starts automatically
sudo systemctl enable grafana-server.service
Start Grafana
sudo systemctl start grafana-server.service
Check the status, ensure Grafana has started without errors
sudo systemctl status grafana-server.service
Setup Grafana Dashboard
Ensure port 3000 is open, example of adding to ubuntu firewall
sudo ufw allow 3000/tcp
In your browser navigate to http://<your validators ip address>:3000
. The default login username and password is admin/admin
You will be asked to reset your password, please write it down or remember the password as you will need it for the next login.
You will need to create a datasource. Navigate to Home->Connections->Data sources
Click on Add data source
Click on Prometheus
Set URL to "localhost:9090", then test and save the connection
Navigate back to your home page, on the top right in the menu select Import dashboard
Import the Avail Node Metrics file
You will have a new dashboard that opens and that you can use to monitor your node