Monitoring Tor .onion Websites (uptime alerts)

By Michael Altfield, on March 12th, 2021

This article will present a few simple website availability monitoring solutions for tor onion services.

Problem

So you’ve just setup an Onion Service for your website, but how often do you actually check that it’s working? Maybe it’s a .onion alias to an existing website, and you usually only check it on the clearnet. What’s to prevent the darknet presence of your website from going down for weeks without you noticing?

Indeed, it’s important to monitor your .onion websites so that you can discover and fix issues before your customers do. But how? Most of the popular uptime monitoring solutions (pingdom, freshping, statuscake, etc) certainly can’t monitor .onion websites.

This guide will enumerate some solutions for monitoring .onion websites, so you get an email alert if your site goes down.

Assumptions

This guide was written in 2021. The commands used should work on Debian 10, but may need modifications for other GNU/Linux systems.

Solutions

This section will list several ways to monitor an onion service’s uptime with varying complexity and features.

Solution #1: `curl` + cron

Our most simple solution: just call curl from a cron job. Unless you’re using TAILS or Whonix, you probably need to install tor, curl, and the package for the mail command — which can be done with the following commands

sudo apt-get install tor curl mailutils

Now, create a cron job to run every minute. Note that we use the --socks5-hostname argument to tell curl to use tor, else it wouldn’t be able to reach a .onion domain.

cat > /etc/cron.d/tor-hidden-service-monitoring <<'EOF'
* * * * * root url='http://qrmfuxwgyzk5jdjz.onion/'; curl -IL --socks5-hostname 127.0.0.1:9050 "${url}" || echo "Unable to access ${url}" | mail -s "ALERT: hidden service inaccessible" youremail@example.com
EOF

Now, if the website http://qrmfuxwgyzk5jdjz.onion/ goes down, you’ll immediately receive an email alert to youremail@example.com to notify you of the outage.

Solution #2: mon

The biggest issue with the above-listed “curl + cron” solution is intermittent or false-positive alerts. To prevent this, most enterprise uptime alerting monitoring services solve this by requiring an alert to remain in a critical state for multiple consecutive runs before the alert is triggered.

Rather than reinventing the wheel by adding a persistence state to the “curl + cron” solution, we can use the linux mon tool — which is a wonderfully simple-yet-powerful, general-purpose monitoring system for *nix.

First, install the depends. In this case we also need torsocks to torify the built-in http.monitor.

sudo apt-get install tor torsocks mon

Now, create a new monitor that wraps mon‘s built-in http.monitor with torsocks, so that it can reach .onion addresses

cat > /usr/lib/mon/mon.d/torsocks_http.monitor <<'EOF'
#!/bin/bash
################################################################################
# File:    torsocks_http.monitor
# Version: 0.1
# Purpose: Wraps the http.monitor script with torsocks to monitor .onion sites
#  https://tech.michaelaltfield.net/monitoring-tor-onion-websites-uptime-alerts
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2021-03-12
# Updated: 2021-03-12
################################################################################

export DIR_PATH=`dirname ${0}`
exec /usr/bin/torsocks --isolate ${DIR_PATH}/http.monitor -t 60 "$@"

EOF

A couple notes about the above script:

We use the --isolate argument, which tells torsocks to use stream isolation on every execution
We override the default timeout of 30 seconds to 60 seconds

Next, edit the main mon.cf file by adding the following lines to /etc/mon/mon.cf

hostgroup torproject_onion qrmfuxwgyzk5jdjz.onion

watch torproject_onion
        service http
                interval 5m
                monitor torsocks_http.monitor
                period 
                        alertafter 10
                        alertevery 1h strict
                        alert mail.alert youremail@example.com
                        alert restartTor.sh
                        upalert mail.alert youremail@example.com

The above block is pretty self-explanatory. It creates a hostgroup named “torproject_onion” with the domain name = qrmfuxwgyzk5jdjz.onion. The following block says to watch this hostgroup every 5 minutes using the torsocks_http.monitor script that we created above (which just wraps the built-in http.monitor perl script).

The period section is where things get interesting. First of all, we tell it to only alert once every hour, and only after 10 consecutive failures. If that happens, then it triggers two “alerts”:

An email is sent to youremail@example.com via /usr/lib/mon/alert.d/mail.alert and
It executes /usr/lib/mon/alert.d/restartTor.sh

Obviously the last one is a hack, but I found tor needs a restart once in a while. This is also why I’ve set the alertafter so high, as I don’t want restart tor unless it’s really necessary. The restartTor.sh script itself is very simple. Create it with the following commands.

cat > /usr/lib/mon/alert.d/restartTor.sh <<'EOF'
#!/bin/bash
################################################################################
# File:    restartTor.sh
# Version: 0.2
# Purpose: Restarts tor service
#  https://tech.michaelaltfield.net/monitoring-tor-onion-websites-uptime-alerts
# Authors: Michael Altfield <michael@michaelaltfield.net>
# Created: 2021-03-12
# Updated: 2022-10-14
################################################################################

LOG_PATH='/var/log/mon/restartTor.log'

msg="$(date -u --iso-8601=seconds) INFO: Restarting Tor"
echo ${msg} >> ${LOG_PATH}
 
sudo systemctl restart tor.service
echo -e "\texit code: ${?}" >> ${LOG_PATH}

EOF

And give the mon user permission to restart the tor service as root with sudo

sudo echo "mon ALL=(root) NOPASSWD: /usr/bin/systemctl restart tor.service" > /etc/sudoers.d/mon
sudo chown root:root /etc/sudoers.d/mon
sudo chmod 0400 /etc/sudoers.d/mon

Finally, to apply all the configuration changes above, give the mon service a restart.

systemctl restart mon.service

You can monitor the execution of mon by tail‘ing its history.log and dtlog files. You may need to wait up to 5 minutes.

tail -f /var/log/mon/history.log /var/log/mon/dtlog

And you can manually test the torsocks_http.monitor with the following command

root@host:~# time /usr/lib/mon/mon.d/torsocks_http.monitor qrmfuxwgyzk5jdjz.onion

HOST qrmfuxwgyzk5jdjz.onion: ok
HTTP/1.1 200 OK
Date: Thu, 28 Jan 2021 15:17:25 GMT
Server: Apache
Content-Location: index.html.en
Vary: negotiate,accept-language,Accept-Encoding
TCN: choice
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-Xss-Protection: 1
Referrer-Policy: no-referrer
Content-Security-Policy: default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline';
Last-Modified: Wed, 27 Jan 2021 15:57:29 GMT
ETag: "3cae-5b9e3d3ff6c40"
Accept-Ranges: bytes
Content-Length: 15534
Cache-Control: max-age=3600
Expires: Thu, 28 Jan 2021 16:17:25 GMT
Connection: close
Content-Type: text/html
Content-Language: en


real    0m1,489s
user    0m0,033s
sys     0m0,008s
root@host:~#

Solution #3: IaaS provider

Unfortunately, I don’t know of a single Uptime Monitoring IaaS provider that supports checks on .onion websites.

If you happen to know of one, then please link to it in the comments, and I’ll add it to this section.

Solution #4: Tor2Web proxy + IaaS provider

In the meantime, if you require integrating into one of your existing monitoring providers, another potential option is to setup a proxy to the darknet from the clearnet using Tor2Web or similar.

https://github.com/tor2web/Tor2web

Tor2Web was originally designed by Aaron Swartz and Virgil Griffith in 2008 to provide a means for users without tor to visit .onion websites from the clearnet — thus providing anonymity to the server, but not to the client.

This can be useful if, for example, you want to integrate your uptime history into your existing web infrastructure status page using a free service, such as:

To set this up, simply point your Website Uptime Monitoring solution-of-choice at your Tor2Web instance, which will make your darknet site available on the clearnet (just for testing availability). If your .onion site is down, then Tor2Web will also return the error to your Website Uptime monitor.

Solution #??

Do you have a clever way that you monitor your tor .onion websites? If so, please leave a note in the comments 🙂

I wrote this article because there’s very little support for monitoring of .onion websites’ in traditional uptime monitoring tools, and there’s very little information about alternative solutions on the ‘net.

If you maintain a .onion site for your org and are responsible for its availability, please document your tricks for the community and leave a comment below.