Goaccess

This article is written for intermiediate level techies interested in privacy centric web analytics.

Until now, I’ve been using Matomo (formerly Piwik) or Open Web Analytics (OWA) for privacy centric web analytics. One of the challenges I’ve had, running dozens of websites, is that the analytics serve shows up as a third party privacy tracker to work effectively, and many people now use third party privacy tracker blockers in their privacy-centric browser setups. I needed a solution that doesn’t miss out on visitors with such a setup.

This is when I discovered an awesome new solution called goaccess. First of all, it works in both terminal as well as can be rendered in real-time HTML to be viewable in your browser.

In order to get started with my steps, you should install goaccess on the machine you want to view the analytics on, which can be your laptop or desktop.

$ apt install goaccess # on debian

or in a terminal on osx using brew:

$ brew install goaccess 

The other requirement which is too much for this article, is to setup an ssh config file, so you can connect to your webserver with a simple “ssh server”, typically done with a passwordless key (PKI) and known_hosts on the remote machine. An example in ~/.ssh/config might be:

 Host server
     User myusername
     Hostname myserver.example.com
     IdentityFile ~/.ssh/id_rsa

There are two ways of handling this now. The first one is to scp your webserver logs to your machine:

$ rsync -avrt server:/home/myusername/logs/myserver.example.com/https/ myserver.example.com/

and then to view all the logs:

$ zcat /var/log/nginx/access.log.{1..15}.gz | ./goccess /var/log/nginx/access.log - 

or to view just the latest access log in your browser:

$ goaccess myserver.example.com/access.log* -a > myserver.example.com.html

and now you can open myserver.exmaple.com.html in your browser.

This was great, but I also wanted a real-time solution to monitor my logs securely. In terminal:

$ ssh server "tail -f /home/myusername/logs/example.com/https/*.log"  | goaccess -

or to view in a browser:

$ ssh server "tail -f /home/myusername/logs/example.com/https/*.log"  | goaccess -o report.html --real-time-html -

That’s it for today.

The last thing I’d like to solve for is to monitor multiple servers at once. I came across multitail which had me excited, and it s great tool for other purposes, but it doesn’t correctly emulate tail – both adding ncurses content, but also feeding the initial logs as a single blob instead of line-by-line. I emailed the author asking, “is there a commandline argument to make it line-based (so it works like tail)?” and got back a “nope”, so back to the drawing board.