In my previous MinIO guides I have gone through setting up MinIO, enabling an external credential provider, adding replication, and using Prometheus for advanced reporting. All this was done with a publicly accessible MinIO web interface. Having gone through all this however, I then decided to put MinIO behind a Tailscale VPN to further secure my data.

I’ve recently begun using Tailscale for my remote access, and given every system I use with MinIO is part of my Tailscale, I figured that leveraging the VPN could only improve my security.

Some assumptions I am making going in:

1. You have followed guides 1. Guides 2-4 are optional.
2. When configuring any replication or access, a hostname was used and not IP addresses. If IP addresses were used, some reconfiguration may be needed.We will continue to use our existing hostnames for access and replication, not hostnames that Tailscale provides us.
3. Access to the MinIO servers via ports 80 and 443 will be restricted at the end of this.
4. We’ll continue to use an identity provider for login, even with the VPN in place, if previously configured using Guide 2.
5. My guide assumes Cloudflare is used for DNS and leverages a plugin for certificate generation. Other DNS providers may be supported.

Change Certs to use DNS-01 Challenge

Your MinIO server will no longer be publicly accessible, so previously configured cert issuance with Certbot using LetsEncrypt will cease to work. What we configured in Guide 1 had a reliance on a port 443 check for a certificate to be issued. To get around this we will use a DNS-01 challenge leveraging an API which allows DNS validation from a non public facing server. In my case I am using Cloudflare. Different plugins will be needed for other DNS providers, and not all DNS providers may be supported. If not, look at other options to generate a valid SSL certificate for your server.

If using Cloudflare, generate a Cloudflare API Token

1. Log into your Cloudflare account
2. Load you Profile
3. Select API Tokens
4. On the API Tokens page, click the Create Token button
5. Use the ‘Edit zone DNS’ template
6. Under Zone Resources select ‘Include’ and pick your MinIO’s domain
7. Optionally configure IP Filtering by adding your servers IP address and filtering on ‘Is In’ as the operator. If filtering on IP, it may be necessary to have both the IPv4 and IPv6 server address set.
8. Create the token.
9. Note the new API token, you will need it soon.

Next we will configure the Cloudflare Certbot plugin to use the Cloudflare API token.

Start by installing the Cloudflare Certbot plugin.

apt install certbot python3-certbot-dns-cloudflare

Create a certbot configuration file

mkdir /etc/certbot
nano /etc/certbot/credentials

Add a single line with your API Token, replacing with your own token.

dns_cloudflare_api_token = abc123

Finally update the files permissions.

chmod 600 /etc/certbot/credentials

Now update existing SSL certificates to use the API key. Run the following for any certs you have configured previously, swapping in your address. You’ll probably have a cert for the MinIO S3 service and the web interface, at minimum.

sudo certbot certonly -d minioconsole-01.mysite.com --elliptic-curve=secp384r1 --dns-cloudflare --dns-cloudflare-credentials /etc/certbot/credentials --dns-cloudflare-propagation-seconds 30

This will update the cert generation to use the API key and remove the need for access to be verified over port 443. The longer propagation time of 30 seconds, over a default 10 seconds, helps with ensuring the processes completes on time. Without this I had timeouts.

Lastly confirm that this succeeded and that a schedule task is setup.

certbot renew --dry-run

Install Tailscale

Install Tailscale and ensure it can connect to your Tailscale network. Rather than repeat the steps, refer to the comprehensive list of guides that Tailscale.com publishes. Ensure you connect once and establish an IP address for the server.

Update DNS

Change your hostnames DNS records to Tailscale internal IPs. You can get these from the devices section of your Tailscale account or from a client. Hence forth your hostname will resolve to an IP address on your VPN network, that only you can access.

Configure UWF – Close Port 80/443

The next step is to configure the firewall. The aim is to deny access to 443/80 from external traffic, while still allowing traffic over your VPN.

The first step is to ensure that the UFW firewall will not be bypassed by our docker contains. By default Docker will bypass UFW. To fix this update the firewall rules as per this Github project.

Open the UFW firewall rules file

nano /etc/ufw/after.rules

Add the following at the end of the file

# BEGIN UFW AND DOCKER
*filter
:ufw-user-forward - [0:0]
:ufw-docker-logging-deny - [0:0]
:DOCKER-USER - [0:0]
-A DOCKER-USER -j ufw-user-forward

-A DOCKER-USER -j RETURN -s 10.0.0.0/8
-A DOCKER-USER -j RETURN -s 172.16.0.0/12
-A DOCKER-USER -j RETURN -s 192.168.0.0/16

-A DOCKER-USER -p udp -m udp --sport 53 --dport 1024:65535 -j RETURN

-A DOCKER-USER -j ufw-docker-logging-deny -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 192.168.0.0/16
-A DOCKER-USER -j ufw-docker-logging-deny -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 10.0.0.0/8
-A DOCKER-USER -j ufw-docker-logging-deny -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 172.16.0.0/12
-A DOCKER-USER -j ufw-docker-logging-deny -p udp -m udp --dport 0:32767 -d 192.168.0.0/16
-A DOCKER-USER -j ufw-docker-logging-deny -p udp -m udp --dport 0:32767 -d 10.0.0.0/8
-A DOCKER-USER -j ufw-docker-logging-deny -p udp -m udp --dport 0:32767 -d 172.16.0.0/12

-A DOCKER-USER -j RETURN

-A ufw-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] "
-A ufw-docker-logging-deny -j DROP

COMMIT
# END UFW AND DOCKER

Restart UFW.

sudo ufw reload

UFW will now apply to Docker ports. Now go through and remove any firewall rules allowing inbound traffic on ports 443 or 80 from public addresses. To do this first run the following:

ufw status numbered

Then delete the rule with

ufw delete 2

As rules will be re-numbered when those below them are deleted, be very careful to recheck the rule numbers each time you delete a rule. Alternatively, delete from highest number to lowest. You won’t want to accidentally delete any other required rules. In my case I only had SSH rules open when I was done with this.

From here add a rule now to allow HTTPS traffic for Tailscale traffic only.

ufw allow proto tcp from 172.16.0.0/12 to 100.64.0.0/10 port 443

172.16.0.0/12 is the default Docker IP range. 100.64.0.0/10 is the default Tailscale IP range for devices. Adjust those IPs as necessary.

Restart the firewall one more time, just to be sure everything is applied.

sudo ufw reload

Test Everything is Working

Having updated SSL and the firewalls, everything should now be done. From a PC on the same Tailscale network, test with the VPN active that you can connect to the MinIO console. Now, turn off Tailscale and observe when your client is not connected to a VPN, connectivity will fail. I use MinIO for file backup and additionally tested that backups work, but only while on the VPN.

Log into MinIO. The Google Credential Provider (if you followed previous guides) should also still work OK.

If you are using site to site replication, this will also work however you will need to repeat this guide for each server. As per the assumptions, if MinIO is using hostnames for replication, all data should replicate cleanly as soon as you have each nodes DNS, SSL and firewalls updated. Just ensure your server uses the updated DNS records that you configured in CloudFlare so it can pickup the Tailscale IP addresses.

Conclusion

That’s it. Now your MinIO instance is running over a Tailscale VPN and is no longer publicly exposed. If your use of MinIO is from PCs you manage, this may be worth considering. Also, much of what was done can probably be applied to WireGuard or other VPN options if you wish.