jonikorhonen.xyz

Running OpenBSD web server with httpd, relayd and Let’s Encrypt in a Digital Ocean droplet

Published Last updated

I’ve used Digital Ocean to host a Debian server with very bare bones and barren website for some years, but recently I took interest in OpenBSD and decided to take a look and soon wondered what it would take to run it as a Digital Ocean droplet. This writeup mostly serves as a reference for future me but might be of interest for others.

Setting up droplet and readying for install

Start with creating a new droplet, distribution doesn’t really matter. I went with Debian Stretch mostly out of habit. For a plan I chose the most basic one, which should be more than enough for hosting simple web server. It comes at $5 (+ VAT where applicaple) per month. Datacenter region is up to you to choose most convenient one. I went with Frankfurt 1 as it is geographically closest one. Rest of the options were left as is.

After a minute or so your new and shiny droplet is up and running, Hooray! Now turn it off in Power tab. Go to Networking tab and take note of public IP address, public gateway and subnet mask. Next go to Recovery tab and select Boot from Recovery ISO and turn power back on in the Power tab.

If web console window didn’t show up, open one by clicking Console in topright corner of droplet management section. Wait for recovery OS to boot up and once boot menu with options shows up select 6 to start a root shell. From there fetch the installer and prepare it for boot. At time of this exercise version 7.0 was the freshest one. Adjust as necessary.

wget https://cdn.openbsd.org/pub/OpenBSD/7.0/amd64/miniroot70.img

Write the installer to the first hard drive

dd if=miniroot70.img of=/dev/vda bs=512k

Go back droplet control panel to power off droplet, then set recovery option back to Boot from Hard Drive and power on the droplet. Reopen web console and you should see OpenBSD booting up.

Install OpenBSD

Now you have web console with some options. Select (I)nstall and follow installer prompts. Adjust values as needed.

  • Keyboard layout: Most appropriate for you
  • System hostname: What ever you want
  • Network interface: vio0
  • IPv4 address: Public IP of your droplet
  • Netmask: Subnet mask of your droplet
  • IPv6 address: none
  • Default IPv4 route: Public gateway of your droplet
  • Domain name: example.com
  • DNS nameservers: Pick your preferred poison
  • Password for root account: hunter2
  • Repeat root password: hunter2
  • Start sshd(8) by default: yes
  • Expect to run X Window System: no
  • Setup a user: <Your user name>
  • Full name: Just hit enter or give name
  • Password: hunter2
  • Retype password: hunter2
  • Timezone: UTC
  • Allow root ssh login: no
  • Root disk: sd0
  • Use disk: whole
  • Disk layout: a for auto or c for custom. Your choice.
  • Location of sets: http
  • Proxy?: none
  • HTTP Server: Hostname for mirror eg. cdn.openbsd.org. Hit ? for a list of available mirrors.
  • Server directory: Hit enter
  • Select file sets: -game* -x* and then done

Wait for the installer to fetch and install sets, for last two questions just hit enter. Wait while kernel is built. Reboot.

Configuring

If everything went as it should now you can ssh in to your droplet with its public IP address or domain name if you’ve set up DNS. It’s recommended to read afterboot documentation man afterboot.

doas

Doas is OpenBSD analog for sudo in Linux world. Let’s configure doas for group wheel

vim /etc/doas.conf
permit persist keepenv :wheel

User created in install is already a member.

Essential tools

As this is server vim should be no X11 flavor, git is nice to have and with rsync we can copy our website to server

doas pkg_add vim--no_x11 git rsync

Setup rsync

Rsync needs to know uid and gid of our user to set correct owner for files

$ id
uid=1000(<Our user>) gid=1000(<Our user>) groups=1000(<Our user>), 0(wheel)

$ group info daemon
name    daemon
passwd  *
gid     1
members daemon
doas vim /etc/rsyncd.conf
[global]
use chroot = yes
max connection = 5
log file = /var/log/rsyncd.log

[web]
path = /var/www/htdocs/example.com
read only = false
list = yes
uid = 1000
gid = 1

Enable and start rsync daemon

doas rcctl enable rsyncd
doas rcctl start rsyncd

And finally try to copy files over to server

rsync -a -P --delete ./website/ <Our user>@example.com:/var/www/htdocs/example.com

This copies files from ./website/ (trailing slash!) in source system to /var/www/htdocs/example.com in target server. Only files with newer timestamp in source than target are copied over

  • -a rsync archive mode
  • -P don’t set directory timestamps
  • --delete delete files from target server not found from source

Let’s Encrypt

Let’s Encrypt provides SSL/TLS certificates for websites, for free, and in a user-friendly way. Let’s set it up.

Set up Let’s Encrypt account

Let’s encrypt needs an account key for verification of domains and to request the signed certificate. We’ll want to back up the account key, as it allows managing and revoking certificates. It’s also needed to set up acme-client later.

Locally (Yes, I do infact use Arch):

pacman -S certbot
mkdir -p certbot/{cfg,letsencrypt,logs}
certbot register --config-dir certbot/cfg --work-dir certbot/letsencrypt --logs-dir certbot/logs

This creates a JSON formatted private key under letsencrypt/cfg/accounts/.../private_key.json. Examples to get an RSA formatted private key out of the json file in Java and Go can be found from here. Some finagling was required :>.

Now that account key creation business is out of way save it to safe location and copy it to server to /etc/acme/letsencrypt.pem

Acme client configuration

Configure acme client

doas vim /etc/acme-client.conf
api_url="https://acme-v02.api.letsencrypt.org/directory"
authority letsencrypt {
    api url $api_url
    account key "/etc/acme/letsencrypt.pem"
}
domain example.com {
    alternative names { www.example.com }
    domain key "/etc/ssl/private/example.com.key"
    domain full chain certificate "/etc/ssl/example.com.crt"
    sign with letsencrypt
}

Configure httpd to respond to the challenge

doas vim /etc/httpd.conf
prefork 5

ext_ip="*"

server "example.com" {
    alias "www.example.com"
    listen on $ext_ip port 80

    location "/.well-known/acme-challenge/*" {
        root "/acme"
        request strip 2
        directory no auto index
    }
}

Enable and start httpd

doas rcctl enable httpd
doas rcctl start httpd

Try to get the certificate

doas acme-client -v example.com

If everything worked command should terminate with

acme-client: /etc/ssl/example.com.crt: created

Utilize periodic system maintenance user script to run a daily job for renewing the certificate

doas vim /etc/daily.local
#!/bin/sh

# -v for verbose output for logging in /var/log/daily.out
acme-client -v example.com
# Reload relayd to make it use reneved certificate.
# Added pre-emptivily here as we setup relayd to terminate TLS
# connections in later step.
rcctl reload relayd

and make it executable

doas chmod +x /etc/daily.local

Let’s Encrypt certificates are valid for 90 days, so renewing won’t do anything until certificate is closer to expiration.

Httpd

As we are using relayd for reverse proxying and terminating TLS connection, httpd configuration for our website is rather simplistic. Update httpd configuration with

doas vim /etc/httpd.conf
prefork 5

ext_ip="*"

server "example.com" {
    alias "www.example.com"
    listen on $ext_ip port 80

    ... Let's Encrypt challenge unchanged

    location * {
        block return 301 "https://$SERVER_NAME$REQUEST_URI"
    }
}

server "example.com" {
    alias "www.example.com"
    listen on $ext_ip port 8080 # relayd reverse proxies to this port

    log style forwarded # log connection information from X-Forwarded-* headers

    root "/htdocs/example.com"
}

types {
    include "/usr/share/misc/mime.types"
}

Test configuration with

doas httpd -n

and if that worked, restart httpd.

doas rcctl restart httpd

Relayd

We’ll use relayd as reverse proxy to terminate connections and set request/response headers. Start by configuring

doas vim /etc/relayd.conf
log state changes
log connection errors
prefork 5

table <httpd> { 127.0.0.1 }

http protocol "wwwsecure" {
    tls keypair "example.com"

    # Return HTTP/HTML error pages to client
    return error
    # Depending on use case, this might be needed
    #match request header set "Connection" value "close"

    # X-Forwarded headers
    match request header set "X-Forwarded-For" value "$REMOTE_ADDR"
    match request header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"

    # Set best practice security headers https://securityheaders.com to check and modify as needed
    match response header remove "Server"
    match response header append "Strict-Transport-Security" value "max-age=31536000; includeSubDomains"
    match response header append "X-Frame-Options" value "SAMEORIGIN"
    match response header append "X-XSS-Protection" value "1; mode=block"
    match response header append "X-Content-Type-Options" value "nosniff"
    match response header append "Referrer-Policy" value "strict-origin"
    match response header append "Content-Security-Policy" value "default-src https:; \
        style-src 'self' 'unsafe-inline'; \
        font-src 'self' data:; \
        script-src 'self' 'unsafe-inline' 'unsafe-eval';"
    match response header append "Permissions-Policy" value "accelerometer=(none), camera=(none), \
        geolocation=(none), gyroscope=(none), microphone=(none), payment=(none), usb=(none)"

    # Set recommended tcp options
    tcp { nodelay, sack, socket buffer 65536, backlog 100 }

    pass request guick header "Host" value "example.com" forward to <httpd>
}

relay "wwwsecure" {
    listen on 0.0.0.0 port 443 tls
    protocol wwwsecure
    forward to <httpd> port 8080
}

relay "wwwsecure6" {
    listen on :: port 443 tls
    protocol wwwsecure
    forward to <httpd> port 8080
}

Enable and start relayd

doas rcctl enable relayd
doas rcctl -d start relayd

the d switch makes it easier to find problems in config as fat-fingering is almost inevitable.

Deploy your website

Now that all this is taken care of it’s time to deploy our website to /var/www/htdocs/example.com (see section about rsync earlier).

To test the setup, go to https://example.com and you should see your awesome website!

SSL Labs

Go to SSL Labs and perform SSL Server test. Our httpd, relayd and Let’s Encrypt setup should score A+ easily.

Afterthoughts

Setting up OpenBSD and services was mostly very effortless once I found some documentation for older versions on getting it running on VPS provider that doesn’t offer it natively. Configuring httpd and relayd was made much easier than I originally dreaded thanks to excellent man pages and resources on web from where I could piece together works for me™ setup.

Acknowledgements

In no particular order: OpenBSD project for OS and accompanying software; Let’s Encrypt for making TLS certificates availlable for any Jack or Jill; vinh, dre, Adyxax for existing writeups with steps making it easier for me to set up all this.

Shameless shilling

Get $100 in credits by joining Digital Ocean through referral link down below

DigitalOcean Referral Badge