Become Home Blog Resources Consulting

Sovran

Server Setup Basics

Enki
13 Aug 2024

This is a post I've been meaning to do for a while. While it's simple to explain
how to set up an app for self-hosting, it's pointless to host an app on a weak
foundation. It's a massive pain in my ass to start every how to with a section on
server setup, so I'm also making this post for myself as a reference on how I like
to set up a server for apps I'm hosting. I'll start with basic stuff like proper login
with SSH and non-root user set up and making users for each app. I'll also touch
on NGINX setup, some quality of life tools that make server management easier,
log management and basic network security.
‍
SSH
Users
Logs
Backups
Basic Network Safety
NGINX
Quality of Life Tools
DNS
Docker

SSH
First is login. You’ll need a way to access your device securely. Don't even mess
with username and password. You want to use SSH (Secure Shell) and make
sure that SSH is the only way to log in. To do that, you’ll need an SSH key and a
new user account. On a newly provisioned VPS, you'll be logged in as root, and
you want to protect the root account. First off on the VPS or remote machine
make a new regular user with and add them to the “sudo” group with:

sudo adduser newuser

sudo usermod -aG sudo newuser

Now on your local machine run:

ssh-keygen -t ed25519 -C "[email protected]"

Follow the instructions, it should ask you where you want to save the file and if
you want a password or not. Make sure you set a string one. To copy the public
key over to your server run on your local machine:

ssh-copy-id -i ~/.ssh/id_ed25519.pub newuser@your_server_ip

Keep in mind newuser@your-server-ip is the username and the remote device

you are trying to copy your public key into. When you get prompted for a
password, it will be the password for the account on the remote device, NOT the
password you just made for the SSH key. Once verified, it will copy over the
public key, and you can now log in Via SSH. To turn off username and password
login, type in:
‍

sudo nano /etc/ssh/sshd_config

Find these values and set them as you see them here.
‍

Port 2222 # Change default port (use a number between 1024 and 65535)
PermitRootLogin no # Disable root login
PasswordAuthentication no # Disable password authentication
PubkeyAuthentication yes # Enable public key authentication
AuthorizedKeysFile .ssh/authorized_keys # Specify authorized_keys file loca
AllowUsers newuser # Only allow specific users to login

This disallows every login method besides SSH under the user you copied your
public key to. Stops login as Root and only allows the user you specify to log in.
Hit CTL+S to save and CTL+x to get out of the file editor. Restart SSH:
‍

sudo service ssh restart

This might boot you out of the session. If it does, this is a good time to test the
other login methods to see if they are declined before continuing. Also, it should
go without saying, but you need to keep the private key safe and if you lose it
you will not be able to get in remotely anymore.You can further lock down your
login with:
‍

Protocol 2 # Use only SSH protocol version 2

MaxAuthTries 3 # Limit authentication attempts
ClientAliveInterval 300 # Client alive interval in seconds
ClientAliveCountMax 2 # Maximum client alive count

Now, let's dive into users a bit more and see how we can leverage them for a bit
of organization and security.

Users
Users are important when it comes to managing a Linux server. There is an idea
in server management called the “Principle of The Least Privilege” this basically
means that you want to give an app or process the minimum amount of
privileges that it needs to do its job. Root has unlimited power, and no app really
needs this. Making a user for apps that you're running accomplishes a few
things. It can limit potential damage if an application you are running is
compromised. It adds isolation when running more than one app, it helps with
auditing so you know what app is using what system resources.
In short, users are a great way of helping organize your system and helps you
troubleshoot if and when things go wrong. To add a new user, run:
‍

sudo useradd -rms /usr/sbin/nologin -c "a comment" youruser

This command makes a user and gives them a home directory for app data but
does not allow login as the user. The -c flag is optional, but It's nice to know what
the user is for, like “Running Nextcloud” or whatever. Clone app files into the /opt
directory with:
‍

sudo mkdir /opt/myapp

This command makes a user and gives them a home directory for app data but
does not allow login as the user. The -c flag is optional, but It's nice to know what
the user is for, like “Running Nextcloud” or whatever. Clone app files into the /opt
directory with:
‍

sudo chown appuser:appuser /opt/myapp

Ok, with this your login is locked down, and you should have a decent idea about
how to use users. Next is logs.
‍

Logs
Logs are crucial to system administration. They keep track of system health, help
troubleshoot issues and detect threats. So you want to set up proper log rotation
so they do not take up too much space on your system, plus are easier to read
and manage. To set up proper log rotation, you want to edit the logrotate.conf file
located in /etc. Individual application configurations are typically stored in
/etc/logrotate.d/, so an example configuration for NGINX would look like:
‍

/var/log/nginx/*.log {
weekly
missingok
rotate 52
compress
delaycompress
notifempty
create 0640 www-data adm
 sharedscripts
postrotate
[ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
endscript
}

This configuration rotates logs weekly, keeps 52 weeks of logs, compresses old
logs, makes new logs with the right permissions and then signals NGINX to
reopen log files after rotation. You can test it with:

sudo logrotate -d /etc/logrotate.conf

This will show what it will do without actually rotating logs. With this all set up,
you can start to do more advanced stuff like triggering alerts based on log
entries. Now this is good for a single server but if you manage more than one
server it's a good idea to look into tools like Grafana Loki, Graylog and Fluentd. I
won't go into detail here, but if you're looking to up your log game, these a decent
place to start.
‍

Backups
Backups, and more importantly, testing your backups, are extremely important in
server management. Remember: a backup is not a backup unless you test it.
Untested backups are essentially useless.

There are three main types of backups. Full, Differential, Incremental. Full
backups are a complete copy of all data on a disk. Takes the most resources, but
is the easiest to restore from. Differential backups back up all the changes since
the last full backup, it's a middle ground strategy for backups on both space and
restoration speed. An incremental backup backs up data that was changed since
the last backup, this is the fastest backup option but can be the most complex to
restore.

I think of it like this. I use incremental backups for things like photos and
documents or project files and folders that get edited a lot. I'll use a full backup
for backing up and entire server or disk. Differential backups Ill use for backing
up full folders like /etc, /opt and log folders.

Now what about storage? If you follow the 3-2-1 rule, you will be golden. 3 copies
of your data, 2 storage types, and 1 offsite backup. I'd say if this seems like too
much, the “offsite” storage is the most important and not one to skip. In case of a
catastrophic meltdown, having a hard disk with your backups is invaluable.
Offsite / offline backups can also save your ass from ransomware. So keep that
in mind. There is a huge amount of backup software out there. This link is for
exploring some more professional backup tools. This link has file sync, transfer
and could storage solutions. I use a combo of sync-thing, Borg backup and good
old-fashioned FTP.

Remember, that backup, logs and server monitoring is an evolving process

based on your needs. The specific strategy you implement should be tailored to
your needs and the criticality of your data.

Basic Network Safety

The next step in securing a server is to lock down ports that need don’t need to
be exposed to the internet and banning things that try to log in when they should
not. UFW and Fail2Ban are two tools that are in widespread use for this. They
are simple and easy to use, UFW lets you set traffic rules for ports and Fail2Ban
will ban and IP address when it knocks on a port they should not be or if they fail
to log in after some predefined rules. UFW or uncomplicated firewall often comes
preinstalled on a lot of VPS services, same with Fail2Ban, but if you are on a
new machine and you're unsure, run:
‍

sudo apt install ufw

sudo apt install fail2ban

UFW

We will worry about Fail2Ban later, for now let's focus on UFW setup. First run
some default policys with:

sudo ufw default deny incoming

sudo ufw allow outgoing

This is considered best practice, as it follows the “the least privileges” idea I
touched on earlier. It reduces attack surface on your machine and gives you
precise control over what you do expose. In short, this configuration creates a
balance between security and functionality. Your server can reach out to the
internet as needed, but external entities can only connect to your server in ways
you've explicitly allowed. Now let's allow some stuff in.
‍

sudo ufw allow ssh

sudo ufw allow 80
 sudo ufw allow 443

If you are going to be running a web server, you need port 80 and port 443 open.
80 is HTTP and 443 is HTTPS. By default, port 22 is SSH, if you changed this
you need to specify the port instead of using the “allow ssh” command. Here are
some other useful commands:
‍

#List rules with numbers:

sudo ufw status numbered
#Delete by number:
sudo ufw delete NUMBER
#Delete by rule specification:
sudo ufw delete allow 80
#You can allow connections from specific IP addresses:
sudo ufw allow from 192.168.1.100
#You can also only allow an IP to connect to a specfic port with:
sudo ufw allow from 192.168.1.100 to any port 22
#If you neeed to allow a range of ports:
sudo ufw allow 6000:6007/tcp
#To further protect from brut force attacks you can rate limit specific por
sudo ufw limit ssh
#This would limit port 22 to 6 connections in 30 seconds from a single IP.

#Adding this goves you more info

sudo ufw status verbose
#and to reset incase you need to start over:
sudo ufw reset
#and to enable and disable:
sudo ufw enable
sudo ufw disable

#finaly to enable logging and adjusting the log level:

sudo ufw logging on
sudo ufw logging medium # levels are low, medium, high, full

On to Fail2Ban now.

Fail2Ban
‍
The main configuration is located in /etc/fail2ban/jail.conf, but it's recommended
to create a local configuration file:
‍

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

 sudo nano /etc/fail2ban/jail.local

‍
There are some basic settings in the [DEFAULT] section of the jail.local section
those are:
‍

bantime = 10m
findtime = 10m
maxretry = 5

‍
Ban time is how long an IP is banned. Find time is the time frame in witch
Fail2Ban looks for repeated failure, and max retry is the number of failures
before an IP is banned. You can tune these as you see fit. There are also custom
jails you can set, Fail2Ban also supports jails for commonly used services like
SSH. There are even more steps you can take, but I think this covers the basics.

NGINX

There are a small mess of web servers out there that you can use. Apache,
Caddy, nginx, IIS to name a few. I use Nginx. It's what I know, and it works really
damn well. Nginx (pronounced engine-x) is a web server, reverse proxy, and load
balancer. As a web server, it excels at serving static content and can handle
loads of concurrent connections with fairly low resource usage. As a reverse
proxy, it can sit in front of your application servers and forward traffic to them
while enchaining the apps' security. Its load balancing aspects can effectively
balance traffic between servers, improving reliability and scalability.

When installed via apt, the default location for nginx is /etc/nginx/ the nginx.conf
is mostly used for global server configuration and includes filed from the
/etc/nginx/sites-enabled folder. This modular structure allows for easy
management of multiple sites. Two folders to be aware of are the sites-enabled
folder and the sites-available folders. You can think of the sites available as a
staging place to test your site configurations, while the sites enabled is for live
sites and apps. A common practice is to set up and test your configuration in the
sites in the sites available, then when you're ready to go live and get an SSL cert,
you link the file to the sites-enabled folder. You do that with:
‍

ln -s /etc/nginx/sites-available/yoursitefile /etc/nginx/sites-enabled

Then reload nginx and double check nginx status with:

‍

sudo systemctl reload nginx

 sudo systemctl status nginx

Your site should be live now.

Below, I’ll show you some boilerplate Nginx site configurations. Be sure to look
into your app or sites needs as these are just starting points. For static sites, this
is a decent starting point.

Basic Static Website Configuration:

server {
listen 80;
listen [::]:80;
server_name example.com www.example.com;
root /var/www/example.com/html;
index index.html index.htm;
location / {
try_files $uri $uri/ =404;
}
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
add_header Content-Security-Policy "default-src 'self' http: https: dat

# Logging
access_log /var/log/nginx/example.com.access.log;
error_log /var/log/nginx/example.com.error.log warn;

# SSL configuration (uncomment after running Certbot)

# listen 443 ssl http2;
# listen [::]:443 ssl http2;
# ssl_protocols TLSv1.2 TLSv1.3;
# ssl_prefer_server_ciphers on;
# ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256

# Certbot will add its own SSL certificate paths

# ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
# ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
}

Proxy Pass Configuration:

server {
listen 80;
listen [::]:80;
server_name app.example.com;
location / {
 proxy_pass http://localhost:3000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
add_header Content-Security-Policy "default-src 'self' http: https: dat

# Logging
access_log /var/log/nginx/app.example.com.access.log;
error_log /var/log/nginx/app.example.com.error.log warn;

# SSL configuration (uncomment after running Certbot)

# listen 443 ssl http2;
# listen [::]:443 ssl http2;
# ssl_protocols TLSv1.2 TLSv1.3;
# ssl_prefer_server_ciphers on;
# ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256

# Certbot will add its own SSL certificate paths

# ssl_certificate /etc/letsencrypt/live/app.example.com/fullchain.pem;
# ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem
}

WebSocket Upgrade Configuration:

server {
listen 80;
listen [::]:80;
server_name ws.example.com;
location / {
proxy_pass http://localhost:8080;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
 add_header Content-Security-Policy "default-src 'self' http: https: dat

# WebSocket timeout settings

proxy_read_timeout 300s;
proxy_send_timeout 300s;
# Logging
access_log /var/log/nginx/ws.example.com.access.log;
error_log /var/log/nginx/ws.example.com.error.log warn;

# SSL configuration (uncomment after running Certbot)

# listen 443 ssl http2;
# listen [::]:443 ssl http2;
# ssl_protocols TLSv1.2 TLSv1.3;
# ssl_prefer_server_ciphers on;
# ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256

# Certbot will add its own SSL certificate paths

# ssl_certificate /etc/letsencrypt/live/ws.example.com/fullchain.pem;
# ssl_certificate_key /etc/letsencrypt/live/ws.example.com/privkey.pem;
}

The basic configuration is for serving a simple static site. It specifies the domain
name, listens on port 80 for both IPv4 and IPv6, sets the root directory for the
site, configures error handling with try_files, adds some basic headers that
protect from common web vulnerabilities, sets up logging for access and errors
and includes a section for SSL that is commented out. Most of the SSL config will
be handled by certbot, but there are a few lines in there that add some SSL
security that can be uncommented after certbot is ran.
‍
The proxy pass configuration is similar to the basic configuration, but instead of
serving files directly, it proxies requests to a local application (in this case,
running on port 3000).

The third configuration file is geared towards apps that need website
connections, it's a lot like the proxy pass configuration with some changes to
allow web sockets.

Ok, any bit about web servers is not really complete without talking about SSL.
For casual use, certbot is a pleb's best friend. It's free, it is fast, and it fucking
works. I use the python version of certbot. You can install that with:
‍

sudo apt install certbot python3-certbot-nginx

Once it's installed you can simply run “certbot” in your terminal, this will detect
the configs in your sites-enabled folder and ask what you want to do (renew,
reissue, etc…). Follow the walk-through, certbot gives you It's pretty straight
forward.
‍
So nowadays certbot when getting a new cert will set up auto-renew for you, so
it's a sit-and-forget kinda task. But to make sure it worked you can run:

sudo systemctl status certbot.timer

if this is up and running, you should be good to go if you're using systemd.

‍

Quality Of Life Tools

On the topic of tools that make managing your system easier, I'm going to
present some tools I use on my servers that I think make management just a bit
nicer. Not going to do a deep dive on any tool. All of these are optional and in no
particular order. A lot of these I found on the site terminal trove, a great site to
browse if you're a terminal junkie like me.

First tool, Btop this is in my personal must haves list. Btop is a terminal monitor
of resources. It shows you real time visuals of usage stats for your box’s CPU,
RAM, disks, network and running possesses it's written in C++ and can be
installed via most package managers.

For servers that have a lot of outside connections, i.e. a nostr relay, a tool like
Neoss is helpful. Neoss aims to replace usual ss command for basic usage. It
provides a list of in use TCP and UDP sockets with their respective stats. Its
main advantage over SS raw output is its clear and simple TUI (terminal user
interface) that allows you to sort, refresh and navigate what is connected to your
machine. It's installed Via NPM, meaning you need JavaScript installed.

GoAccess is a terminal based log analyzer for web servers. It's great for a quick
real time look at logs while in the terminal, but it can also generate real time
HTML, JSON, and CSV reports. GoAccess can be installed via most package
managers, works on all platforms.

Next on the list is MC or “midnight commander” Its a powerful text based file
manager with a two panel display and lots of features for manipulating files and
directories. It's also cross-platform and can be installed via most package
managers.

In the same thread of server file management is NCDU. This one is in my must-
have list. It is a disk usage analyzer that is designed to find space hogs. It's fast
and very simple to use. It can be installed on most systems and package
managers. Windows will need Linux subsystems installed to use it.

Hopefully you find some use out of these. The last topic I'd like to touch on is
DNS it's a bit topic, so I'm not going to do a massive deep dive, but if you're self-
hosting it helps to have some of the basics of DNS down. ing doesn’t work.
DNS
DNS or The Domain Name System is a core part of how the internet as we know
it works. Love it or hate it, it's what we have to work with If you want to be
accessible to the wider internet. (I dislike what it currently is it, but I’m not
opening that can of worms here.) Basically, Think of DNS like a phone book. It’s
what allows you to type duckduckgo.com instead of “52.250.42.157” every time
you need to search the internet. It translates something easy for humans to
remember into the information needed by computers to actually reach
“duckduckgo.com”

If you're hosting on a VPS, the only thing you really need to know is to know how
to point an A record at your server's IP after you decide on a domain to use.
Pretty much all VPS hosts can give you a static IP, so that's mostly a set and
forget type deal.

Hosting from home presents some challenges. One prominent one is (and a valid
question that I often hear) not having a static IP address. Nowadays with the
number of devices online needing IP addresses we do a lot of juggling, and most
IP addresses are assigned dynamically unless you pay for it from your ISP. But
there is a solution. The answer to this is called Dynamic DNS or DDNS. This
allows automatic updating of DNS servers every time an IP address changes.
There are a mess of ways to set up dynamic DNS. You can host your own
service or use a host. Here is a link with some hosts and projects to check out.

In a nutshell, it works like so. You chose a provider or set up your own. You get a
domain, install the client on your home router or server and the client periodically
checks to see if the IP address has changed, if so it updates your DNS record for
that domain.

Docker
I'm not gonna cover how to install docker here. It's best to follow the official
installation guide anyway. But I want to touch on a few things. First off, docker is
useful as hell for testing new apps. But that's about as far as I take it. I personally
do not like using docker all that much, and where possible run applications
directly. Here are some pros and cons to keep in mind.

Docker Pros
Consistency is a big one it can make things more constant between
development, testing, and deploying if your system can run docker you can run
most docker apps. It can help with isolation, reducing conflicts between apps. In
some cases it can help with efficiency as it takes less resources than traditional
VM’s. It can help with scaling as it's pretty easy to spin up more containers and
the microservice architecture can be useful because you can break down an
application into smaller manageable services, allowing for independent scaling of
said services. Lastly the community is large, so the documentation is good, and
community support is always helpful, plus there is a wide range of ready to go
docker images for deployment.

Docker Cons
I’ll start with overhead. While it's better than a traditional VM, it uses more
resources than running something directly on the host, and I/O operations can be
slower. The fact that docker shares the system's kernel means that a
compromised app could affect the system. Persistent data is doable but adds a
layer of complexity that can cause data loss with new users, it also makes
backups more complex. Networking can also be more complex with docker,
making it not as straightforward. It's also good to note that if you use UFW or
firewalld for a firewall, docker bypasses those rules. Docker is only compatible
with iptables. Also, while a well managed docker container can help manage
server resources, an improperly manged on can be detrimental to resources as
well. Containers can get too large, effecting disk size, and misconfiguration can
use too many of your servers resources. It also adds extra layers of complexity
when monitoring and debugging applications, especially across multiple
containers.

At the end of the day, it's your system. But I wanted to lay out some pros and
cons when it comes to using Docker. Moving on.

Wrap Up
Well, that about does it for the basics of server setup and tools. There is a a
script that I wrote that will do most of this for you. I wrote it to make my own
server setup faster. You can get that here, it includes all of my must-haves and
does some basic configuration. Tweak it to your own needs, and as always stay
safe out there and ping me on nostr or simplex if you have questions or if I
fucked something up in this post.

Zap Me ⚡️