This domain name is tebibyte.io. You have to admit it’s pretty cool.
Tebibyte was originally only a storage server intended for my family and close friends. I was the one with more IT knowledge and free time, so I set up the server with a 2010 Pentium and 8 TB of storage. Those times were simply quite pleasant. In the following months I kept adding more features to it, such as single sign-on, email and other services. As we used it more, I started to appreciate the efficiency of such practice, in terms of cost, human labor, and privacy. When we upgraded our server hardware, we did not buy a prebuilt server largely because of the cost. Instead, we used another existing computer, pooled from our users. The additional cost was low.
And I only used very little time managing the server that benefits a number of users. The key to reducing cost and labor is good IT management. Even for such a scale as 20 users, I can imagine a huge difference between good management and a bad one. But this is also behind a great knowledge barrier. For example, when I did not know how to write systemd unit files, I could only use screen to keep a service running. However, the knowledge itself is less than key, but rather how it is systematically obtained and organized. This affects both high-level concepts and very specific software projects.
Thus, to greatly organize research, I have compiled this document as a learning catalog. Use this memo as a compass if you are quite new to server administration or if you are not systematic with it.
This document is always updated.
General Good Practice
Keep in mind that implementation-specific configurations are only the final boring step of server administration. Understand each step you do. Make decisions in an engineering sense.
The Architecture of Things
Web Services
There are a two major designs:
- A directory with resource files and PHP or other scripts that are called by the webserver when a path is visited.
- If the scripts are left out, the site just becomes purely static.
- Any scheduled jobs can only be triggered explicitly.
- A long-running program that binds to a port ready to directly accept requests.
- This port is usually proxied.
- Any scheduled jobs are run automatically.
Web services typically need you to configure a database and file storage location.
Identity
A separate identity provider (IdP) is considered essential if you are adding more services. There are a large number of IdPs available, many of which do not need adapter code written. They of course utilize standardized IdP protocols, such as SAML, LDAP, Kerberos and OpenID Connect/OAuth2.
How we did it: Tebibyte uses OpenID Connect, Keycloak implementation. It only manages authentication, not authorization.
An integrated solution that combines directory, Kerberos authentication, DNS and CA is FreeIPA, but it is quite archaic. Recent candidates are: Ory, Kanidm.
Ideally, all authentication should be passed to the IdP (“passthrough authentication”), and the web service does not manage authentication. In that case, when the user logs out from the IdP, all web apps subsequently recognize the user as logged out. The IdP is also able to keep log of all authentication events. Note that in such an architecture, if the IdP is slow to respond, the web service also slows down. (This was why Google was down in late 2020.)
Web Services
There are designs that redirect the user to a dedicated, central IdP page for login (three-sided connections: web service, IdP, user), and designs that use the service’s own login page (two-segment chained connections: user, web service, IdP; “embedded”). The latter does not inherently support single sign-on (sign in only once, into all apps.) Any single sign-on support as well as other authentication features (such as two-factor and passwordless) need to be explicitly coded and this is even considered bad practice.
How we did it: Tebibyte uses a dedicated, central page for login, with support for various two-factor mechanisms. (There is no option for embedded login in OpenID Connect. However, it is possible to set different themes for each app.)
The web services themselves can have their own authentication subsystems. This is a tricky case. Most web services do not use passthrough authentication, and instead keep track of the user sessions on their own, once the IdP reports initial login success. This means that logging out from one app will at most also log out from the central authentication service, but not from any other already-logged-in services. (To enforce logging out, the workaround is to make all apps log out when the browser is closed.)
Usually, when a web service uses embedded login, it does not use passthrough authentication. When using centralized login, it may or may not use passthrough authentication.
How we did it: While the administrator is unable to create a fork of all web services to use passthrough authentication, all Tebibyte users are aware of this logic.
There are also features that are supported on a per-feature basis, such as groups and extra attributes.
How we did it: We use a Keycloak group mapper to tell the services about the groups.
Other Protocols
The authentication mode varies from program to program. Using passthrough authentication is becoming the industry standard.
How we did it: Our SMTP and IMAP ports use passthrough authentication (both OAUTHBEARER and XOAUTH2) thanks to Postfix and Dovecot.
System Accounts
UNIX system accounts can use an IdP backend with a PAM module.
How we did it: We are only using this after migrating all system users to another machine. It can be problematic to let a service manage the system that supports itself.
System Organization
Learn the File Hierarchy Standard (FHS) or whatever standard for your system. Take note of any locations that may be incompliant or difficult to remember.
Use you package manager. Use other package managers (such as Python’s PIP) carefully and not for the whole system.
Create the service users and set permissions for their directories. Use your init manager (systemd or otherwise) to manage startup dependencies.
Storage
User data storage can either be a mounted system volume on the same server as the service software, or a separate server typically through some object storage interface (which forms three-sided connections: web service, storage, user). If scaling is expected, it can be very suitable to first deploy a dedicated storage server on the same machine to make any future migrations easier. If so, assign whole disks to a storage server, not directories.
If redundancy is a concern, use a disk array. But a disk array is not a backup. You should ideally keep a backup somewhere easily accessible, and another backup at a different location.
Database
If you are using a UNIX-like server and the database is on the same machine as the web app, consider using a UNIX socket. This not only handles connection authentication on some systems but also achieves zero copy.
Make sure to put your database on an SSD, and do not forget to back it up.
Network
If you are using a single machine for everything, try to imagine that each service is actually on a separate machine. In your configuration files, before cramming everything onto a single apex domain (or even worse, localhost (or even worse, 127.0.0.1)), assign them wisely and write the correct subdomains for each component in your very first configuration file. (You get to decide what is considered a component.) Use CNAME in DNS. This way, when you only migrate certain services, nothing will break.
TLS
Use TLS for all possible protocols and preferably disable insecure versions. Your Let’s Encrypt certificate is applicable to not only the HTTPS port but all TLS ports. Make this a natural practice. Security works better when everyone is used to it.
Some ports use “StartTLS”, a command the client sends to establish TLS connection on an existing plaintext connection.
When you update the TLS certificates, reload all services that use the certificates, like the web and mail servers.
Use TLS outgoing requests even for same-machine connections across components, such as OpenID Connect login requests from the web service to the IdP. This way, when you only migrate certain services, nothing will break.
HTTP Proxy
Pay attention to proxy-specific headers. Also keep track of port numbers. Disable external access of naked ports.
Firewall
Use them whenever you need.
The modern email security framework consists of DKIM, SPF, and DMARC. Other online spam detection services can be an addition.
Web Server
With a correct web server configuration, the served directory structure can be very different from the structure stored on the server. An example is this website, where there are actually no /~user/ directories stored on the server.
In addition to the main configuration files, there are per-directory .ht* files for fine control. Use this to avoid a thousand location cases in your main configuration files just for access control.