Installing on a fresh Stretch system is relatively simple. First, add the Bytemark package signing key:

curl -sSL https://secure.bytemark.co.uk/key/repositories-2014.key | sudo apt-key add -

Second, add the following to /etc/apt/sources.list.d/symbiosis.list:

#
# Bytemark Symbiosis Packages
#
deb     http://symbiosis.bytemark.co.uk/stretch/ ./
deb-src http://symbiosis.bytemark.co.uk/stretch/ ./

Once that is in your sources, run:

apt-get update
apt-get install --install-recommends bytemark-symbiosis

At the end of this process, you should have a fully functioning Symbiosis system with all of the features documented here available to you for use.

Users of Bytemark’s Cloud Servers can get a fresh install of Symbiosis by simply selecting the image at VM install time. The panel is accessed at https://panel.bytemark.co.uk. Once logged in, using the Add a cloud server button, a machine can be installed within a few seconds. Select the Stretch Symbiosis image as per the screenshot below.

Debian have comprehensive release notes, of which chapter 4 covers the recommended upgrade procedure. We have provided a shorter version for this, which is immediately below:

The first thing to do is make sure that you have backups. These should be kept in /var/backups/localhost, and they should be up to date.

Next, alter /etc/apt/sources.list. Change all instances of the word jessie to stretch. If you have backports, you can remove them, and any entries for Jessie LTS should also be removed. Then change the Symbiosis repository lines to match those shown in the previous section.

You can then proceed with the upgrade by running:

apt-get update
apt-get dist-upgrade

Following the upgrade, to use PHP7 you will need to disable and enable the appropriate Apache modules:

a2dismod php5
a2enmod php7.0
systemctl restart apache2

You can swap back at any time by disabling the php7.0 module and enabling the php5 module instead.

The updated version of Roundcube relies on PHP7.0 by default. If you would like to continue using PHP5, you will need to install the php-net-idna2 package by running:

apt-get install php-net-idna2

As this is an upgrade for all the software on the system, a large number of questions may be asked about configuration files during the upgrade. Some of these will relate to packages Symbiosis has installed as dependencies, and the answers to these questions are given below.

That should be everything; you may have been asked other questions if you have installed extra packages on your system - answer them as you see fit.

This release of Symbiosis includes a number of new features that are summarised in Chapter 1, What’s new since the last release.

Each component that makes up Symbiosis is separately packaged as follows. Each package can be installed individually if needed.

Symbiosis is an attempt to encourage best practice at all times in systems administration, whilst keeping things as simple as possible, and free of surprises. As a result there are a few general rules to bear in mind when tinkering with your system.

# DO NOT EDIT THIS FILE - CHANGES WILL BE OVERWRITTEN

# Checksum MD5 586732ff59e60115d0ec1c4905c72773

# ------------------------------------------------------------------------------
# /etc/exim4/symbiosis.d/10-acl/40-acl-check-mail/00-header
# ------------------------------------------------------------------------------

# ACL that is used after the MAIL command
acl_check_mail:

# ------------------------------------------------------------------------------
# /etc/exim4/symbiosis.d/10-acl/40-acl-check-mail/90-default
# ------------------------------------------------------------------------------

# Allow anything not already denied to connect
  accept

All the files required for a website for the domain my-brilliant-site.com are kept in /srv/my-brilliant-site.com/public/htdocs/.

If you wish to use CGI scripts for your domain, then simply copy them to a directory named cgi-bin/ beneath the public/ directory. They must all be marked as executable. This means setting the permissions to 755. In FileZilla, right click the file and select File Permissions… from the menu. The file should have Execute set for the owner, group, and public permissions.

For example, for my-brilliant-site.com the scripts would live in /srv/my-brilliant-site.com/public/cgi-bin/.

Any executable files in that directory will now be treated as CGI scripts for your domain. For example if you created the file /srv/my-brilliant-site.com/public/cgi-bin/test.cgi This would be referred to as: http://my-brilliant-site.com/cgi-bin/test.cgi

Each hosted website can have visitor statistics automatically generated and accessible at http://my-brilliant-site.com/stats/. These statistics will be updated once per day, and the raw access logs will be made available as /srv/my-brilliant-site.com/public/logs/.

As of the Stretch release of Symbiosis, these daily statistics are disabled by default. If you wish to continue using them, you’ll need to enable them explicitly with the creation of a stats file in the website’s configuration directory. For example, for my-brilliant-site.com, the stats file should exist at /srv/my-brilliant-site.com/config/stats.

If you had previously disabled stats with the creation of a file config/no-stats, this should be removed automatically following a dist-upgrade.

It is also possible to customise the statistics generated by editing the file config/webalizer.conf. This file is documented at the Webalizer project website.

If there are many sites on the same machine, then it is possible to customise all the sites' Webalizer configurations by editing the template that is available at /etc/symbiosis/apache.d/webalizer.conf.erb. Configuration files will be updated when the statistics are next generated, but only for sites whose configurations either do not exist, or have not been edited by hand.

You can view new websites before any DNS changes are made.

For example, if the virtual machine example.default.bytemark.uk0.bigv.io is hosting www.my-brilliant-site.com, i.e. the directory /srv/my-brilliant-site.com/public/htdocs/ has been created, then the website can immediately be viewed at http://my-brilliant-site.com.testing.example.default.bytemark.uk0.bigv.io.

There are some important things to note though: - There is no www part added to the domain name — it is just the directory name prepended to .testing.example.default.bytemark.uk0.bigv.io. - This testing alias isn’t guaranteed to work in all cases, for complex site setups it might not work entirely as expected. - The testing alias only allows the testing of websites. Therefore FTP logins, email delivery, or checking is explicitly unsupported.

In this scenario, you have registered two domains for example my-brilliant-site.com and my-brilliant-site.co.uk, but you want the same content to be served at both addresses. There is no need to create two separate directory structures, you can just set up one directory structure and then create a soft link (aka symbolic link or symlink) to the second.

This creats a symbolic link of my-brilliant-site.co.uk pointing at my-brilliant-site.com. Now browsing to my-brilliant-site.co.uk will show the same content that appears at my-brilliant-site.com.

If a document tree were created in /srv/my-brilliant-site.com/public/ then that site would be available under two hostnames:

There are people who prefer to use only a single name, and to automatically redirect visitors using the wrong name to using the preferred name. This can easily be achieved by using Apache’s mod_rewrite facility.

If you prefer all visitors see the www-based site you could create the file /srv/my-brilliant-site.com/public/htdocs/.htaccess with the following contents:

RewriteEngine on
RewriteCond %{HTTP_HOST} !^www.*$ [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=301,L]

This examines each incoming request, and if the hostname doesn’t begin with "www." then it is prepended to the request and a redirect is issued.

It is perfectly possible to alter the way Symbiosis configures Apache, either for an individual domain, or for all domains hosted on the server.

Symbiosis hosts sites on a server in one of two ways, based on the IP address that site has configured. If it uses one of the server’s primary IP addresses, then it is assumed that the site is hosted using the "mass-hosting" configuration. If the site has a secondary IP assigned then Symbiosis generates an individual snippet for that site, and Apache is configured to use that snippet when dealing with HTTP requests for that domain. Both configuration techniques are configured using a template, which allows the server’s administrator to fiddle with, and tweak the configuration.

In /etc/symbiosis/apache.d/ there are a number of templates that are used to generate configuration snippets for both the mass-hosting, as well as individual sites.

By default, access requests for each site on a machine will go to public/logs/access.log. If the site has SSL enabled, the request logs will go to public/logs/ssl_access.log. These logs get rotated once a day, and compressed after two days.

The error logs for a site will go to one of two places, depending on how the site is configured. If the site has its own SSL certificate, or otherwise has its own IP address, then the error logs will go to public/logs/error.log, or public/logs/ssl_error.log. Otherwise the error logs will go to /var/log/apache2/zz-mass-hosting.error.log.

Finally, if a request is received for a domain that is not present on the box, then it is logged to zz-mass-hosting.access.log if it received on the primary IP of the machine. If the request comes on any other IP then it is logged to other_vhosts_access.log. Both of these last two files are located in /var/log/apache2 .

Here is an example configuration layout for the domain my-brilliant-site.com, all of which is contained under /srv/my-brilliant-site.com/.

Until recently, having a certificate signed by a trusted authority involved having varying degrees of identity checks made, and paying a fee. Vendors that are able to sell you a certificate include Rapid SSL and Comodo.

In December 2015, a new, free, automatic SSL certificate service started issuing certificates. This servers is called LetsEncrypt, and it is supported by a number of big names on the internet, including Facebook and Mozilla. This service has now been successfully providing domain-verified certificates for a few months, and is used by Symbiosis to generate trusted certificates for all the domains on a machine.

Symbiosis can also generate self-signed certificates on occasions where it has not been possible to use LetsEncrypt.

Symbiosis has the idea of "providers" to generate SSL keys, requests, and certificates. By default Symbiosis will use the LetsEncrypt provider, but you can specify another provider, e.g. *SelfSigned" by putting the word "selfsigned" in config/ssl-provider for the domain in question.

If you wish to use LetsEncrypt, put "letsencrypt" in config/ssl-provider.

If you wish to disable automatic certificate provisioning entirely, you can put the word false into config/ssl-provider.

Symbiosis uses a command ‘symbiosis-ssl` to manage domains’ certificates. It is run on a daily basis to check and replace certificates that are due to expire in the next 10 days, or are otherwise missing.

This command can also be used to verify sites' certificates.

$ symbiosis-ssl --verbose
* Examining certificates for app.my-brilliant-site.com
  Current SSL set 0: signed by /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1, expires 2016-05-25 23:00:00 UTC
* Examining certificates for my-brilliant-site.com
  Current SSL set 0: signed by /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1, expires 2016-05-25 10:32:00 UTC
* Examining certificates for example.default.bytemark.uk0.bigv.io
  Current SSL set 4: signed by /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1, expires 2016-05-29 10:26:00 UTC

This command shows the current set in place for each site. A set is a collection of files, consisting of

Each set is kept in its own directory in config/ssl/sets, and the current set is symlinked to config/sets/current.

In the above output the host example.default.bytemark.uk0.bigv.io is currently using set 4. That means its directory layout will look like:

/srv/example.default.bytemark.uk0.bigv.io/config/ssl/current -> /srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/4
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/letsencrypt/account_key
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/0/ssl.combined
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/0/ssl.crt
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/0/ssl.csr
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/0/ssl.key
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/1 # (with a complete set of ssl.key etc)
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/2 #
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/3 #
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/4/ssl.bundle
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/4/ssl.combined
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/4/ssl.crt
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/4/ssl.csr
/srv/example.default.bytemark.uk0.bigv.io/config/ssl/sets/4/ssl.key

If you wish to use a different certificate than the one Symbiosis has generated for your domain, you can use the certificate signing request that was created for the certificate that should already be in place. You can then go to your provider with this request, and ask them to generate a certificate.

The CSR generated will be for the "bare" domain, with all aliases as Subject Alternative Names.

If you wish to inspect the CSR, you can run

openssl req -in ssl.csr -text -noout

To add a third party SSL certificate for a website, upload your ssl.crt, ssl.key, and ssl.bundle files from the certificate provider to your server. For a domain example.com, these should be located at:

/srv/domain.com/config/ssl.crt
/srv/domain.com/config/ssl.key
/srv/domain.com/config/ssl.bundle

Ensure ssl-provider is set to false, by running: echo "false" > /srv/domain.com/config/ssl-provider

Remove the SSL current symlink if present, as this will otherwise take precedence: rm /srv/domain.com/config/ssl/current

Reconfigure the Apache vhost to activate the certificate: sudo symbiosis-httpd-configure -v domain.com

Here is an example configuration layout for the domain my-brilliant-site.com, all of which is contained under /srv/my-brilliant-site.com/.

The mail servers have been set up with standard port assignments as follows. These are all the standard ports for the protocols.

In order for a domain to be configured to accept email, one of two things must be present. Either the domain must have a mailboxes/ directory present, or one of the files config/default_forward or config/aliases must be present.

For example, if the domain my-brilliant-site.com would like to host mail normally, i.e. one mailbox per user hosted on the same machine, then the directory /srv/my-brilliant-site.com/mailboxes/ should be created. Then in there, one directory per user should be created. If bob@my-brilliant-site.com would like to receive mail, then /srv/my-brilliant-site.com/mailboxes/bob/ should be created.

Once this directory has been created, this mailbox can be accessed via IMAP/POP3, or Roundcube webmail. For example, the mailboxes for mybrilliant-site.com would be accessible at mybrilliant-site.com/webmail.

Assuming that this is the only directory inside /srv/my-brilliant-site.com/mailboxes/ then only mail addressed to bob@my-brilliant-site.com will be accepted. Any other mail addressed to my-brilliant-site.com will be rejected.

If you would like to accept all mail for my-brilliant-site.com, regardless of who it is addressed to, then create the file /srv/my-brilliant-site.com/config/default_forward. The contents of this file should be a single email address, or a comma-separated list of email addresses. For example, to forward all mail to bob@my-brilliant-site.com, unless the recipient’s mailbox already exists, then /srv/my-brilliant-site.com/config/default_forward should contain bob@my-brilliant-site.com.

If you would like the domain nomail.my-brilliant-site.com not to receive any mail at all, then remove the directory /srv/nomail.my-brilliant-site.com/mailboxes/ and ensure that the file /srv/nomail.my-brilliant-site.com/config/default_forward does not exist.

A new feature in this release is the ability to have unix users with email accounts based in their home directories. These will receive emails for the host name of the machine, which you can find out by running hostname on the command line. The result of this will display the domain in the email address the system users will get, eg, the part after the @. The other half will be dictated by their username, eg, "admin" or "my-user".

To start with, we create a .password file in, eg, /home/my-user/. Initially this can contain the password in plain text.

Once the password file is in place, the new user will be able to login and collect email. Logins over SMTP, IMAP, and POP3 will all work identically to a normal email user, with the same ports and SSL/TLS requirements. The principal difference is the username is just their bare username, without a domain. E.g. for a user clare, her login for SMTP/IMAP/POP3 etc, is just clare.

Unix users' Maildir directories will reside in /home/my-user/Maildir by default. This allows these users to use system mail readers such as mutt in order to read and send email, obviating the need to use IMAP and SMTP.

These users are also able to control the following files:

The password for a mailbox should be set by the contents of a file named password inside a user’s mailbox directory. The contents of this file may be in plain text, or encrypted. If plain text is used, the system will automatically encrypt the password.

To encrypt all email passwords on the system, you can run

`symbiosis-email-encrypt-passwords --verbose`

The --verbose flag is there to provide more output.

Users are able to change their own passwords through the webmail system, which by default is Roundcube.

To change a password in Roundcube, log in and select Options. This will being up the Options page :

From there, select Change Password, where you must enter your current password, and enter a new one. Passwords which are ether too weak or too short will be rejected by the system.

All email addresses can be used with a suffix. This allows people to filter their email by the To: address. The separator between the local part and suffix is the + sign.

For example, Bob signs up to a shopping site at http://example.com. He might use bob+example@my-brilliant-site.com his email address when signing up, such that he can filter all email from that shop.

Symbiosis can enforce users' mailbox size with quotas. This will prevent mail from being delivered to a user if their mailbox grows too large.

A default quota for each individual mailboxes in a domain can be specified in config/mailbox-quota. A per-mailbox quota can be defined in a file named quota which resides in a user’s mailbox directory.

These files both have the same format, which is just a number of bytes over which mail should not be delivered. This number can have a suffix of k, M, or, G which represent kilobytes, megabytes, and gigabytes, or ki, Mi, or Gi to represent kibibytes, mebibytes, and gibibytes, respectively.

For example, to limit the size of each mailbox for the domain my-brilliant-site.com to 200MB, i.e. 200,000,000 bytes, put 200M in /srv/my-brilliant-site.com/config/mailbox-quota.

To grant bob@my-brilliant-site.com a 1GiB quota, i.e. 1,073,741,824 bytes, put 1Gi in /srv/my-brilliant-site.com/mailboxes/bob/quota.

Quotas in a user’s mailbox directory take precedence over the default quota.

Sieve is a standard language that users can employ to filter their email on the server. Additionally using any one of a number of clients, users can edit their filtering rules without needing shell access to the server.

Each user can create a number of scripts in a directory called sieve.d/, with the current script being kept in a file called sieve.

Only one of these scripts can be active at a given time for each user; add to an existing file rather than creating a new one if you require extra filters.

There are two methods of forwarding email. The first is a per-mailbox forwarding service, and the second is a per-domain service. For the per-user service, a file named forward should be put in a user’s mailbox directory. The per-domain service uses the same file format as the per-user service, but the file should be uploaded to config/default_forward instead.

For example, bob@my-brilliant-site.com would set up a file called /srv/my-brilliant-site.com/mailboxes/bob/forward.

If all the mail for my-brilliant-site.com needed to be forwarded elsewhere, then the file would be called /srv/my-brilliant-site.com/config/default_forward.

Both of these files can be interpreted in two ways. Firstly they can be a comma separated list of email addresses. For example, if Bob wanted to forward his email onto Charlie and Dave, his forward file might read

charlie@example.com, dave@example.com

The second way these files are interpreted is as an Exim filter file. The full specification is documented at the Exim project site.

Here are some examples of what is possible.

To forward mail on, but keep a copy

# Exim filter
unseen deliver charlie@example.org
unseen deliver dave@example.com

To rewrite all mail for a domain to example.com. This is probably best used in config/default_forward.

# Exim filter
deliver $local_part@example.com

The Exim documentation has further examples of what is possible.

It is possible to set a vacation message for a user by putting a message in file called vacation in the user’s mailbox directory.

For example, for bob@my-brilliant-site.com, the message would go in /srv/my-brilliant-site.com/mailboxes/bob/vacation. On Bob’s return, the people who received vacation messages are logged to /srv/my-brilliant-site.com/mailboxes/bob/vacation.log. Once he’s read it, that file, along with /srv/my-brilliant-site.com/mailboxes/bob/vacation and /srv/my-brilliant-site.com/mailboxes/bob/vacation.db should all be removed.

Each domain can have a list of aliases. This is just a file that contains a list of local parts, and a list of places they should be sent on to. This file should be in the config/ directory and is named aliases.

For example, my-brilliant-site.com has a list of dummy addresses that should be sent on to Bob. So the aliases file would be kept at /srv/my-brilliant-site.com/config/aliases and contains the following.

webmaster   bob@my-brilliant-site.com
chairman    charlie@example.com
staff       bob@my-brilliant-site.com, charlie@example.com, dave@example.com

This ensures that webmaster@my-brilliant-site.com is sent to bob@my-brilliant-site.com; chairman@my-brilliant-site.com is sent to charlie@example.com; staff@my-brilliant-site.com is sent to bob@my-brilliant-site.com, charlie@example.com, and dave@example.com.

Symbiosis comes with SpamAssassin and ClamAV installed to protect your email users against spam and virus in their inbox. To enable these features, simply create the files config/antispam or config/antivirus as appropriate. This will configure that domain to reject email if it is considered to be spam, or if it contains a virus.

If you’d rather accept all email and simply tag it as spam, put the word tag in config/antispam. This will also cause the email to be delivered into the Spam/ folder for that user.

The configuration for SpamAssassin for the admin user is kept in /srv/.spamassassin/user_prefs. Here you can adjust what score is needed to reject spam, and which tests are used during scanning. This file will only appear after a mail has been received with spam detection turned on, but one can be created and configured before this occurs.

The file contains comments and instructions, and further tips can be found on the SpamAssassin wiki.

In brief, to cause more mail to be rejected, you need to reduce the threshold score. Therefore change the line reading # required_score 5 should be changed to required_score 4. Notice that the # has been removed at the start of the line to un-comment it.

Similarly if mail is being rejected, you can increase the score.

Further instructions can be found on the SpamAssassin wiki.

There is no facility to train the SpamAssassin Bayesian learner yet.

Headers are added to messages when spam or virus scanning is enabled. These can be used by email clients to filter email, for example in to spam or quarantine folders.

With spam scanning enabled, any email that is accepted has the following headers added

The score is determined by SpamAssassin, and is the basis for acceptance or rejection. The higher the score, the more certain SpamAssassin is that the message is unwanted. The default threshold for rejection is 5.

The bar is a length of pluses or minuses that provide an easy-to-parse representation of the score. A positive score is given pluses, a negative score minuses. For example a score of 5.6 would be represented as ++++++; a score of -2.2 would be represented as --.

The status is always either innocent or spam, depending on the score.

When virus scanning is enabled, the header X-Anti-Virus is added to messages that have been scanned. This is set to either infected or clean.

The content of an email can be changed if it’s marked as spam by SpamAssassin. For example, you may wish to prepend an email’s subject with SPAM to highlight it in your inbox. To do this, append the following code block to the end of the /etc/exim4/system_filter file:

if $h_X-Spam-Status: contains "spam"
then
    headers add "Old-Subject: $h_subject"
    headers remove "Subject"
    headers add "Subject: *** SPAM *** $h_old-subject"
    headers remove "Old-Subject"
endif

There are three lists from Spamhaus that can be used to reject email based on the sender’s IP address, namely

These lists are combined to form the Zen list.

The following instructions will enable use of these lists on our example domain my-brilliant-site.com.

That is all that is needed to start using the Spamhaus Zen blacklist. If you’d rather use a combination of lists create one or more of the following files:

While publicly maintained blacklists like spamhaus are much easier to rely on and lower maintenance, at some point you might find occasion to block specific email senders. Symbiosis allows blocking based on these criteria:

To block with one of these criteria, you can use:

Each entry to these files should be on a new line.

It is also possible to explicitly allow email from senders that would otherwise be blacklisted by adding entries in similarly named files under /etc/exim4/whitelist.

Symbiois can now easily be configured to limit the number of outbound emails, either per-user, or per-domain. You might want to rate limit to prevent anyone taking advantage of your server, maybe using it to send out SPAM. The limit can be configured on a per-user basis with the following file :

/srv/domain.com/mailboxes/bob/ratelimit

and on a per-domain basis with the following file :

/srv/domain.com/config/mailbox-ratelimit

In both cases, the contents of the file should be a number, which represents the allowed limit per day. If the file is left blank, the default of 100 is applied. Senders who breach this limit will be sent an error email to explain why their message has not been sent, and discarded.

If the domain has a config/ip file in place then the IP in that file will be used for outgoing email. This can be useful if your machine has a number of IP addresses and you’re suffering from deliverability issues. config/

Symbiosis allows you to blacklist email senders by:

Execute the following actions as root user rather than admin.

To block a specific email address, simply add the address to the file /etc/exim4/blacklist/by_sender.

The by_sender list accepts email addresses like malefactor@example.com or wildcarded ones like *@example.com to block a whole domain. Note that this blacklist is matched against the Envelope Sender address, rather than the From address.

To block by IP, add the IP address to /etc/exim4/blacklist/by_ip

The by_ip list accepts IP addresses like 192.168.66.6 or ranges like 10.66.6.0/24. This is used to blacklist by the IP address of the connecting machine.

Finally the by_hostname list accepts hostnames like bad.example.com, or wildcarded like *.example.com. This is used to blacklist against the reverse DNS of the IP of the host connecting.

If you have multiple active domains with email hosted on your server, enabling SNI will allow you to use a unique SSL certificate for each. This will help avoid the "certificate mismatch" errors you may see when attempting to login to one of your email accounts.

To configure SNI support for Exim, please see this guide.

And to configure SNI support for Dovecot, please see this guide.

Here is an example configuration layout for the domain my-brilliant-site.com. All the following files are kept in /srv/my-brilliant-site.com/.

Domains' XMPP configuration is kept in /etc/prosody/config.d with each domain having its own snippet. Feel free to edit these snippets as you see fit, as once edited they will never get overwritten automatically.

If you do wish to restore the configuration to the default, you can run symbiosis-xmpp-configure --force --verbose.

Symbiosis uses a configuration template for the XMPP server. This is kept in /etc/symbiosis/xmpp.d/prosody.template.erb. This is the place to adjust things for all domains running on the server.

Once you’ve adjusted that to your liking, you can run symbiosis-xmpp-configure --verbose to apply your changes.

Basic per-domain authentication is controlled by the config/ftp-password file. This file contains the plain-text or hashed password for the FTP user whose username is the domain name. This user is limited to accessing the public directory for that domain.

For example, /srv/my-brilliant-site.com/config/ftp-password contains the password for the FTP user my-brilliant-site.com, and that user will be limited to accessing /srv/my-brilliant-site.com/public.

This authentication method is controlled by the config/ftp-users file. This file contains more than just the password. Each line in the file represents a different user, and contains the username, password, base directory, and quota. Comments in the file start with #.

# username:password:directory:quota
bab:babs password:/path/to/base:10M

The directory and quota fields are optional. If the password field is empty, the user will not be able to log in.

In the above example, if that file was kept at /srv/my-brilliant-site.com/config/ftp-users then the user babs@my-brilliant-site would be able to log in with the password babs password. She’d be limited to the accessing files and directories below /path/to/base, and uploads to that that directory would be prohibited if it contains more than 10 Megabytes of data.

It is possible to use the other forms of authentication provided by Pure-FTPd. The Pure-FTPd manual gives a good run down of all the various ways to do it. Here the two most common ways have been documented.

echo /etc/pure-ftpd/pureftpd.pdb > /etc/pure-ftpd/conf/PureDB
touch /etc/pure-ftpd/pureftpd.pdb
ln -s /etc/pure-ftpd/conf/PureDB /etc/pure-ftpd/auth/50puredb
service pure-ftpd restart

pure-pw useradd foo -u 1000 -g 1000 -d /path/to/home -m

echo 1 > /etc/pure-ftpd/conf/PAMAuthentication
ln -s /etc/pure-ftpd/conf/PAMAuthentication /etc/pure-ftpd/auth/50pam
serivce pure-ftpd restart

There are two ways of specifying a quota. The default quota for a domain goes in config/ftp-quota. This controls the quota for the per-domain user in public, as well as the default quota for users specified in config/ftp-users. Its format is the same as that for email quotas.

For the multi-user configuration file, a user’s quota can be specified in the final field, again in the same format as that used for email quotas.

All usual firewall configuration can be carried out by creating and deleting files in /etc/symbiosis/firewall/. In this directory there are a number of subdirectories. Permissions for inbound connections are stored in /etc/symbiosis/firewall/incoming.d/, and outbound connections in /etc/symbiosis/firewall/outgoing.d/.

These files are all of the format number-name. The number determines the position of the rule in the firewall, the name is the name of the service that we wish to permit. These names are stored in /etc/services. There are also names that do not correspond to services, which are documented in the next section.

Additionally if the name is not known then the file format can be number-number where the first number specifies the position of the rule in the firewall, and the second number is the port that should be opened. For example, the files 10-http and 10-80 achieve the same effect.

Finally, each file can contain a list of hostnames or IP addresses to which that rule will apply, one per line. For example, if addresses were added to an incoming rule, named incoming.d/10-accept, all connections from those addresses would be accepted. If a file were added named outgoing.d/20-reject and address added to that file, then outgoing connections to those addresses would be rejected.

For example, to allow an incoming connection to arrive at your machine, and be accepted, on port 22, you would create the file /etc/symbiosis/firewall/incoming.d/10-ssh. The firewall will update as soon as the file has been created, so no commands are needed to be run.

If you were wishing to ensure that your host would only accept incoming SSH requests from your office you might create the same file with the contents office.my-brilliant-site.com.

This would ensure that when the firewall was generated incoming connections on the SSH port would be accepted from the host office.my-brilliant-site.com but not from anywhere else.

There are a number of rules that don’t naturally fit the convention described above. This list describes rules that have been written specially for Symbiosis to cope with these situations. Each rule described below can be used in both incoming.d/ and outgoing.d/, and for both IPv4 and IPv6 addresses, unless otherwise specified.

These rules are used in the same way as those described in the previous chapter. Files are added in the incoming.d/ or outgoing.d/ directory with the name prefixed by a number giving the position of the rule. The files can contain addresses or hostnames, one per line, against which the rule should be applied.

These rules are all contained in /usr/share/symbiosis/firewall/rule.d/. It is perfectly possible to write your own rules based on those in this directory, but they should be kept in /usr/local/share/symbiosis/firewall/rule.d/.

This example should be read in conjunction with the previous sections. A machine has the following firewall rules defined for its incoming connections.

This would set up a firewall that would do the following tests, in order:

These rules would be installed for IPv4 and IPv6 connections using iptables and ip6tables respectively. To inspect the firewall rules at any given time, you can run sudo iptables -L -v -n which will return the current firewall status. In this example, the rules would look like this.

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target    prot opt in out source          destination
    0     0 ACCEPT    all  --  lo *   0.0.0.0/0       0.0.0.0/0
   13  1012 whitelist all  --  *  *   0.0.0.0/0       0.0.0.0/0
    0     0 blacklist all  --  *  *   0.0.0.0/0       0.0.0.0/0
    0     0 ACCEPT    all  --  *  *   0.0.0.0/0       0.0.0.0/0 state ESTABLISHED
    0     0 ACCEPT    all  --  *  *   0.0.0.0/0       0.0.0.0/0 state RELATED
    0     0 ACCEPT    icmp --  *  *   0.0.0.0/0       0.0.0.0/0 icmp type 8
    0     0 ACCEPT    icmp --  *  *   0.0.0.0/0       0.0.0.0/0 icmp type 0
    0     0 ACCEPT    icmp --  *  *   0.0.0.0/0       0.0.0.0/0 icmp type 11
    0     0 ACCEPT    tcp  --  *  *   1.2.3.4         0.0.0.0/0 tcp dpt:22
    0     0 ACCEPT    udp  --  *  *   1.2.3.4         0.0.0.0/0 udp dpt:22
    0     0 ACCEPT    tcp  --  *  *   0.0.0.0/0       0.0.0.0/0 tcp dpt:80
    0     0 ACCEPT    udp  --  *  *   0.0.0.0/0       0.0.0.0/0 udp dpt:80
    0     0 ACCEPT    tcp  --  *  *   0.0.0.0/0       0.0.0.0/0 tcp dpt:666
    0     0 ACCEPT    udp  --  *  *   0.0.0.0/0       0.0.0.0/0 udp dpt:666
    0     0 ACCEPT    tcp  --  *  *   0.0.0.0/0       0.0.0.0/0 tcp dpt:25
    0     0 ACCEPT    udp  --  *  *   0.0.0.0/0       0.0.0.0/0 udp dpt:25
    0     0 REJECT    all  --  *  *   0.0.0.0/0       0.0.0.0/0 reject-with icmp-port-unreachable

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target    prot opt in out source          destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target    prot opt in out source          destination
    0     0 ACCEPT    all  --  *  lo  0.0.0.0/0       0.0.0.0/0
    7  1388 ACCEPT    all  --  *  *   0.0.0.0/0       0.0.0.0/0 state ESTABLISHED
    0     0 ACCEPT    all  --  *  *   0.0.0.0/0       0.0.0.0/0 state RELATED
    0     0 REJECT    all  --  *  *   0.0.0.0/0       0.0.0.0/0 owner UID match 33 reject-with icmp-port-unreachable

Chain blacklist (1 references)
 pkts bytes target    prot opt in out source          destination
    0     0 REJECT    all  --  *  *   71.63.72.4      0.0.0.0/0 reject-with icmp-port-unreachable
    0     0 REJECT    all  --  *  *   61.145.118.190  0.0.0.0/0 reject-with icmp-port-unreachable

Chain whitelist (1 references)
 pkts bytes target    prot opt in out source           destination
   13  1012 ACCEPT    all  --  *  *   212.110.163.132 0.0.0.0/0

This listing shows how the rules in the files under /etc/symbiosis/firewall/ are translated into iptables rules. It also shows that by default all connections on the loopback interface lo are permitted, and that the whitelist and blacklist tables have references in the INPUT, i.e. incoming, table before the rules defined in /etc/symbiosis/firewall/incoming.d/ are applied.

IPv6 rules follow the same format, and can be checked by running sudo ip6tables -L -v -n.

The Symbiosis firewall package should allow you to carry out the most common tasks, simply by creating files named after the services you wish to permit or deny.

However there are times when you might wish to make your own custom additions, and for this purpose the firewall package allows you to run an unlimited number of custom scripts/programs once it has loaded the rules - these scripts may perform arbitrary actions, but will be most typically used to update the firewall rules, via the iptables or ip6tables commands.

The program run-parts is used to execute scripts in /etc/symbiosis/firewall/local.d/, after the firewall has finished loading. This means that the scripts have to have to fulfil the naming conditions described in the run-parts(8) manual page. Essentially the script should be marked executable, and only contain alphanumeric characters in its name.

The symbiosis-firewall-blacklist tool runs four times an hour, and is designed to scan your server’s logfiles for abusive behaviour from malicious remote hosts. Malicious activity which is detected will result in the remote host being denied further access to your server.

Currently we regard malicious activity as:

Every 15 minutes various logfiles are scanned for certain patterns to search for new malicious IPs, and the firewall is updated.

These patterns are defined in /etc/symbiosis/firewall/patterns.d/. For example, for SSH the following pattern definition is used:

#
#  The logfile we look for matches within.
#
file = /var/log/auth.log 

#
#  Any matches will be denied access to these ports.
#
#  Comma-separated values are expected.
#
ports = 22 


#
#  Patterns we'll match upon.
#
Failed password for invalid user [^ ]+ from __IP__ port [^ ]+ ssh2 
Failed password for [^ ]+ from __IP__ port [^ ]+ ssh2

If an IP matches one of those patterns in the period since the last check was made, it is added to the blacklist.

Disabling the firewall completely will disable the blacklisting behaviour, but you might also wish to disable that separately.

To do this, login over SFTP as admin and create the file /etc/symbiosis/firewall/blacklist/disabled. This will immediately disable and clear the blacklist.

The symbiosis-firewall-whitelist tool runs once per hour, and is designed to perform the opposite task to the symbiosis-firewall-blacklist script - in short it is designed to ensure that any remote host which has successfully connected to your server in the past isn’t (accidentally) blacklisted in the future.

Every hour the script will examine the successful logins which have been observed recently. Each IP address which has successfully been the source of a login attempt will be permitted access to the system on a global basis, and will thus not be locked out.

As with the automatic blacklist, IPv6 addresses are masked to a /64, which is the smallest recommended assignment for an end site.

To disable the automatic whitelist, login over SFTP as admin and create the file /etc/symbiosis/firewall/whitelist.d/disabled. This will immediately clear the whitelist, and prevent further updates.

You can add your own entries to the whitelist, which never expire, by creating entries in the directory /etc/symbiosis/firewall/whitelist.d/. Create the file /etc/symbiosis/firewall/whitelist.d/<ip address> and the specified IP address will not be blacklisted, or refused access to your server.

Symbiosis now comes with basic SYN-ACK/ACK flood protection. These are simple but effective denial of service attacks, which can leave the network stack inundated. Wikipedia has an article on the matter for the curious

To enable the protection, create the following file :

/etc/symbiosis/firewall/incoming.d/00-syn-ack-flood-protection

If you wish you may disable the firewall completely, allowing remote users to connect to any service you have running upon your machine.

We’d not recommend that you disable the firewall, because it does provide a increase in system security, but if you wish it is possible by executing the following two commands:

touch /etc/symbiosis/firewall/disabled
sudo symbiosis-firewall flush

The presence of the disabled rule will not itself clear the firewall, merely prevent further updates to it, which is why the flush command is needed.

All configuration of the firewall is conducted via the presence or absence of files in a number of directories beneath /etc/symbiosis/firewall/. Actions and rules are all kept under /usr/share/symbiosis/firewall/.

This is an example of the records Symbiosis generates for my-brilliant-site.com. They are created automatically and stored in config/dns/my-brilliant-site.com.txt.

DNS records example. 

#
#  Nameserver records. 
#
.my-brilliant-site.com::a.ns.bytemark.co.uk:300
.my-brilliant-site.com::b.ns.bytemark.co.uk:300
.my-brilliant-site.com::c.ns.bytemark.co.uk:300

#
#  The domain name itself 
#
=my-brilliant-site.com:89.16.174.65:300

#
#  Useful aliases. 
#
+ftp.my-brilliant-site.com:89.16.174.65:300
+www.my-brilliant-site.com:89.16.174.65:300
+mail.my-brilliant-site.com:89.16.174.65:300

#
# A record for MX 
#
+mx.my-brilliant-site.com:89.16.174.65:300

#
# The domain name itself -- AAAA record and reverse. 
#
6my-brilliant-site.com:200141c80001596d0000000000000065:300

#
# Useful aliases -- AAAA records only
#
3ftp.my-brilliant-site.com:200141c80001596d0000000000000065:300
3www.my-brilliant-site.com:200141c80001596d0000000000000065:300
3mail.my-brilliant-site.com:200141c80001596d0000000000000065:300

#
# AAAA record for MX
#
3mx.my-brilliant-site.com:200141c80001596d0000000000000065:300

#
#  MX record -- no IP defined, as this is done separately above. 
#
@my-brilliant-site.com::mx.my-brilliant-site.com:15:300

In addition to these records for each domain, a wild-card A record is needed for the hostname such that the .testing. prefix works. If your machine is at Bytemark, this has already been setup for your machine’s Bytemark alias, for example example.default.bytemark.uk0.bigv.io.

If your machine is not hosted at Bytemark, or your hostname does not end in bytemark.co.uk then you will need to set this alias up. Adding the following line to your DNS file will work, assuming the domain is hosted at Bytemark. This assumes that your machine is called host.example.com and that your machine’s IP address is 1.2.3.4.

+*.host.example.com:1.2.3.4

Symbiosis allows adding a custom TTL to a domain. If you’re unfamiliar, you can read more about time to live (TTL) here. You can configure this by creating the file :

/srv/my-brilliant-site.com/config/ttl

The contents of the file should be a number, and it represents the time a name server can cache the record in seconds. A lower TTL is good for making frequent changes, as clients won’t cache for too long. A longer TTL is good for times when DNS is unavailable for some reason.

There’s also an easy way to add a DMARC policy on a per-domain basis. If you’re unfamiliar with DMARC, Wikipedia has an article . It provides indication that emails are protected by SPF and/or DKIM. It can be configured by creating a file in the format :

/srv/my-brilliant-site.com/config/dmarc

You may leave this as an empty file, and Symbiosis will use its defaults. If you prefer, the file can contain your own DMARC string.

If you wish to move your domains between two machines running Symbiosis and using the Bytemark content DNS service, you must contact Bytemark Support to arrange the domain to be moved between content DNS accounts.

This results from the necessity for ensuring that only people with the proper authorisation can change live DNS data. Once a domain has been hosted on our network, a content DNS account will have sole authority for it.

If you purchase a second server and move some of your domains onto it, or purchase a domain from another Bytemark customer you must contact us to move authority for the domain into the correct account.

Until this is done, although the Symbiosis system will be creating and uploading data it will not be to the account with the authority to make the data live.

SPF and DKIM are standards that help mail servers decide if email is legitimate, ensuring it is more likely to reach the intended recipient’s inbox instead of being rejected or marked as spam. Both these technologies require creation of one or more DNS records.

The crontab can also be tested. To do this you have to SSH to the machine, usually as admin to run the command.

For example, to test the my-brilliant-site.com crontab navigate to /srv/my-brilliant-site.com/config/ and run symbiosis-crontab --test crontab.

The my-brilliant-site.com crontab reads

# Send any output to Bob
#
MAILTO=bob@my-brilliant-site.com

#
# run at 18:40 every day
#
40 18 * * *       echo Hello Dave.

#
# run at 9am every Monday - Friday
#
0 9 * * mon-fri wget http://www.my-brilliant-site.com/cron.php

#
# Run once a month
#
@monthly          /usr/local/bin/monthly-job.sh

Therefore the output generated is

 Environment
 ------------------------------------------------------------------------
 HOME = /srv
 LOGNAME = admin
 PATH = /usr/local/bin:/usr/bin:/bin
 MAILTO = bob@my-brilliant-site.com
 ========================================================================

 Jobs next due -- Local time 2010-06-17T17:57:37+01:00
 ------------------------------------------------------------------------
 Date                       Command
 ------------------------------------------------------------------------
 2010-06-17T18:40:00+01:00  echo Hello Dave.
 2010-06-18T09:00:00+01:00  wget http://www.my-brilliant-site.com/cron.php
 2010-07-01T00:00:00+01:00  /usr/local/bin/monthly-job.sh
 ========================================================================

There are various automated tasks which are executed upon a Symbiosis system. These scheduled tasks are responsible for automating things such as:

The following section document precisely which jobs are installed by default, along with their purpose.

These are the system tasks which are installed by default:

There are two ways to do this, either using the MySQL command line tool, or via phpMyAdmin. This section will cover doing it with the latter.

You should now be able to access the MySQL database remotely, using this new username and password.

In Symbiosis the Backup2l configuration is generated from the snippets in /etc/symbiosis/backup.d/conf.d/.

Additionally we’ve configured the backup software to easily execute a number of scripts before and after the backup is performed:

To list the contents of your backup area you need to run backup2l with the "-l" flag:

all.1: /etc/.pwd.lock
all.1: /etc/GeoIP.conf.default
all.1: /etc/X11/Xresources/x11-common
all.1: /etc/X11/Xsession
all.1: /etc/X11/Xsession.d/20x11-common_process-args
all.1: /etc/X11/Xsession.d/30x11-common_xresources
all.1: /etc/X11/Xsession.d/40x11-common_xsessionrc
all.1: /etc/X11/Xsession.d/50x11-common_determine-startup
...

Here you will see the contents of the /etc/ directory which have been archived.

If you’d like to restrict this view you can apply a regular expression to filter the results. For example we can list the files which match the pattern passwd with this command:

~$ sudo backup2l -l passwd
Listing locations...
all.1: /etc/exim4/passwd.client
all.1: /etc/passwd
all.1: /etc/passwd-
all.1: /etc/phpmyadmin/htpasswd.setup
all.1: /etc/pure-ftpd/pureftpd.passwd
...

To illustrate how this works, an example is used. We’re looking for a backup of the file /etc/passwd.

That has restored the file to etc/passwd in the current directory. It is not recommended to run this program in the / directory, as any existing files will get overwritten.

It is also possible to pick which version of a file you wish to restore.

We have now successfully restored the file to etc/passwd in the current directory. We can check by running ls -la etc/

total 16
drwxr-xr-x  2 root  root  4096 2008-09-09 09:56 .
drwxr-xr-x 14 root  root  4096 2008-09-09 09:51 ..
-rw-r--r--  1 root  root  1118 2008-06-19 11:29 passwd

The Symbiosis system assumes that it has access to an associated external storage area. It will try and use rsync to upload the backups to this area, via a script named /etc/symbiosis/backup.d/post-backup.d/99-upload-backup.

If the host is on Bytemark’s network, this script can establish the backup space name automatically. Otherwise you can specify it manually by setting the full rsync path in /etc/symbiosis/dns.d/backup.name.

Before each backup a second script will synchronise the remote backup space locally, ensuring that a complete set of backups are held in both places. This means that if disaster strikes your machine, it is straightforward to recover your backups. This is done by running /etc/symbiosis/backup.d/pre-backup.d/00-download-backup.

This also helps to maintain the integrity of the differential backups provided by backup2l by replacing any files accidentally removed from the local backup directory before the backup starts.

It is possible to reduce the size of the backups stored locally. The first thing to do is check the current status of the backups by running sudo backup2l -s. This will present a summary of the current backups. For example:

backup2l v1.5 by Gundolf Kiefer

Summary
=======

Backup       Date       Time  |  Size   | Skipped  Files+D |  New  Obs. | Err.
- ----------------------------------------------------------------------------
all.1        2010-08-10 02:52 |   41.7M |       0     3836 | 3836     0 |    0
all.11       2010-11-01 04:45 |   38.1M |       0     3935 | 1517  1418 |    0
all.12       2011-01-21 04:27 |   39.7M |       0     3985 |  561   511 |    0
all.121      2011-01-30 04:38 |   10.5M |       0     4001 |  137   121 |    0
all.122      2011-02-08 03:54 |    1.5M |       0     4029 |  129   101 |    0
all.123      2011-09-07 05:08 |   33.8M |       0     3892 | 1437  1574 |    0
all.124      2011-09-16 05:07 |    1.3M |       0     4791 |  956    57 |    0
all.125      2011-09-25 04:45 |    868K |       0     5676 |  928    43 |    0
all.126      2011-10-04 05:15 |   11.3M |       0     6559 |  990   107 |    0
all.127      2011-10-13 04:29 |    894K |       0     7444 |  928    43 |    0
all.128      2011-10-22 04:59 |    345K |       0     8329 |  935    50 |    0
all.13       2011-10-31 05:03 |   45.7M |       0     9218 | 6833  1600 |    0

Filesystem            Size  Used Avail Use% Mounted on
/dev/vda               10G  1.9G  7.6G  20% /

From here it is possible to see which levels of backups that can be pruned. In the above example the third-level backups all.121 to all.128 can be pruned, as there has been a subsequent second level backup, all.13. The downside of this is that any changes contained in those backups will be lost, and only changes from the all.12 will be available.

To prune these backups run sudo backup2l -p 121. This will then show Backup2l removing all.121 and all its dependent backups.

backup2l v1.5 by Gundolf Kiefer

Purging <121>...
  removing <all.121>
  removing <all.122>
  removing <all.123>
  removing <all.124>
  removing <all.125>
  removing <all.126>
  removing <all.127>
  removing <all.128>

Finally we need to make sure these deletions are synchronised to the remote backup space, to ensure that our deleted files do not mysteriously return again prior to the next backup run.

sudo /etc/symbiosis/backup.d/post-backup.d/99-upload-backup

Which will provide output similar to that shown below.

Sending backups to example.backup.bytemark.co.uk::example/example.default.bytemark.uk0.bigv.io...
building file list ... done
deleting localhost/all.lock
deleting localhost/all.128.tar.gz
....
deleting localhost/all.121.error.gz
deleting localhost/all.121.check
localhost/

sent 2.95K bytes  received 22 bytes  1.98K bytes/sec
total size is 400.59M  speedup is 134742.36

Those level three backups will no longer exist.

The configuration is kept in /etc/symbiosis/backup.d/conf.d. If you need to make changes, you should make them to the files in that directory and then run make to generate the live configuration file.