Direct, Ad Hoc Deployment

This second step boils down to executing (as a normal user - the sudo permissions will be requested only whenever necessary) our deploy-us-web-native-build.sh script, with no specific argument.

Deployment for Systemd Integration (recommended)

If relying on systemctl (hence on our us-web-as-native-build.service file), the following procedure shall be preferred for this second step:

# Get ready first (as a regular user; sudo permissions to be
# requested only whenever necessary):
#
$ ./deploy-us-web-native-build.sh --no-launch

# Useful, as Systemd shall detect the update of the US-Web service file:
$ sudo systemctl daemon-reload

# Then stop properly any prior US-Web instance:
$ sudo systemctl stop us-web-as-native-build.service

# If EPMD still thinks a previous US-Web instance is running:
# (unlikely to be needed, and could impact any other running
# Erlang program, so be cautious...)
#
#$ sudo killall epmd

# Relaunch US-Web, and initiate the certificate renewal process:
$ sudo systemctl start us-web-as-native-build.service

Then, as a first check of that launch, one may execute:

$ sudo systemctl status us-web-as-native-build.service

See the next section for more in-depth checking and monitoring.

Checking Launch Outcome

If having requested the renewal of SSL certificates (the certificate_support entry of the US-Web configuration file being set to renew_certificates), incoming HTTP requests will be directly served yet will be redirected to HTTPS ones that will complete successfully only once proper certificates will have been obtained (hence after a few dozens of seconds after server startup).

These certificates can be checked (e.g. in terms of timestamps) in the certificates subdirectory of the one designated by the us_web_data_dir entry (typically in /opt/universal-server/us_web-data/certificates).

Finally, from a client host, one may also check the availability of the target webserver(s); refer to the Remote Monitoring section for that.

US-Web Application Base Directory

This directory allows US-Web to select the various runtime elements it is to use.

This US-Web Application Base Directory may be set in the US-Web configuration file thanks to its (optional) us_web_app_base_dir entry.

If not set there, US-Web will try to use the US_WEB_APP_BASE_DIR environment variable instead.

If not usable either, US-Web will try to determine this directory automatically, based on how the server is launched (with our native deployment approach or as an OTP release).

Of course failing to resolve a proper application base directory will result in a fatal launch error to be reported at start-up.

Let's designate this directory from now on with the US_WEB_APP_BASE_DIR pseudo-variable.

Separating the US-Web Software from its data

It is useful to regularly upgrade the US-Web code while maintaining the data it uses as it is (not losing it because of a mere software update).

This notably includes the information such as the state of the log analysis tool (i.e. the synthesis aggregated after having processed a potentially longer history of access logs when rotating them) or the currently generated certificates (which are not always easy to renew frequently, if having to respect rate limits) [4].

These various states are stored in the US-Web data directory, which can be set, by decreasing order of priority:

• in the us_web_data_dir entry of the US-Web configuration file, as an absolute directory, or as one that is relative to the US-Web application base directory (see US-Web Application Base Directory), then as US_WEB_APP_BASE_DIR/data
• otherwise in the US_WEB_DATA_DIR environment variable

If the default path for this location is us-web-data (relatively to the US-Web instance base directory, typically deployed in /opt/universal-server/us_web-native), a recommended practice is nevertheless to set this location explicitly outside of that instance so that the instance can be upgraded/replaced without any loss of state.

So the us_web_data_dir entry might be set in the US-Web configuration file for example to /opt/universal-server/us_web-data. Of course, as always, proper permissions shall be set in this data tree as well.

Let's designate this directory from now on with the US_WEB_DATA_DIR pseudo-variable.

This directory may notably gather following elements:

• the log-analysis-state subdirectory (if log analysis is enabled), containing the working state of the tool for web access analysis, typically Awstat, thus containing awstats*.baz.foobar.org.txt files for the preprocessing of its reports
• the certificates subdirectory (if certificate support is enabled), containing the related keys and certificates for the domains of interest:
  • if managing domains D in ["foobar.org", "example.com"], then D.csr, D.key and D.crt will be stored or produced there (refer to this section of the documentation of LEEC for further explanations)
  • LEEC-related transverse security elements may be found there (if using it to renew certificates) as well, notably (see this LEEC documentation section):
    • us-web-leec-agent-private.key, the RSA private key of the US-Web LEEC agent so that it can safely authenticate to the ACME servers it is interacting with and afterwards validate https handshakes with clients
    • lets-encrypt-r3-cross-signed.pem, the certificate associated to the Certificate Authority (Let's Encrypt here) that is relied upon
  • dh-params.pem, a key generated by LEEC to allow for safer Ephemeral Diffie-Helman key exchanges

One may create from this certificates subdirectory a GNUmakefile symlink pointing to GNUmakefile-for-certificates to easy any checking or backup of these certificates.

Catch-alls, HTTPS, Challenge Types and Wildcard Certificates

Having wildcard certificates (which is the default now) - thus relying on the dns-01 challenge - allows in turn to define catch-alls (host-level and/or domain-level) routes and wildcard DNS entries (such as * IN CNAME foobar.org.).

Conversely, if setting another challenge type (refer to the challenge_type configuration entry, typically then set to the http-01 challenge), defining catch-all routes is not recommended if using https, as these unanticipated hostnames are by design absent from the corresponding certificates, and thus any user requesting them (possibly due to a typo) will have their browser report a security risk (such as SSL_ERROR_BAD_CERT_DOMAIN).

Similarly, wildcard DNS entries should be then avoided (on the other hand, they allow not to advertise what are the precise virtual hostnames supported; note though that with https, these virtual hostnames will be visible in the SANs).

Other Configuration Files

Regarding authbind

Many distributions will parameter authbind so that only the TCP port 80 will be available to "authbound" programs.

If wanting to run an HTTPS server, the TCP port 443 will be most probably needed and thus must be specifically enabled.

For that, relying on the same user/group conventions as before, one may enter, as root:

$ cd /etc/authbind/byport
$ cp 80 443
$ chown us-web-user:us-group 443

Otherwise an exception will be raised (about eperm / not owner) when US-Web will try to create its HTTPS listening socket.

Checking EPMD

If using the default US-Web EPMD port, checking whether an instance is running is as simple as:

$ export ERL_EPMD_PORT=4508; epmd -names
epmd: up and running on port 4508 with data:
name us_web at port 50002

Then executing kill-us-web.sh will kill any live US-Web instance and unregister it from its EPMD (without killing any EPMD daemon).

Overall Local Inquiry

The get-us-web-native-build-status.sh script (still in priv/bin, as all US-Web shell scripts) may then be used to investigate any problem in a streamlined, integrated way.

Alternate (yet often less convenient) solutions are to run systemctl status us-web-as-native-build.service or, often with better results, journalctl -xe --unit us-web-as-native-build.service --no-pager to get some insights.

General Logs

A deeper level of detail can be obtained by inspecting the general logs (as opposed to the webserver ones, discussed in next section), which regroup the VM-level, technical ones and/or the applicative ones.

VM logs are written in the ${REL_BASE_ROOT}/log directory (e.g. /opt/universal-server/us_web-latest/us_web/log), which is created when the release is started first. Generally, run_erl.log (if launched as a daemon) will not help much, whereas the latest Erlang log file (ls -lrt erlang.log.*) is likely to contain relevant information.

So one may consult them for example with:

$ tail -n 50 -f /opt/universal-server/us_web-latest/us_web/log/erlang.log.1

As for our higher-level, applicative traces, they are stored in the the us_web.traces file, in the directory defined in the us_web_log_dir entry of the US-Web configuration file. This last file is specified in turn in the relevant us.config configuration file (see the us_web_config_filename key for that), which, in development mode, is itself typically found in ~/.config/universal-server while, in production mode, is likely located in /etc/xdg/universal-server.

In practice, this trace file is generally found:

• in development mode, if testing, in priv/for-testing/log, relatively to the base directory specified in us_app_base_dir
• in production mode, generally in /var/log/universal-server of /opt/universal-server

This path is updated at runtime once the US-Web configuration file is read; so typically it is renamed from /opt/universal-server/us_web-x.y.z/traces_via_otp.traces to /var/log/universal-server/us-web/us_web.traces. If available, it is surely the most complete source of information.

Webserver Logs

These logs, which are maybe less useful for troubleshooting, designate access and error logs, per virtual host (e.g. for a virtual host VH, access-for-VH.log and error-for-VH.log). Their previous versions are archived in timestamped, compressed files (e.g. error-for-VH.Y-M-D-at-H-M-S.xz).

These files will be written in the directory designated by the us_web_log_dir entry of the US-Web configuration file. Refer to the previous section to locate this directory (for example it may be /var/log/universal-server/us-web).

HTTP Client Check

First of all, is this US-Web instance available at all? Check with:

$ wget http://baz.foobar.org -O -

One may define one's dedicated convenience script for that, like a check-baz-website-availability.sh script whose content would be:

#!/bin/sh
target_url="http://baz.foobar.org"
echo "  Checking whether '${target_url}' is available:"
wget ${target_url} -O - | head

More elaborate scripts may be devised, testing various virtual hosts with various protocols (http/https), looping continously should a failure be detected, like in:

#!/bin/sh
test_url()
{
   target_url="$1"
   #echo "  Checking whether '${target_url}' is available:"
   if wget -q "${target_url}" -O - 1>/dev/null; then
       echo " ('${target_url}' is available)"
       return 0
   else
       echo "######### Problem: '${target_url}' is NOT available! #########" 1>&2
       return 1
   fi
}

# Testing base domains, some virtual hosts, and HTTP/HTTPS availability as
# well:
com_test_url="http://foobar.com"
# Testing also a virtual host:
com_vh_test_url="http://hello.foobar.com"
# As HTTPS:
com_vh_https_url="https://other.foobar.com"
com_urls="${com_test_url} ${com_vh_test_url} ${com_vh_https_url}"
# Testing also the other domains:
net_test_url=[...]
all_urls="${com_urls} ${net_urls} ${org_urls} ${info_urls} ${biz_urls} [...]"
all_good=1
while [ $all_good -eq 1 ]; do
   all_good=0
   for u in ${all_urls}; do
       if ! test_url $u; then
           all_good=1
       fi
   done
   echo
   if [ $all_good -eq 1 ]; then
       echo "At least an URL could not be reached, testing again..."
   fi
   sleep 2
done
echo "Good news, all tested domains are available from this local host!"
echo

Trace Listener Check

As for the applicative traces, they may be monitored remotely as well, thanks to the monitor-us-web.sh script.

This script operates based on a configuration file, us-web-for-remote-access.config (typically located in ~/.config/universal-server), that allows to designate the US-Web instance to access for the client host, and the various settings for that.

For example:

% This file is used by the US-Web framework.
%
% This is a configuration file to enable the interaction with a remote US-Web
% instance (e.g. a production server to be reached from a client host), typically
% to monitor traces, to trigger the generation of log access reports, etc.

% The hostname where the US-Web instance of interest runs:
{us_web_hostname, "baz.foobar.org"}.

% Its cookie, as set in its configuration file:
{remote_vm_cookie, 'foobar-8800a3f8-7545-4a83-c925-ca115a7b014b'}.

% If specifically overridden for thay server:
{us_web_config_server_registration_name, foobar_prod_us_web_cfg_srv}.

% A range of TCP ports that may be used for out-of-band inter-VM communication
% (not using the Erlang carrier; e.g. for send_file):
%
% (this range must be typically in-line with any one set at the level of
% firewalls)
%
{tcp_port_range, {50000, 55000}}.

Then running the aforementioned monitor-us-web.sh script should launch a trace browser synchronised to the trace aggregator of the targeted US-Web instance, displaying all traces (i.e. both all that have been emitted since the start of that instance and all newer ones, being then displayed in direct).

Here is a user-submitted screenshot (edited for confidentiality) of the LogMX-based trace browser corresponding to the native US-Web trace interface (refer to Ceylan-Traces for further details):

Deployment of a Nitrogen-enabled US-Web

US-Web must be deployed accordingly, typically with the --support-nitrogen option of the deploy-us-web-native-build.sh script [6].

Deployment of the Nitrogen-based website / application

Then of course a Nitrogen-based website must have already been developed, an example of which being, in US-Main, the us_main_nitrogen_app web application.

Such a website is typically obtained by cloning the official Nitrogen repository (or possibly our fork thereof) and running:

$ make slim_cowboy PREFIX=$MY_NITROGEN_APP_ROOT PROJECT=my_nitrogen_app

(we prefer relying on a separate system-wide Erlang installation rather than on an full OTP release)

The root tree of interest then lies in $MY_NITROGEN_APP_ROOT/my_nitrogen_app/site.

The whole of it may be put in VCS, or at the very least its include, src, static/{css,images}, templates subtrees.

Operating directly from within the rebar build tree

(not recommended in a development phase)

If having modified and recompiled a Ceylan prerequisite (e.g. WOOPER), then, before generating the release, run from its root (e.g. us_web/_build/default/lib/wooper):

$ make rebar3-copy-beams REBAR_BUILD_DIR=../../

Or, simply run make rebar3-compile REBAR_BUILD_DIR=../../ (and no need to run make release afterwards).

Operating from _checkouts build trees

(recommended in a development phase, as a lot more flexible/unified than the previous method)

Create a us_web/_ckeckouts directory containing symbolic links to repositories of dependencies (e.g. myriad) that may be updated locally; or, preferably, execute the create-us-web-checkout make target to automate and streamline this step.

Batch Mode

One can update us_web/conf/sys.config in order to toggle batch mode (e.g. to enable/disable a graphical trace supervision) after build time.

It deals with the configuration of the Traces application, so it comes very early at start-up (at application boot, hence before US-Web specific settings can be read, so this option would be difficult to put in the US configuration files).

Location of Applications

The location of the US-Web application is bound to differ depending on the context of use (development/production deployments, host-wise, etc.), whereas it is needed to be known at the very least by its start/stop scripts.

So its path must be specified (most preferably as an absolute directory) in the US-Web configuration file. However, at least for test configuration files, relying on an absolute directory would not be satisfactory, as the user may install the US-Web framework at any place and testing should not require a manual configuration update.

As a result, should an application location (e.g. US-Web) not be specified in the configuration, it will be looked-up in the shell environment (using the US_WEB_APP_BASE_DIR variable) for that and, should this variable not be set, as a last-resort an attempt to guess that location will be done.

System-related Hints

Besides the use of specific, unprivileged user/group thanks to the use of authbind (set with a minimal depth level), the system resources (notably the maximum number of file descriptors) are reported (among the traces; search for System description) at the startup of the US-Web server.

Moreover the launch scripts (namely start-us-web-native-build.sh and start-us-web-release.sh) raise the limit in terms of maximum number of file descriptors opened by the US-Web process, so that the connection acceptor does not have to reduce its accept rate (typically because of rogue bots performing pseudo-denials of service).

Finally, any webserver shall rely preferably on: - a sufficiently paranoid firewall (see section in Execution Hints) - an uninterruptible power supply (UPS), protecting the server and the telecom appliances involved

Web-related hints

• most paths (e.g. default_web_root, in the US-Web configuration) can be defined as relative ones (mostly useful for embedded tests; otherwise absolute paths shall be preferred); in this case they will be relative to the runtime current directory, typically [...]/us_web/_build/default/rel/us_web/ in development mode
• the default_domain_catch_all atom allows to designate any domain-level host (e.g. foobar.org) that did not match previous host rules
• in the context of a given host (e.g. foobar.org), the default_vhost_catch_all atom allows to designate any of its virtual hosts (e.g. bar, to be understood as bar.foobar.org) that did not match previous path rules [8]
• refer to us_web/priv/for-testing for an example setup and configuration files
• the web roots shall be owned by the user running US-Web (e.g. chown -R us-web-user:us-group /opt/www)

In terms of (UNIX) Processes

A running US-Web server will not be found by looking up beam or beam.smp through ps; as an OTP release, it relies first on the run_erl launcher, like shown, based on ps -edf, in:

UID         PID    PPID  C STIME TTY TIME     CMD
us-web-user 767067    1  0 Feb15 ?   00:00:00 /usr/local/lib/erlang/erts-x.y/bin/run_erl
  -daemon /tmp/erl_pipes/us_web@MY_FQDN/ /opt/universal-server/us_web-US_WEB_VERSION/log
  exec "/opt/universal-server/us_web-US_WEB_VERSION/bin/us_web" "console" ''
  --relx-disable-hooks

This can be interpreted as:

• not running as root, but as a dedicated, weakly privileged user
• its parent process (PPID) is the first overall process (as a daemon)
• STIME is the time when the process started
• no associated TTY (runs detached in the background)

This launcher created the main, central, us_web (UNIX) process, parent of all the VM worker (system) threads.

pstree -u (or ps -e --forest) tells us about the underlying process hierarchy:

[...]
  |-run_erl(us-web-user)---beam.smp---erl_child_setup---inet_gethost---inet_gethost
  |                             |-158*[{beam.smp}]
[...]

The 158 threads must correspond to:

• 128 async ones (-A 128)
• 30 "normal" threads (on a computer having a single CPU with 8 cores with Hyperthreading, hence 16 logical cores)

Using htop, one can see that the run_erl process spawned a us_web one (namely /opt/universal-server/us_web-US_WEB_VERSION/bin/us_web) that is far larger in (VIRT) memory (e.g. 5214MB, instead of 5832KB for the former).

us_web in turn created the numerous threads.

RSS/RSZ (Resident Set Size) is a far better metric than VIRT/VSZ (Virtual Memory Size); indeed VIRT = RSS + SWP and:

• RSS shows how much memory is allocated to that process and is in RAM; it does not include memory that is swapped out; it includes memory from shared libraries (as long as the pages from those libraries are actually in memory), and all stack and heap memory used
• VIRT includes all memory that the process can access, including memory that is swapped out, memory that is allocated but not used, and memory that is from shared libraries

Knowing that, with ps, one may add -L to display thread information and -f to have full-format listing, a base command to monitor the US-Web processes is: ps -eww -o rss,pid,args | grep us_web, with:

• -e: select all processes
• -w (once or twice): request wide output
• -o rss,pid,args: display RSS memory used (in kilobytes), PID and full command line

(apparently there is no direct way of displaying human-readable sizes)

See also our list-processes-by-size.sh script; typical use:

$ list-processes-by-size.sh
   Listing running processes by decreasing resident size in RAM (total size in KiB):
 RSS  PID      COMMAND
[...]
1204  1695242  /usr/local/lib/erlang/erts-11/bin/run_erl -daemon /tmp/erl_pipes/us_web@somehost.foobar.org/ /opt/universal-server/us_web-x.y.z/log exec "/opt/universal-server/us_web-x.y.z/bin/us_web" "console" ''  --relx-disable-hooks
67776 1695243 /opt/universal-server/us_web-x.y.z/bin/us_web -A 128 -- -root /opt/universal-server/us_web-x.y.z -progname opt/universal-server/us_web-x.y.z/bin/us_web -- -home /home/us-web-user -epmd_port 4526 -- -boot /opt/universal-server/us_web-x.y.z/releases/x.y.z/start -mode embedded -boot_var ERTS_LIB_DIR /usr/local/lib/erlang/lib -config /opt/universal-server/us_web-x.y.z/releases/x.y.z/sys.config -name us_web -setcookie us_web_initial_cookie -- -- console --relx-disable-hooks --
[...]

This confirms that, even if higher VIRT sizes can be reported (e.g. 5214M, hence roughly 5GB), the RSS size may be 67776 (KB, hence 66 MB), i.e. very little, and does not vary much.

Indeed, in terms of RSS use (for a few domains, each with a few low-traffic websites, if that matters), we found:

• at start-up: only about 67MB
• after 5 days: just 68 MB
• after 24 days: 76 MB
• after 55 days: 80-88 MB

A more recent server instance is using, after more than 70 days, 60 MB of RSS memory, whereas after 30 days another ones was 108 MB. Anyway quite low.

Trace Monitoring

Use the us_web/priv/bin/monitor-us-web.sh script in order to monitor the traces of an already running, possibly remote, US-Web instance.

Note that the monitored US-Web instance will be by default the one specified in any us-monitor.config file located in the US configuration directory.

One may specify on the command-line another configuration file if needed, such as us-monitor-for-development.config.

Node Test & Connection

If desperate enough, one may also run, possibly from another host, and based on the settings to be found in the configuration files:

$ ERL_EPMD_PORT=XXXX erl -name tester -setcookie CCCC -kernel inet_dist_listen_min MIN inet_dist_listen_max MAX

1> net_adm:ping('us_web@foobar.org').
pong

Then one may switch to the Job control mode (JCL) by pressing Ctrl-G then r to start a remote job on the US-Web node.

Objective

When running a reference server for a set of given domains, one should avoid stopping US-Web, updating its code and attempting to relaunch it: not only because this may imply a non-negligible downtime, but also, more importantly, because, should a failure happen, it would have to be fixed live, in the hurry, whereas no one can access anymore any of the hosted sites. Not a convenient situation.

A better, safer (but not simpler) approach is instead to install one's newer server instance as a test one on the target host, build it and test it on a temporary setting (on an auxiliary TCP port) and, if and only if the operation is fully satisfactory, to switch the same setting on the actual port (once the prior reference instance has been shut down); if on the contrary something goes wrong with the test, it can be safely fixed out of the critical path, not implying a specific downtime.

Test Installation

For such a test bed to exist, the usual installation step shall be followed by a specific, separate configuration step.

Installing is merely a matter of transferring the deploy-us-web-native-build.sh script to one's server and to run it (as a normal user), as deploy-us-web-native-build.sh --no-launch so that it cannot interfere with any reference version running.

A check-foobar-website-availability.sh script may be useful to efficiently check the state of that reference version.

Test Configuration

A first action is to devise a corresponding configuration file and to launch a corresponding instance on dedicated TCP ports.

It will require a dedicated US-Web configuration file, let's name it foobar-us-web-for-testing.config. It will have to be referenced by a corresponding US configuration file (namely us-config), both of them expected to be in a universal-server directory in a test-specific root directory, e.g. in TEST_DIR=/opt/universal-server/us_web-data/configuration-for-testing/universal-server - thus as /opt/universal-server/us_web-data/configuration-for-testing/universal-server/{foobar-us-web-for-testing,us}.config - rather than in a standard root directory such as /etc/xdg.

The most direct/reliable approach is to copy the reference versions of these two files and modify them as little as possible.

The test us-config file should specify:

• a cookie of its own (e.g. foobar-testing-be3be613-96d8-4203-a502-73cc42a5e5e2)
• (the EPMD port may be set only in its US-Web counterpart configuration file)
• possibly different users (preferably not, as wanting to stick as close as possible to the target settings)
• updated directories (possibly all - like us_log_dir - pointing to TEST_DIR)
• the updated us_web_config_filename, like foobar-us-web-for-testing.config

This foobar-us-web-for-testing.config associated file could then specify:

• an EPMD port of its own (e.g. 4515); if using our iptables.rules-Gateway.sh script (refer to this section for more details), one may check in /etc/iptables.settings-Gateway.sh the us_{low,high}est_epmd_port variables in order to pick an EPMD port in the LAN-allowed input range, so that hosts from the LAN may connect to the test US-Web instance on the gateway if needed
• us_web_data_dir and us_web_log_dir entries pointing to the aforementioned TEST_DIR
• updated, non-clashing TCP ports, for example http_tcp_port being 8000 and https_tcp_port being 8443
• for extra caution, certificate_mode may be set to development (rather than production)
• challenge_type can be set, knowing its default is dns-01
• for routes, for the sake of testing, a single domain may be kept

Finally, if relying on a dns-01 challenge, the credentials for the DNS provider will have to be available. Creating a symlink in the test-specific root directory to, say, /etc/xdg/universal-server/leec-ovh-credentials-for-foobar.org.txt should be sufficient for that.

Firewall Configuration

A second step is, firewall-wise, to enable (temporarily) the access to the test instance.

The target EPMD port must already fall in a LAN-supported range, yet we also want to be able to properly test as if we were a random Internet user, thus connecting from the WAN.

So:

$ iptables -A INPUT -p tcp --dport 8000 -m state --state NEW,ESTABLISHED -j ACCEPT
$ iptables -A INPUT -p tcp --dport 8443 -m state --state NEW,ESTABLISHED -j ACCEPT

Test Code Updates

Tests may show that some modules have to be fixed, and thus that the server installation has to be updated accordingly (rather than be deployed again from scratch). The sync-all-to-server make target is useful to synchronise the code base of a given layer on the server from a development computer; for that the CEYLAN_SYNC_TARGET_ROOT must be set to the root path on the server; for example:

$ make sync-all-to-server CEYLAN_SYNC_TARGET_ROOT=/opt/universal-server/us_web-native

Launching the Test Server

For this use, systemctl would not be convenient. A better approach is to start one's test server explicitly, based on the configuration files that were just devised.

First of all, no prior test instance should exist, otherwise they would clash EPMD-wise. ps -edf|grep -i epmd may help.

Then the target test instance can be launched, as root:

$ cd /opt/universal-server/us_web-latest/us_web/priv/bin
$ ./start-us-web-native-build.sh /opt/universal-server/us_web-data/configuration-for-testing/universal-server

Testing it

Either the launch was reported as succeeded or not.

In the latter case, a launch time-out was reported. The log of the Erlang VM (typically /opt/universal-server/us_web-native/us_web/log/erlang.log.1) will probably give relevant information. May a prior test instance is on the way, clashing in the same EPMD instance, or regarding the target TCP ports?

In the former case, a success is assumed. The best way to confirm this is to check the overall result by ourself, with wget http://foobar.org:8000 -O -.

Further insight to be obtained based on /opt/universal-server/us_web-data/configuration-for-testing/us_web.traces.

Test Teardown

Finally, now that the test is over, we can close up the mountain with:

# The -n option is used to have port numbers rather than aliases:
$ iptables -L --line-numbers -n

$ iptables -D INPUT X

# Again, as the previous deletion decremented the number of the last line:
$ iptables -D INPUT X

Awstats Installation

On Arch Linux, one should follow these guidelines (example for version 7.8):

$ pacman -Sy --needed awstats

Awstats will then be installed in /usr/share, including the /usr/share/webapps/awstats/cgi-bin/awstats.pl script and the /usr/share/webapps/awstats/icon/ directory.

Some permission fixes (to be done as root) might be needed first:

$ chmod +r /usr/share/webapps/awstats/icon/os/*

Awstats Configuration

Log analysis will be triggered periodically by the US-Web server rather than on-demand via CGI Perl scripts, and its result, i.e. the web pages generated from the access logs, will be available in the meta website (e.g. mymeta.foobar.org; refer to Auto-generated Meta Website for more information).

More precisely, and as already mentioned, in the US-Web log directory (see us_web_log_dir), dedicated access and error log files will be generated for each known virtual host. For example the accesses to a baz.foobar.org virtual host will be written by the US-Web server in a corresponding access-for-baz.foobar.org.log file.

At server start-up, the US-Web meta module (us_web_meta) will have generated a suitable Awstats configuration file (namely awstats.baz.foobar.org.conf) that will trigger the generation of the corresponding static web pages (awstats.baz.foobar.org.*, notably awstats.baz.foobar.org.html) in the web root of the meta website.

These configuration files are now placed in /usr/local/etc/awstats (they were previously in the conf subdirectory of the root specified in us_web_app_base_dir).

Indeed, if starting from version 7.8, Awstats allows these configuration files to be specified as absolute paths, its previous versions:

• either required such configuration files to be in /etc/awstats, /usr/local/etc/awstats, /etc or in the same directory as the awstats.pl script file
• or, if the configuration files could be specified as absolute paths, the generated pages would then include some faulty links because of that

US-Web retained the most controllable, less "system" directory, /usr/local/etc/awstats. All these locations are mostly root-only, whereas the US-Web server is designed to run as a normal, non-privileged user and is to generate there these Awstats configuration files.

Such a target directory shall thus be created beforehand, and made writable by the user specified in us_web_username.

Each virtual host (say: baz.foobar.org) will have its configuration file deriving from priv/conf/awstats.template.conf, where the following patterns will be replaced by relevant ones (through keyword-based replacements):

• US_WEB_VHOST_LOG_FILE to become the full path to the corresponding access log (e.g. access-for-baz.foobar.org.log, in us_web_log_dir)
• US_WEB_VHOST_DOMAIN to become the virtual host domain (e.g. baz.foobar.org)
• US_WEB_LOG_ANALYSIS_DATA_DIR to become the directory in which the working data (e.g. state files) of the web analyzer (here Awstats) shall be written

Awstats icons are copied to the icon directory at the root of the meta website.

The Awstats database, typically located in /var/local/us-web/data, will be updated once an access log file will be rotated; just after, this log file will be compressed and archived under a relevant filename, such as access-for-baz.foobar.org.log.2020-2-1-at-19h-48m-12s.xz.

Awstats Troubleshooting

Various issues may prevent log reports to be available.

Let's try with a real US-Web uncompressed log file first (e.g. xz -d access-vhost-catchall.log.test.xz), supposing that it corresponds to a my-test virtual host).

Then configure Awstats (e.g. through a /usr/local/etc/awstats/awstats.my-test.conf file) to process that log file; for that, run on that host:

$ perl /usr/share/awstats/tools/awstats_configure.pl

Then, to debug the whole process, use, as root:

$ rm -f /usr/share/webapps/awstats/cgi-bin/awstats*.txt ; echo ;
   LANG=C /usr/share/webapps/awstats/cgi-bin/awstats.pl
     -config=my-test -showdropped

Most problems should become visible then.

To do the same for a series of web logs in the context of US-Web, one can have them analysed first thanks to:

$ for f in /usr/local/etc/awstats/awstats-*conf; do echo ;
   LANG=C /usr/share/webapps/awstats/cgi-bin/awstats.pl
     -config=$f -update ; done

Then all web reports can be generated manually with:

$ for f in /usr/local/etc/awstats/awstats-*conf; do echo ;
   LANG=C /usr/share/webapps/awstats/cgi-bin/awstats.pl
     -config=$f -output ; done

For mono-domain certificates

Rather than using a standalone ACME agent such as the standard one, certbot, we prefer driving everything from Erlang, for a better control and periodical renewal (see the scheduler provided by US-Common).

Various libraries exist for that in Erlang, the most popular one being probably letsencrypt-erlang; we forked it (and named it LEEC, for Let's Encrypt Erlang with Ceylan, to tell them apart), in order notably to support newer Erlang versions and to allow for concurrent certificate renewals (knowing that one certificate per virtual host will be needed).

Three action modes can be considered to interact with the Let's Encrypt infrastructure and to solve its challenges. As the US-Web server is itself a webserver, the slave mode is the most relevant here.

For that, the us_web_letsencrypt_handler has been introduced by US-Web.

By default, thanks to the US-Web scheduler, certificates (which last for up to 90 days and cannot be renewed before 60 days are elapsed) will be renewed every 75 days, with some random jitter added to avoid synchronising too many certificate requests when having multiple virtual hosts - as they are done concurrently, if not SANs are used.

For wildcard certificates

US-Web relies here also as much as possible on LEEC (see this section), even if, at least currently, the standard certbot is used internally; here US-Web, thanks to its US-Common Scheduler, acts mostly like an integrated crontab on steroids.

Purpose

The goal of a reverse proxy is to have a (proxy) server seamlessly (read: without doing a HTTP forward - thus with unchanged URLs from the point of view of the client) redirect incoming traffic to other "proxied" webservers (possibly other US-Web instances) running on other hosts and/or ports.

A typical use case here [14] is having an US-Web instance serve a domain name of interest (e.g. foo.org) whereas more than one of its virtual hosts are Nitrogen-based, knowing that no two of such virtual hosts shall be executed by the same (Erlang) VM (as notably their Nitrogen page modules would clash).

For example a reverse proxy could ensure that connecting to the https://my-server.foo.org (virtual) host is transparently forwarded to http://foo.org:8150 (a port which could be filtered by the local firewall - thus with no possible direct access from the outside network).

So this latter, autonomous webserver my-server instance would run thanks to its own VM on TCP port 8150, yet would only by reachable by connecting to the former, centralising webserver, which would act as a reverse proxy. As many other virtual hosts as wanted could be declared (e.g. as https://my-other-server.foo.org) to the reverse proxy, each pointing then to its own actual webserver instance (e.g. http://foo.org:8151).

Challenges

The HTTP/HTTPS routing shall be fully transparent, notably not breaking any connectivity (e.g. regarding Comet, AJAX, Websockets) and complying with the proper use of the underlying protocols (e.g. so that browsers do not interpret a proxying as the spoofing of SSL certificates).

Implementations

There are many solutions in order to obtain a reverse proxy; it may be done:

• directly at the firewall level, typically by configuring iptables accordingly
• thanks to nginx, as a NGINX Reverse Proxy (probably the most popular option)
• at the Erlang-level: see for example cowboy_revproxy, Surrogate or yaws_revproxy

US-Web Reverse Proxy

For better control US-Web will implement its own reverse proxy.

For that it will automatically modify header fields (e.g. Host, Connection, X-Real-IP; the source IP could also by set) in the proxied requests.

The responses from proxied servers could be buffered until they are fully received - to better manage slower clients - yet, as here the proxied servers may offer rather interactive uses, buffering may not be desirable.