Installing Unscrambl Qbo Decisions¶

Introduction¶

Unscrambl Qbo Decisions is an on-premise next generation enterprise marketing automation product. Built for marketers, it enables marketing teams to plan and manage the customer lifecycle end-to-end and to deliver personalized campaigns quickly moving from micro segments to a segment of one.

It runs on RedHat 7 (or CentOS 7) and RedHat 8 (or CentOS 8).

In the next sections, we will go over the software dependencies that must be satisfied prior to the installation of Unscrambl Qbo Decisions.

Dependencies¶

The server where Unscrambl Qbo Decisions is installed must be configured according to a few guidelines. The software also relies on a set of external dependencies that must be installed prior to setting it up for use. In this section, we discuss both of these issues.

Operating System configuration¶

The server (or servers) where Qbo Decisions runs must be properly configured as far as certain OS and shell resource limits.

While these limits are going to be checked during installation, we will provide certain helpful guidelines in this section.

First, the number of file descriptors that can be used by a process must be equal to or greater than 640,000.

The soft and hard limits are defined as part of the OS configuration (defined in /etc/security/limits.conf) and used/enforced by the (Bash) shell.

In the shell, the soft limits for each of these resources (including the maximum number of open files) can be inspected as follows:

ulimit -Sa

And the hard limits as follows:

ulimit -Ha

Naturally neither limit can go beyond the OS limit that applies to the server.

If the current limit is too low for the current shell, but sufficient as far as the OS is concerned, you can update your local .bashrc raising the limit as follows:

ulimit -n 640000

Nevertheless, in most cases, after a fresh OS install, it is necessary to update /etc/security/limits.conf to meet Qbo Decisions’s needs.

We recommend that you consult your OS documentation, but usually the following settings can be added towards the end of the file (assuming you want the limits applied to all users):

*    soft    nofile 640000
*    hard    nofile 640000

# End of file

After applying this change, a new login must be made with the userid that will be used by Qbo Decisions (no reboot is necessary).

External Dependencies¶

The external software dependencies are required by Qbo Decisions are specific to the operating system version and architecture where the software will run.

There are two types of external dependencies. The ones provided by the OS vendor and the ones from external vendors.

Starting with the dependencies provided by external vendors, requiring a manual installation, you will need to obtain:

optional: IBM InfoSphere Streams, IBM SPSS, as well as Oracle WebLogic, if these are capabilities you have acquired in your particular Unscrambl Qbo Decisions installation. These are commercial products and you should consult their respective documentation to have them installed in your environment. Their integration with Unscrambl Qbo Decisions is described later in this section, where the installer will request additional information regarding their location in the file system.

Installing the OS-managed packages¶

When it comes to OS-managed dependencies, i.e., dependencies that can be installed using the operating system package management, the installer will look for and warn you about missing dependencies.

The installation package comes with a utility, dependency_checker, that can be used to ensure that all dependencies are in place ahead of the installation.

This utility inspects the environment for RedHat- or Ubuntu-provided software (referred to as OS-provided software in the rest of this documentation) as well as for specific Python packages required by Unscrambl Qbo Decisions, which are provided as a virtualenv environment, pre-configured to match the Unscrambl Qbo Decisions needs.

Operating System-software packages must be installed using the regular mechanism employed to download and install them, usually yum on RedHat and apt-get on Ubuntu.

When using dependency_checker to extract the list of required dependencies, the output will be similar to (but not necessarily the same as) the following:

$ ./drive/bin/dependency_checker -l
List of OS package dependencies:

advance-toolchain-at8.0-runtime: 8.0 (installed)
mariadb: 5.5 (not installed)
mariadb-libs: 5.5 (installed)
mariadb-server: 5.5 (installed)

List of Python package dependencies (available in the Unscrambl Qbo Decisions virtualenv):
...

Note that, in this example, one external dependency (mariadb) is not currently installed. In this case, assuming this host is running RedHat Linux, yum must be used to install mariadb.

Note

Installing OS packages on a server without Internet connection

In many cases, the server (or cluster) where Unscrambl Qbo Decisions is going to be installed is not directly connected to the Internet.

In such cases, the installation of additional OS-level packages can be accomplished by having access to the Operating System installation CD/DVD or, simply, to an .iso image with the OS installation.

If your installation is RedHat-based, use one of the following alternatives:

if a DVD is available, please follow the DVD-based yum repository directions outlined by RedHat to create a locally available yum repository.
if an .iso file is available, please follow these .iso file directions to create a locally available yum repository.

If your installation is Ubuntu-based, use one of the following alternatives:

if a DVD is available, please follow the DVD-based apt repository directions outlined by Canonical to create a locally available aptitude repository.
if an .iso file is available, mount it first and then use the mounting point in the steps above as the location when running apt-cdrom. To mount the .iso perform the following steps either sudo-ed or by logging in as root:
```
$ mkdir -p <mounting point location>
$ mount -o loop <file>.iso <mounting point> location
```

When installing external operating system-managed dependencies, as long as the major and minor version numbers match, the dependency is considered satisfied.

Configuring MariaDB for use by Unscrambl Qbo Decisions¶

Unscrambl Qbo Decisions requires a relational database to store configuration as well as runtime data used by the applications as it runs.

Currently, the only supported database is MariaDB.

Installing MariaDB¶

If you need to have it installed prior to installing Unscrambl Qbo Decisions, please refer to MariaDB’s online installation instructions to become acquainted with both the installation and configuration process. The actual installation is done via apt-get or yum, depending on whether you are installing it on Ubuntu or RedHat.

MariaDB is executed as a service on both Ubuntu as well as on RedHat and its behavior is controlled by a configuration file (usually located in /etc/my.cnf.d/server.cnf on RedHat or /etc/mysql/my.cnf on Ubuntu). In this file, specifically in the mysqld section of the configuration, the following entry must be commented out or removed:

bind-address = 127.0.0.1

This change will ensure that other hosts in the cluster can interact with the MariaDB server.

If the version for the MariaDB server is older than 10.3.1, the InnoDB engine must also be appropriately configured with the following settings in the [mysqld] section:

innodb_file_format=Barracuda
innodb_file_per_table=ON
innodb_large_prefix=ON

The configuration modifications only become active after a server restart, which requires restarting the specific operating system service as follows. On RedHat:

$ sudo systemctl restart mariadb

And on Ubuntu:

$ sudo systemctl restart mysql

Finally, ensure that the MariaDB server is automatically started at boot time by appropriately configuring init, systemd, cron or whatever mechanism is in place for automating the startup of services.

Securing MariaDB¶

Once the installation is performed, it’s also recommended that MariaDB’s installation and configuration be hardened:

$ sudo mysql_secure_installation

The simplest way to make the MariaDB server (mysqld) available to Unscrambl Qbo Decisions is to install it on the same host where Unscrambl Qbo Decisions is going to be placed. It is also necessary to ensure that the server is using its default port (3306):

$ netstat -lptn | grep 3306
tcp   0   0 0.0.0.0:3306   0.0.0.0:*   LISTEN   11430/mysqld

Note that placing the MariaDB server on another host as well as using a different port number can be done, but both of these options require additional configuration.

Populating the MariaDB timezone tables¶

Several tables in the MariaDB system database exist to store time zone information.

The MariaDB installation procedure creates the time zone tables, but does not load them.

To do so manually, as root run the following command (MariaDB’s root password will be requested):

$ mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root -p mysql -D mysql

There will be one or two warnings similar to:

Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.

which you can ignore.

Note that loading the time zone information is not necessarily a one-time operation because the information changes occasionally.

When such changes occur, applications that use the old rules become out of date and you may find it necessary to reload the time zone tables (using the instructions above) to keep the information used by the MariaDB server current. Please refer to the MariaDB documentation for more information.

Setting up a MariaDB user¶

Once MariaDB is properly installed, it must be configured with a user and password to be used by Unscrambl Qbo Decisions when establishing connections with the database server. By default, both the user and password are set to drive.

To create a MariaDB user called drive with the password drive, start MariaDB’s interactive shell using MariaDB’s root user:

$ mysql -u root -p

And issue the following command:

MariaDB [(none)]> CREATE USER '<user>'@'%' IDENTIFIED BY '<password>';

Replacing <user> and <password>, with drive, specifically:

MariaDB [(none)]> CREATE USER 'drive'@'%' IDENTIFIED BY 'drive';

Once the user is created, it must be given privileges to create the Unscrambl Drive databases:

MariaDB [(none)]> GRANT ALL PRIVILEGES ON `drive_%` . * TO 'drive'@'%';
MariaDB [(none)]> FLUSH PRIVILEGES;

You can now close the MariaDB interactive shell by pressing CTRL-D (the Control and the D key, together) or by using the exit command.

At this point you should be able to login to MariaDB using the user drive and authenticate using the initial, default, password you just configured:

$ mysql -u drive -p

If the new user has been properly configured, you will once again be greeted by the MariaDB interactive shell:

$ mysql -u drive -p
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 39
...
MariaDB [(none)]>

Using a non-default MariaDB configuration¶

The use of MariaDB on a different host as well as the use of a different user name and/or password requires updating the Unscrambl Qbo Decisions configuration file unscrambl/etc/drive.json.

This JSON file can be edited using a regular text editor.

To change the password used by the Unscrambl Qbo Decisions database server user, it’s necessary to first encrypt the new password so it’s not stored in clear text in the configuration file. Unscrambl Qbo Decisions includes a tool that can be used to perform this operation:

$ $UNSCRAMBL_HOME/bin/password_encryptor -p <new_password>

Once you have the newly encrypted password, modify the drive|database|{userName,userPassword}, and, if necessary 1, the solution|database|{userName,userPassword} entries in the configuration file with the new password as well as a new user name.

1: As mentioned, some Unscrambl Qbo Decisions installations have optional features (i.e., a solution) that also make use of a relational database.

Note that the configuration file is nested, so the notation used in the prior sentence means, e.g., the entry userName as well as the entry userPassword are under the entry database, which is under the solution entry.

Note that if you are changing one or both of these settings, MariaDB itself must be updated with this new user/password information.

To change the host and port the MariaDB server uses, locate and update one or more of the following entries: drive|database|address and, if necessary, the solution|database|{userName,userPassword}. The address is a tuple with the following format: hostname:port, e.g., foo.unscrambl.com:3306.

Configuring Redis for use by Unscrambl Qbo Decisions¶

Unscrambl Qbo Decisions uses Redis, an in-memory data structure store.

You might be required to address certain OS-level requirements to ensure that Redis runs efficiently. As the Installation Process is carried out, you might face warnings similar to the following:

Warning

The operating system in this host is not optimally configured to run Redis, details:

overcommit_memory is set to 0. Redis background save process may fail under low memory conditions. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot or run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect. For more details, please refer to http://redis.io/topics/admin
when Transparent Huge Pages (THP) support is enabled in the Linux kernel (as is the case now), it can lead to high latency when Redis forks to persist data to disk. THP support can be disabled by executing the command ‘echo never > /sys/kernel/mm/transparent_hugepage/enabled’ as root, and adding it to your /etc/rc.local in order to retain the setting after a reboot. When making it permanent, make sure that the rc.local file has the execution permission for the owner (if this file is a symlink, ensure that the permission is set for actual file being linked). For more details, please refer to http://redis.io/topics/admin
unable to enforce the TCP backlog settings of 511 yet-to-be-listened TCP connections in Redis. To fix this issue add ‘net.core.somaxconn=511’ to /etc/sysctl.conf and then reboot or run the command ‘sysctl -w net.core.somaxconn=511’ for this setting to take effect immediately, but only until the next reboot. For more details, please refer to http://redis.io/topics/admin

Ensure that, if/when these errors are displayed, the changes suggested by the error messages are put in place.

Installation Package¶

The Unscrambl Qbo Decisions installation tarball includes:

The Unscrambl Qbo Decisions software platform itself, comprising all of the necessary components to run Unscrambl Qbo Decisions-supported applications.
The pre-configured Python virtualenv environment, comprising all Python dependencies required by Unscrambl Qbo Decisions applications to run.
The external open source software required by Unscrambl Qbo Decisions, e.g., Apache Tomcat, Apache Kafka, among others.

Installation Process¶

The installation process must be carried out using the userid that will manage the Unscrambl Qbo Decisions software.

It is recommended that this user be named drive or, if using a hybrid Unscrambl Qbo Decisions/InfoSphere Streams environment, streams.

Prior to starting the actual installation, if HTTPS-secured web access to Unscrambl Qbo Decisions will be made available, a (optional) DNS entry must be configured to provide a user-friendly URL to end users as well as a self-signed or commercial SSL certificate must be on-hand as it will be required to complete the product installation.

It might be helpful to become familiar with the infrastructure used to provide HTTPS access to Qbo Decisions by reading the steps outlined in the section on web proxy configuration before attempting the installation steps.

The installation process begins by un-tarring the software tarball:

$ tar xvfz drive-2.2.0-rhel07.tar.gz

The suffixes 2.2.0 denote the version you are installing and rhel07 the specific operating system (rhel07 for RedHat 7).

Extract the software:

$ mkdir -p /opt/unscrambl/drive/rhel07
$ tar xzvf drive-2.2.0-rhel07.tar.gz -C /opt/unscrambl/drive/rhel07

Again, other locations are acceptable, but /opt/unscrambl/drive/rhel07 is the recommended path. Now, you are ready to perform the configuration steps:

$ cd /opt/unscrambl/drive/rhel07/unscrambl/bin
$ ./installer

installer is an interactive program and will guide you through specific installation and configuration choices:

Unscrambl Qbo Decisions is a commercial product, subjected to End-User License Agreement terms. A paper-based or digital
copy of these terms must have been signed and agreed by someone authorized to do so in your organization, prior to
carrying out this configuration. A non-customer specific copy of these terms is included for your reference in this
installation package (unscrambl/license/eula.pdf). Do you confirm that you are authorized to proceed with the
configuration based on the terms specified in your organization's own license agreement with Unscrambl Inc. (y/n)?

As a first configuration step, you will be asked about the directory that will be used to host the Unscrambl Qbo Decisions instance, i.e., the location in the file system that Unscrambl Qbo Decisions will use to host its services their logs as well as the data that will be kept by the Unscrambl Qbo Decisions data management services.

Next, you will be asked which network interface to use for external TCP/IP traffic. You should choose the interface that provides connectivity to other hosts in the cluster (if any) and/or external services the Unscrambl Qbo Decisions application will interface with:

Please select the network interface to use for external TCP/IP traffic
(default: 'eth0'):

[0] lo: 127.0.0.1
[1] eth0 10.0.0.123
[2] tun0: 10.8.0.45

Selecting the number corresponding to the interface you want (default: '1' for interface 'eth0'): 1

The 'eth0' network interface will be used for all external TCP/IP traffic.

Once a network interface is selected, the installer will proceed to ask which transport protocol should be used between Unscrambl Qbo Decisions’s web-based frontend and its backend services:

Do you want to use HTTPS-based interactions between the Unscrambl Qbo Decisions browser-based interface and its backend
([n]o: HTTP will be used (in production, a proxy such as nginx must be used to secure the client/server
communication via HTTPS); / [y]es: HTTPS will be used with a self-signed SSL certificate (recommended only for
short-term testing) (default: HTTP) (y/n)?

Warning

Unscrambl Qbo Decisions has access to and handles potentially sensitive information:

Authentication information: the use of Unscrambl Qbo Decisions requires a user account. Unscrambl Qbo Decisions has its own directory of users or can, alternatively, defer to an enterprise-wide LDAP server. In either case, user passwords are employed to ensure that a user is both authenticated and authorized to use the system. While Unscrambl Qbo Decisions never stores user passwords in the clear, certain interactions require the transfering of passwords between the front end, web-based interface to the backend. Hence, encryption (through the use of HTTPS) is STRONGLY recommended.
Metadata and structural information about subscriber and corporate data feeds: Unscrambl Qbo Decisions carries out analytics employing data that is often private and sensitive. Once, again, interactions between its web-based interface and its backend require manipulating such data and encryption, in the form of HTTPS interactions, is STRONGLY recommended to preserve end-to-end confidentiality.

While Unscrambl Qbo Decisions is generally hosted in an internal network, never facing non-corporate users, it does integrate with other segments of the enterprise computing environment. Unscrambl recommends that administrators take every possible precaution to protect the integrity and confidentiality of the data consumed and produced by this platform.

There are multiple possible configurations to choose from and each one has certain advantages and risks:

HTTP (without a proxy such as nginx securing the client/server interactions), available network-wide (STRONGLY DISCOURAGED): this is the simplest form of installing Unscrambl Qbo Decisions. Nevertheless, it is insecure as potentially sensitive information is transmitted in the clear over the network, flowing from the user’s browser to the server without any encryption, including passwords (potentially, even the ones used for authenticating via enterprise-wide repositories such as a corporate LDAP server, if such an integration is eventually enabled).

Note

SSL certificates

An SSL certificates is a data file that digitally binds a cryptographic key to an organization’s identity. For instance, when installed on a web server, it activates the padlock used by the browser to indicate a _secure_ connection and, hence, the HTTPS protocol in interactions between the brower and a server.

In general, an SSL certificate is obtained from a Certificate Authority (CA) and it attests the ownership of a public key by the named subject of the certificate, providing an assurance that an interaction is occurring between a client and a properly identified server. A CA acts as a third party, trusted both by the subject (owner) of the certificate and by the party relying upon the certificate.

The Unscrambl Qbo Decisions web service can make use of an SSL certificate to ensure that interactions between the web-based client and server are properly authenticated (i.e., to provide an assurance to the browser-based client that it is indeed speaking to an actual server) as well as encrypted, such that no sensitive information flows over between a client and server in clear text form.

SSL certificates can be purchased from several vendors or obtained for free from organizations such as Let’s Encrypt. Commercial SSL certificates are typically verified and accepted by mainstream web browsers such as Google Chrome and Mozilla Firefox.

SSL certificates can also be provided by any entity hosting a Public Key Infrastructure (PKI). For instance, an organization’s IT department might host its own internal PKI and issue self-signed certificates. These certificates work just like commercial certificates, but they will typically produce a warning or an outright rejection from web browsers, which will not recognized the PKI’s CA. In such cases, upon being directed by the organization’s IT department, the certificate may be added to the browser and accepted as legitimate, thus quieting down the warnings that would, otherwise, be raised every time a web server presenting it is accessed.

Unscrambl Qbo Decisions can be configured with either type of certificate, but Unscrambl strongly recommends that a certificate be obtained from an officially recognized commercial or non-profit CA.

HTTPS, available network-wide (RECOMMENDED for testing only): this configuration is deemed safe, as it minimizes the chances of a sensitive data breach. In such a configuration, the interactions between the browser-based user interface and Unscrambl Qbo Decisions’s backend is carried out via HTTPS interactions (encrypted). In this case, the Unscrambl Qbo Decisions backend service (hosted by Apache Tomcat) is the direct destination of calls performed by a user’s browser to the server servicing the REST APIs.

The disadvantages of this approach are two-fold. First, it employs a self-signed certificate generated by Unscrambl and not a commercial certificate issued by a proper Certification Authority. Second, the URL to access Unscrambl Qbo Decisions will be in the form https://drive.company.com:<port>/drive. In other words, the port where the service runs will be part of the URL. Typically, HTTPS servers bind and run on the privileged port 443 and such a port number can be omitted from the URL. While the Apache Tomcat-based Qbo Decisions backend can be configured to use port 443, this configuration is not recommended since it requires running the web server as root, which opens up the possibility of a complete takeover of the host where the web server runs should an unknown (but possible) security vulnerability be exploited.

Alternatively, it is also possible to fiddle with the Operating System settings to allow non-root-owned applications to use a privileged port. Nevertheless, such a configuration is both non-standard and complex from a system administration standpoint.

If this configuration is chosen, the installer will ask for a non-privileged port to be used by the web server and will automatically create a certificate authority, will issue a CA certificate, and install a self-signed SSL certificate to be used by the web server as part of the installation process.
HTTP, available only in the localhost interface, proxied by an HTTPS web proxy (STRONGLY RECOMMENDED): this configuration is deemed the safest, as it minimizes the chances of a sensitive data breach. As with the prior configuration, the interactions between the browser-based user interface and the web proxy in front of Unscrambl Qbo Decisions’s backend are HTTPS-encrypted.

The web proxy (both Apache httpd and nginx are supported) employs a regular local HTTP connection to the Unscrambl Qbo Decisions backend and a client will interact with the proxy via HTTPS, which will in turn relay the requests to the web server.

While interception of end-user communication with the web server is possible, this can only occur when the host where the web server is installed is compromised and root access is available to the malicious party.

The benefit of this approach is that the web proxy (which is specifically hardened for this task), not Unscrambl Qbo Decisions’s backend, runs as root. Furthermore, the web proxy’s internal design is optimized for these types of interactions, the use of commercially-issued or locally-issued SSL certificates, specific SSL ciphers, as well as many other SSL-related configurations much easier to put in place.

In this configuration, the installation of an SSL certificate as well as the configuration of the HTTPS endpoint is done by installing and configuring the web proxy, a step described in the web proxy configuration section.

Finally, an Unscrambl Qbo Decisions installation can be optionally configured with additional integration points to external software packages including IBM SPSS Modeler Solution Publisher, IBM InfoSphere Streams, as well as Oracle WebLogic Server. Each of these integration points requires a prior licensed installation for each of these software packages. When installing an Unscrambl Qbo Decisions version enabled with one or more of these integration points, additional installation steps will take place. These steps, applicable only to the specific integration points enabled in the installation, will be carried out by the installer program to ensure that it can find the proper version for each of the external packages needed by them. For instance:

Please enter the path to an existing IBM SPSS Modeler Solution Publisher
installation to use: /opt/x86_64/ibm/spss/ModelerSolutionPublisher/17.1

Please enter the path to an existing IBM InfoSphere Streams
installation to use: /opt/rhel07/ibm/streams/4.0.1.0

ERROR: failed to validate IBM InfoSphere Streams
installation at '/opt/rhel07/ibm/streams/4.0.1.0'
Product information file '/opt/rhel07/ibm/streams/4.0.1.0/.product' does not exist
Please provide a valid path to an existing IBM InfoSphere Streams installation

Please enter the path to an existing IBM InfoSphere Streams
installation to use: /opt/rhel07/ibm/streams/4.0.1.0

Please enter the path to an existing Oracle WebLogic Server
installation to use: /opt/unscrambl/drive/noarch/weblogic-10.3.6.0

If everything is correctly configured, a success status message will be printed out:

Unscrambl Qbo Decisions environment was successfully installed...

Now that the configuration is complete, a test can be executed. To run any Unscrambl Qbo Decisions application, the shell where the application is going to run must be configured by invoking the environment_setter script (<drive-install-location>/unscrambl/bin/environment_setter):

$ ./environment_setter

Once the script is source’d, several environment variables are configured and the Python virtualenv environment is activated, indicating that we are ready to start an Unscrambl Qbo Decisions application:

[drive]|$ env | sort | grep UNSCRAMBL
UNSCRAMBL_ARCH=x86_64
UNSCRAMBL_HOME=/opt/unscrambl/drive/rhel07/unscrambl
UNSCRAMBL_JAVA_PATH=/usr/lib/jvm/java
UNSCRAMBL_OS=rhel07
UNSCRAMBL_PACKAGE_VERSION=2.2.0
UNSCRAMBL_PYTHON_PATH=/opt/unscrambl/drive/307/pyenv
UNSCRAMBL_READLINK=readlink
UNSCRAMBL_TOMCAT_PATH=/opt/unscrambl/drive/noarch/apache-tomcat-9.0.20

The preparation of the shell must be made for every shell and session where an Unscrambl Qbo Decisions application will run. Now, we can run a test application (<drive-install-location>/unscrambl/bin):

[drive]|$ cd unscrambl/bin
[drive]|$ ./installation_tester -c -sns
INFO: this application is using '/tmp/<user>/unscrambl' to output and store its
code-generated assets...
.
.

Note that we are invoking the test application installation_tester using the -c flag to indicate that the application should be re-built (just in case, it has been executed before), thus ensuring that a fresh version, based on the current installation, is executed. Normally, Unscrambl Qbo Decisions applications are only rebuilt when needed and taking such a precaution is normally not necessary.

Configuring Unscrambl Qbo Decisions’s Runtime Environment¶

The Unscrambl Qbo Decisions runtime environment can be configured based on the specific performance and reliability requirements of a customer deployment. These runtime environment settings are specified in the following file: $UNSCRAMBL_HOME/etc/runtime_environment.json.

An important runtime configuration is the persistence behavior of PinPoint, which is the Redis-based distributed in-memory store used by Qbo Decisions to maintain subscriber profiles. As a primary mechanism, PinPoint relies on append-only logging to persist its data. As an additional mechanism, it also periodically checkpoints its data to disk.

The checkpointing can be configured using the pinPoint|server|checkpointConfiguration|schedules setting inside the runtime_environment.json file. The default checkpoint configuration is given as follows:

"schedules":
[
    {
        "operationCount": 1,
        "timeInSeconds": 86400
    },
    {
        "operationCount": 1000000,
        "timeInSeconds": 3600
    }
]

This configuration indicates that the state is checkpointed every hour if at least 1 million profiles are updated since the last checkpoint, and every day if at least one profile is updated since the last checkpoint. More schedules can be added or the existing schedules can be updated, as needed. Since the append-only logging already provides persistence, specifying frequent checkpointing is unnecessary and can adversely affect the CPU usage of the PinPoint servers.

Configuring a web proxy¶

Unscrambl Qbo Decisions consists of a set of backend services, accessible via a web-based user interface.

Once installed, an instance of Unscrambl Qbo Decisions can be started. The communication between the web-based user interface and the backend employs REST APIs over HTTP or HTTPS.

Unscrambl strongly recommends that HTTPS be used to ensure that the sensitive information that transits over the network is encrypted.

As seen in the Installation Process section, Unscrambl Qbo Decisions offers the choice of web transport protocols during its installation process and recommends HTTPS with the use of a web proxy as the means to access its private backend services.

This option requires installing Unscrambl Qbo Decisions with HTTP support, where the backend is bound only to the local lo network interface, allowing HTTP access only within the host where the Unscrambl Qbo Decisions web server runs.

Either of two commonly used web proxies, packaged an available in the Linux distributions supported by Unscrambl Qbo Decisions, can be configured to provide the HTTPS-based access to the users interacting with Unscrambl Qbo Decisions via their web browsers: Apache httpd and nginx.

In either case, two steps must be carried out prior to the web proxy installation and configuration. These steps and the specific configuration of a web proxy are outlined next.

Configuring a DNS CNAME entry to point to Unscrambl Qbo Decisions (optional)¶

Ideally, the URL used to access Unscrambl Qbo Decisions will be in the form https://unscrambl.acme.com/drive, where unscrambl designates this locator as an Unscrambl product installation at acme.com (the customer’s Internet domain) and the context path, drive, indicates the name of the product.

Usually, the friendlier unscrambl name will map to an internal server hosting the Unscrambl Qbo Decisions installation, whose name will follow an (often less friendlier) internal IT convention (e.g., drive-cluster003-node001.acme.com). Thus, we recommend that a DNS alias (specifically a CNAME entry) be obtained and registered prior to the Unscrambl Qbo Decisions installation.

Procedures to update the corporate DNS servers vary. We recommend that the system administrator installing Qbo Decisions contacts a local IT specialist to understand the process and to carry out the actual DNS registration.

As an example, an organization using TinyDNS will need to add the following entry to its configuration:

Cunscrambl.acme.com:drive-cluster003-node001.acme.com:120

And invoke the utility (tinydns-data) to activate this entry.

Obtaining an SSL certificate¶

SSL certificates come in different forms. A certificate can be obtained for a single host name, for multiple host names, or as a wildcard, accepting any name under a particular domain. While all of them ought to work with Unscrambl Qbo Decisions, a single-host certificate is sufficient and this option, if procuring a commercial certificate, is often the most economical.

Procedures to obtain a certificate vary both for commercial as well as for self-signed internal certificates.

Please consult a local IT specialist to understand which alternative is the most adequate in your environment.

In the rest of this document, it is assumed that a commercial certificate is on hand and available to be used during the configuration of the web proxy.

Regardless of how a certificate is obtained, in the end, a set of files related to the certificate must be available. In the example below, we are assuming that the certificates are being kept in a directory called unscrambl.acme.com:

unscrambl.acme.com/fullchain.pem: this is the actual certificate plus the intermediate certificates (if any). See the documentation the Apache httpd SSLCertificateFile configuration key and the the nginx ssl_certificate configuration key for more information.
unscrambl.acme.com/privkey.pem: this is the certificate private key for the server. See the documentation the Apache httpd SSLCertificateKeyFile configuration key and the the nginx ssl_certificate_key configuration key for more information.

Once the certificate has been obtained and these files are available either the Apache httpd or nginx must be installed and configured.

In the following sections, we will describe these steps assuming that the installation is being made on a RedHat 7 (or CentOS 7) server, where the Unscrambl Qbo Decisions web backend will eventually run.

Since the installation and configuration require updating system-owned resources, either sudo or root access is required.

Configuring Apache `httpd`¶

The installation requires downloading and installing the necessary OS packages:

$ yum install -y httpd mod_ssl

To ensure that the Apache httpd server is started on boot, the service must be enabled:

$ systemctl enable httpd

Once installed, the configuration enabling the proxying must be added to httpd’s main configuration file /etc/httpd/conf/httpd.conf. The following excerpt demonstrates how this is done, by creating a new virtual host entry:

...
<VirtualHost *:443>
    ServerName unscrambl.acme.com
    ## Unscrambl Qbo Decisions will accessible at https://unscrambl.acme.com/drive
    ProxyPass /drive http://[::1]:8081/drive
    ## Unscrambl Qbo Decisions must be configured with HTTP access restricted to localhost binding to port 8081
    ## Note that the IPv6 address being used, [::1], is the IP address corresponding to the loopback interface.
    ## Unscrambl Qbo Decisions's Tomcat instance by ## default will use the IPv6 protocol stack
    ProxyPassReverse /drive http://[::1]:8081/drive
    ProxyPreserveHost on
    ProxyRequests off
    Header always set Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"
    SSLCertificateFile /etc/httpd/stash/unscrambl.acme.com/fullchain.pem
    SSLCertificateKeyFile /etc/httpd/stash/unscrambl.acme.com/privkey.pem
    SSLEngine on
    SSLProtocol TLSv1.2
    SSLProxyEngine on
</VirtualHost>
...

Note that in our configuration we chose to install the certificate-related files under /etc/httpd/stash. This is not necessary, but is convenient as far as applying the correct SE Linux context to these files, which is carried out as follows:

$ restorecon -vr /etc/httpd/stash

Once this is done, httpd can be started (or restarted):

$ systemctl start httpd

If all is well, you should be able to start Unscrambl Qbo Decisions, open a browser and point it to https://unscrambl.acme.com/drive.

To start Unscrambl Qbo Decisions, see the Starting and stopping Unscrambl Qbo Decisions section. If a failure has happened when starting httpd, see the Testing and troubleshooting the web proxy installation section for additional help with troubleshooting.

Configuring `nginx`¶

The installation requires downloading and installing the necessary OS packages:

$ yum install -y nginx

To ensure that the nginx server is started on boot, the service must be enabled:

$ systemctl enable nginx

Once installed, the configuration enabling the proxying must be added to nginx’s main configuration file /etc/nginx/nginx.conf. The following excerpt demonstrates how this is done, by creating a new server entry:

http {
...
    server {

        listen 443;
        server_name unscrambl.acme.com;

        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload";

        ssl on;
        ssl_certificate /etc/nginx/stash/unscrambl.acme.com/fullchain.pem;
        ssl_certificate_key /etc/nginx/stash/unscrambl.acme.com/privkey.pem;
        ssl_session_timeout 5m;
        ssl_protocols TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers on;

        ssl_session_cache shared:SSL:10m;
        ## Unscrambl Qbo Decisions will accessible at https://unscrambl.acme.com/drive
        location /drive {

            ## Unscrambl Qbo Decisions must be configured with HTTP access restricted to localhost binding to port 8081
            proxy_pass http://[::1]:8081/drive;
            ## Note that the IPv6 address being used, [::1], is the IP address corresponding to the loopback
            ## interface. Unscrambl Qbo Decisions's Tomcat instance by default will use the IPv6 protocol stack
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header X-Forwarded-Server $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_redirect off;
        }
    }
}

Note that in our configuration we chose to install the certificate-related files under /etc/httpd/stash. This is not necessary, but is convenient as far as applying the correct SE Linux context to these files, which is carried out as follows:

$ restorecon -vr /etc/nginx/stash

Once this is done, nginx can be started (or restarted):

$ systemctl start nginx

If all is well, you should be able to start Unscrambl Qbo Decisions, open a browser, and point it to https://unscrambl.acme.com/drive. troubleshooting.

To start Unscrambl Qbo Decisions, see the Starting and stopping Unscrambl Qbo Decisions section. If a failure has happened when starting httpd, see the Testing and troubleshooting the web proxy installation section for additional help with troubleshooting.

Testing and troubleshooting the web proxy installation¶

Once the web proxy is configured and Unscrambl Qbo Decisions has been started, using the web interface is as simple as entering the access URL in the browser window.

There are typically two problems that might occur in a new installation, where either RedHat Linux or CentOS is being used:

the firewall configuration: if the server running the proxy has firewalld installed, HTTPS access is typically blocked by default. While we provide some helpful directions below, we strongly recommend reading the firewalld documentation so the impact of these commands is fully understood.

To check on current status of firewalld, the following command can be executed:

$ firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: enp0s3
sources:
services: dhcpv6-client https ssh
protocols:
masquerade: no
forward-ports:
sourceports:
icmp-blocks:
rich rules:

If https does not appear in the list of services (as it does above), it must be added, either temporarily:

$ firewall-cmd --zone=public --add-service=https
$ firewall-cmd --reload

Or permanently:

$ firewall-cmd --zone=public --permanent --add-service=https

At this point, re-running the status command will reveal whether HTTPS access to the server is now enabled.

the SE Linux configuration SELinux is a set of kernel modifications and tools that have been added to RedHat and CentOS, providing support for access control security policies. It may affect the web proxy in the sense.

Often, if the SELinux access control is not properly configured for the web proxy, there will be two symptoms. First, an access from the browser will be rejected, often, with a “bad gateway” error message. Second, the SELinux log file (ll /var/log/audit/audit.log, root access is required both for search the logs as well as for reconfiguring SELinux policies) will include a rejection of an operation attempted by the web proxy (in the example below, nginx, but very similar for httpd as well), possibly, similar to the following:

$ grep nginx /var/log/audit/audit.log
type=AVC msg=audit(1490570804.119:555): avc:  denied  { name_connect } for  pid=5725 comm="nginx" dest=8080
scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:http_cache_port_t:s0 tclass=tcp_socket

Fiddling with SELinux will have a direct impact on the security of a host as well as on the overall network where that host is located. Unscrambl strongly recommends that changes being done on SELinux policies are undertaken by someone familiar with its configuration. The procedures we indicate below are not to be taken without understanding their potential repercussions.

This audit entry can be better understood by running the following command, which also prescribes the possible fixes:

$ ausearch -c nginx | audit2allow -m nginx

module nginx 1.0;

require {
    type httpd_t;
    type http_cache_port_t;
    class tcp_socket name_connect;
}

#============= httpd_t ==============

#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_connect, httpd_can_network_relay
allow httpd_t http_cache_port_t:tcp_socket name_connect;

You can check the current state of these Booleans settings as follows, which at this point, is most likely off:

$ getsebool -a | grep httpd

Finally, the problem can be resolved by allowing network connections from the web proxy to the actual server (i.e., so it can act as a relay):

$ setsebool -P httpd_can_network_relay on

Alternatively, a new SELinux non-base policy can be put in place:

$ cd /tmp
$ ausearch -c nginx | audit2allow -M nginx
$ semodule -i nginx.pp
$ rm nginx.pp