2.1.2.1 Constant Memory Demand

SZTPD has a constant memory demand that grows in proportion to configuration data, minus all the binary/base64 nodes, which tend to be large to very large in size.

SZTPD reduces configuration data both passively and actively:

• Passively, SZTPD’s data model is highly normalized², thus elimanating the need for redundant definitions.
• Actively in two ways:
  • Binary/base64 values are not held in memory (mentioned above).
  • An eviction policy is used to prune stale objects from configuration.

2.1.2.2 Transitive Memory Demand

SZTPD has transitive memory demands when processing requests from clients. The highest memory-impacting request is a “PUT” request used to replace the entire SZTPD configuration. In this case, the transitive memory demand is proportional to the sum of the old and new configuration sizes.

2.3.1.1 APIs

Each listening port must specify which “interface” it presents. SZTPD defines three interfaces as follows:

1. The “native” Interface

   The native interface presents an API enabling “northbound” clients to configure everything and anything that can be configured.

   The API the native interface presents varies slightly based on the product mode. That is, the configuration data model for “mode-1” and “mode-x” modes, as described in Prompting for Desired Mode Selection, are slightly different.

   There must always be extacly one “native” interface configured. The default listening port described in Defaults presents the “native” interface.

2. The “rfc8572” Interface

   The rfc8752 interface presents the “southbound” bootstrap server API defined in Section 7 of RFC 8572.

   In order for SZTPD to satisfy its business purpose, there must be at least one listening port presenting the “rfc8572” interface. More than one “rfc8572” interface may be configured if, e.g., distinct listening ports are needed to listen on different IP addresses and/or present different TLS certificates⁷.

3. The “tenant” Interface

   The tenant interface is an additional “northbound” interface that is only useful for when the “mode-x” product mode is configured.

   The tenant interface presents a isolated view of each tenant’s data. No tenant is aware of the existance of any other tenant, nor has access to any other tenant’s data. The tenant interface presents an API that is almost exactly like the “mode-1” product mode’s “native” interface.

   A listening port presenting the “tenant” interface may be used to service a multiplicity of tenants. More then one “tenant” interface may be configured if, e.g., distinct listening ports are needed to listen on different IP addresses and/or present different TLS certificates⁸.

2.3.1.2 HTTP vs HTTPS

Each listening port must be configured to present the “HTTP” or “HTTPS” protocol. The default listening port described in Defaults presents the “HTTP” protocol.

When configured to use “HTTPS”, the SZTPD process terminates the TLS connections itself. By contrast, when configured to use “HTTP”, it is expected that an external endpoint (e.g., an HTTP proxy) positioned in front of SZTPD will terminate the TLS connections.

Terminating TLS connections outside of SZTPD offloads some CPU load, which may improve performance when the number of simultaneously bootstrapping devices grows to a large number.

When “HTTP” is used, SZTPD may be additionally configured with information about the external endpoint so that, e.g., when SZTPD sends callback information to remote peers, such as the hyperlinks in account-activation emails, it can address the URLs to the external endpoint instead of SZTPD’s local endpoint.

When “HTTP” is used, it is necessary for the external endpoint to send the client’s certificate in an HTTP header field called “X-Client-Cert” containing a url-encoded PEM. For instance, using NGINX, this is accomplished using the configuration line:

proxy_set_header X-Client-Cert $ssl_client_escaped_cert;"

2.3.2.1 Sending Email

SZTPD sends email for the following purposes:

• Activating newly created administrator accounts.
• Sending account-deactivation warnings to account administrators.

In order to deliver the email, SZTPD, or the host operating system on its behalf, will initiate an outbound SMTP/S connection to a configured SMTP server. Traditionally this is a connection to port 25, 587, or 2525.

2.3.2.2 Sending Notifications

SZTPD sends notifications for the following reasons:

• Notification of device bootstrapping activity.
• Warning of pending expirations.
• Notification of an expiration.

In order to deliver notification, SZTPD uses mutually authenticated HTTPS connection to configured URIs. The default destination port for HTTPS is port 443, but the URIs may encode alternate port number values as needed.

2.3.2.3 Verifying Device Ownership

For deployments using the mode ‘x’ product mode, SZTPD enables the host system to ensure that the serial-numbers configured by tenants are rightfully owned by the tenant.

In order to verify device ownership, SZTPD initiates a mutually authenticated HTTPS connection to the URI configured when enabling this feature.

The default destination port for HTTPS is port 443, but the URIs may encode alternate port number values as needed.

2.3.2.4 Verifying certificate paths

SZTPD may need to verify certificate paths for the following reasons:

• Verifying a remote server’s certificate.
• Verifying certificate-based client credentials.

In order to properly verify certificate paths, it is sometimes necessary for SZTPD to check the current revocation status of certificate paths. Each certificate may specify URIs for where the CRL or OCSP can be obtained and, when needed, SZTPD will initiate a connection to the URI specified in the certificate.

2.3.2.5 Accessing a remote database

When configured to use a remote database (see Persistence Selection), SZTPD will initiate connections to the remote database, specified by the URI given on the command line.

While the database is “remote” to the ‘sztpd’ process it could be running on the same machine.

The remote port accessed varies by database and TLS configuration,

2.3.2.6 DNS resolution

When initiating any of the previously discussed outbound connections, the configurations for the remote host may be given as either an IP address or a domain name. When a domain name is provided, SZTPD resolves the domain name to IP addresses using the system configured DNS resolver. Depending on how the DNS resolver is configured, the host system may initiate a connection to a network-based resolver. Traditionally this is a TCP-based connection destined to port 53.

2.5.1.1 In-memory Database

The in-memory database type “persists” all data in memory, which is automatically lost when the ‘sztpd’ process ends. In-memory databases are fast, but demand additional process memory usage proportional to the size of the data stored.

An in-memory database is an excellent choice for development and test efforts, as it automatically disappears when the ‘sztpd’ process ends. The simulator (see the Administrator’s Guide) uses the in-memory database for this reason.

An in-memory database may also have use in a production environment (e.g., in an SDN context) whereby the intent is for the SZTPD instance to be ephemeral, lasting only as long as needed to bootstrap a specific set of devices.

Use of the in-memory database type is specified by the database URL provided on the command line having the form:

sqlite:///:memory:

2.5.1.2 File-based Database

The file-based database storage persists all the data into a single file located by the path-component in the database URL specified on the command line. File-based databases may be desirable in cases where persistence across power cycles is important and yet dependency on a remote RDBMS is not desirable.

The specified file may reside on a filesystem mounted from a RAID system providing resiliency against disk failures. Care should be to ensure fast and secure access to a remote system (e.g., a NAS or SAN).

The file itself may be backed-up and restored as needed to ensure disaster recovery. If the filesystem does not support taking snapshots while a system is running, it is recommended that the ‘sztpd’ process is either stopped or suspended during the backup operation.

Use of the file-based database type is specified by the database URL provided on the command line having one of the following form:

sqlite:///relative/path/to/sztpd.db     (unix)
sqlite:////absolute/path/to/sztpd.db    (unix)
sqlite:///C:\path\to\sztpd.db        (windows)

2.5.1.3 RDBMS Database

The RDBMS database engine persists all the data into one of several well-known database systems (e.g., MariaDB, Postgres, Oracle, etc.). In theory, SZTPD can use any RDBMS supported by SQLAlchemy, though it is recommended to only use those that SZTPD has been tested against.

Use of an RDBMS database enables large data sets with fast-access to keyed data. Scaling of the database tier supports a variety of performance and availability targets. It is recommended to use database-level encryption to ensure to protection of the SZTPD data at rest.

Use of the RDBMS database type is specified by the database URL provided on the command line having the form:

<engine>://<user>:<passwd>@<host>:<port>/<database-name>

For instance:

   postgresql://sztpd-admin:secret@db.example.com:5432/sztpd-db 
   mysql+pymysql://sztpd-admin:secret@db.example.com:3306/sztpd-db

The RDBMS database may run on the same machine as SZTPD or on another machine. If the database runs on the same machine, it could listen on the loopback address (e.g., 127.0.0.1) on thus not require network protection. However, if the database runs on another machine, even if in a “secure” or private network, it is recommended that the connection to a remote RDBMS is protected by transport level security (TLS), which entails both configuring the RDBMS server to listen for TLS connections and configuring SZTPD to connect to the the RDBMS using TLS.

To direct SZTPD to establish a TLS connection to the RDBMS, the “–cacert” must be specified. This parameter specifies the trust anchor certificate(s) that SZTPD must use to authenticate the RDBMS’s end entity certificate. For example:

$ sztpd --cacert db-cacert.pem postgresql://user:pass@localhost:5432/sztpd

To direct SZTPD to authenticate itself to the RDBMS using a client certificate, the “–key” and “–cert” parameters must be specified. Note that these parameters must be specified in conjunction with the previously mentioned “–cacert” parameter. For instance:

[Note: '\' line endings per RFC 8792]

$ sztpd --cacert db-cacert.pem --key db-client-key.pem -cert db-client-cert.pem \
postgresql://user:pass@localhost:5432/sztpd

RDBMS-specific details are provided in the RDBMS Configuration section.

2.5.2.1 Prompting for Contract Acceptance

Example output:

<Non-Production Use Contract>

First time initialization. Please accept the license terms.

By entering "Yes" below, you agree to be bound to the terms and conditions contained on this screen with Watsen Networks.

Please enter "Yes" or "No": <SZTPD waits for input here>

Entering “Yes” allows SZTPD to proceed to prompting for the mode to run. SZTPD will exit for any other entered response, not just “No”.

Alternatively, the “SZTPD_ACCEPT_CONTRACT” environment variable may be used to accept the contract. For instance:

$ export SZTPD_ACCEPT_CONTRACT="Yes"; sztpd sqlite:///:memory:

2.5.2.2 Prompting for Desired Mode Selection

SZTPD can run in one of two modes¹³:

The mode character represents, roughly, the number or tenants. For instance, ‘1’ is for “one tenant” (i.e., you), while ‘x’ is for “many tenants” (i.e., you plus your customers).

The first time SZTPD runs (i.e., an uninitilized database) it will prompt for a mode to be selected:

Mode: 
  1 - single-tenant
  x - multi-tenant

Mode: <SZTPD waits for input here>

Alternatively, the “SZTPD_INIT_MODE” environment variable may be used to selected the mode. For instance:

$ export SZTPD_INIT_MODE="x"; sztpd sqlite:///:memory:

Each of the modes are described in the following sections.

2.5.2.3 Mode ‘1’

Mode ‘1’ enables a multiplicity of devices to be configured, but without an additional “tenants” layer. From an API perspective, the difference, in comparison to the mode ‘x’ tree diagram, is illustrated below.

Mode ‘1’

+--rw devices
   +--rw device* [serial-number]
      +--rw serial-number     string
      +-- ... // additional device parameters here.

2.5.2.4 Mode ‘x’

Mode ‘x’ enables multi-tenancy, whereby each tenant effectively has their own Mode ‘1’ instance, with the only aspect missing being the ability to configure system-level information (e.g., networking parameters).

The host view is not only a superset of all tenant views, but it also comes with the unique ability to configure plugins, define device-types, and define an “device ownership verification” callback function¹⁴.

The difference in the APIs is illustrated below using tree diagrams (RFC 8340).

Mode ‘x’:

+--rw preferences
   +--rw device-ownership-verification!
   |  +-- ... // ownership verification parameters here
   +--rw plugins
      +--rw plugin* [name]
         +-- // plugin parameters here

+--rw device-types
  +--rw device-type* [name]
     +-- // device-type parameters here
    
+--rw tenants
   +--rw tenant* [name]
      +--rw name                    string
      +--rw device
         +--rw device* [serial-number]
            +--rw serial-number     string
            +-- ... // additional device parameters here.

2.5.3.1 Listening Ports

A freshly installed version will open a single port by default.

This port uses the following parameters:

• local address: 127.0.0.1
• local port: 8080

These values may be changed using environment variables:

• The default local address may be changed using the “SZTPD_INIT_ADDR” environment variable.
• The default local port may be changed using the “SZTPD_INIT_PORT” environment variable.

For instance:

[Note: '\' line endings per RFC 8792]

$ export SZTPD_INIT_ADDR="A.B.C.D"; \
  export SZTPD_INIT_PORT="9090"; \
  sztpd sqlite:///:memory:

The default port presents the HTTP (without TLS) protocol over which the RESTCONF API is presented. A SZTPD-client must use this port to provide an initial configuration, including configuring SZTPD to open other listening ports and/or use TLS.

2.5.3.2 Admin Account

The default server (i.e., one that has never been configured) does not have any administrator accounts defined.

Prior to configuring an admin, no client authentication needs to be supplied.

The very first configuration write request must, at least or in addition to other changes, configure at least one administrator account and, further, this account must have the “unrestricted” access level.

3.1.1.1 Using a MySQL Instance Installed on the Same Machine

Assuming a freshly installed MySQL instance running on the same machine as SZTPD, with an admin account called “root” with no password set, the following commands might be used to initialize a user account for SZTPD:

$ mysql -u root -e "CREATE USER 'testuser'@'localhost' IDENTIFIED BY 'secret';"
$ mysql -u root -e "GRANT ALL ON sztpd.* TO 'testuser'@'localhost';"
$ mysql -u root -e "FLUSH PRIVILEGES;"

Note: the grant above is to the yet-to-be-created “sztpd” database.

At which point SZTPD can be run as follows:

$ sztpd mysql+pymysql://testuser:secret@localhost:3306/sztpd"

Note that SZTPD will itself create a database called “sztpd”¹⁷ and initialize all the tables in it¹⁸.

The following command returns the database to the state before SZTPD ran:

$ mysql -u root -e "DROP DATABASE sztpd;"

And this command returns the database to the state before creating the user:

mysql -u root -e "DROP USER 'testuser'@'localhost';"

Note that, in all of the commands above, the string “localhost” could have been replaced with the string “127.0.0.1”. However, doing so switches from using a UNIX socket file to the TCP/IP stack. This may require setting the “bind-address” variable (e.g., in the “my.cnf” file) to “127.0.0.1” and restarting the MySQL server (i.e., ‘mysql.server restart’).

3.1.1.2 Using a MySQL Instance Installed on Another Machine

Assuming the MySQL instance is installed on another machine, the commands replace “localhost” with the appropriate IP addresses for the two machines.

For instance, assuming SZTPD is running on “11.11.11.11” and the MySQL instance is running on “22.22.22.22”, the MySQL user-creation commands would be:

$ mysql -u root -e "CREATE USER 'testuser'@'11.11.11.11' IDENTIFIED BY 'secret';"
$ mysql -u root -e "GRANT ALL ON sztpd.* TO 'testuser'@'11.11.11.11';"
$ mysql -u root -e "FLUSH PRIVILEGES;"

And for SZTPD to connect to it, the command would be:

$ sztpd mysql+pymysql://testuser:secret@22.22.22.22:3306/sztpd

Note that, in order for the MySQL server to support accepting connections from remote systems, its “bind-address” variable (e.g., in the “my.cnf” file) must either be unset, or set to either a specific IP address (i.e., “22.22.22.22”) or the wildcard address appropriate for the IP stack (e.g., “0.0.0.0” for IPv4). Be sure to restart the MySQL server afterwards (i.e., ‘mysql.server restart’).

3.1.2.1 Configure MySQL to Present the TLS Certificate

While it is possible to configure MySQL dyanmically, editing its configuration file (e.g., in the “my.cnf” file) is more understandable. Building on top of the previous example to bind the local address as well, the following is set:

[mysqld]
bind-address = 22.22.22.22

ssl_ca=mysqld-intermediate-cacerts.pem
ssl_cert=mysql-ee.pem
ssl_key=mysql-ee.key

require_secure_transport=ON

Be sure to restart the MySQL server afterwards (i.e., ‘mysql.server restart’).

3.1.2.2 Configure SZTPD to Authenticate the MySQL Server’s TLS Certifcate

SZTPD is “configured” to authenticate the server’s TLS certificate solely on the command line¹⁹. Specifically:

$ sztpd --cacert mysqld-trust-anchor-certs.pem mysql+pymysql://testuser:secret@22.22.22.22:3306/sztpd

3.1.3.1 Configure MySQL to Authenicate TLS Clients

Building on top of the previous example, the “my.cnf” file is updated to:

[mysqld]
bind-address = 22.22.22.22

ssl_ca=mysqld-all-cacerts.pem
ssl_cert=mysql-ee.pem
ssl_key=mysql-ee.key

require_secure_transport=ON

Be sure to restart the MySQL server afterwards (i.e., ‘mysql.server restart’).

It is additionally necessary to change the “user” definition from before.

$ mysql -u root -e "CREATE USER 'testuser'@'11.11.11.11' REQUIRE X509;"
$ mysql -u root -e "GRANT ALL ON sztpd.* TO 'testuser'@'11.11.11.11';"
$ mysql -u root -e "FLUSH PRIVILEGES;"

Note that additional “REQUIRE” options exist to ensure that the client certificate contains a specific ‘SUBJECT’ and/or is signed by a specific ‘ISSUER’. These tests are in addition to the MySQL server testing that the client certificate has a chain of trust to any of the CA certificates contained in the file specified by the “ssl_ca” variable.

It is also possible to create a user that requires both a client certificate and a password:

$ mysql -u root -e "CREATE USER 'testuser'@'11.11.11.11' IDENTIFIED BY 'secret' REQUIRE X509;"
$ mysql -u root -e "GRANT ALL ON sztpd.* TO 'testuser'@'11.11.11.11';"
$ mysql -u root -e "FLUSH PRIVILEGES;"

3.1.3.2 Configure SZTPD to Send a Client Certificate to MySQL

SZTPD is “configured” to send a client certificate solely on the command line²⁰. Specifically:

# NOTE: '\' line wrapping per RFC 8792

$ sztpd --cacert mysqld-trust-anchor-certs.pem --cert=sztpd-ee.pem \
        --key=sztpd-ee.key mysql+pymysql://testuser@22.22.22.22:3306/sztpd

Or, if also needing to pass a password:

# NOTE: '\' line wrapping per RFC 8792

$ sztpd --cacert mysqld-trust-anchor-certs.pem --cert=sztpd-ee.pem \
        --key=sztpd-ee.key mysql+pymysql://testuser:secret@22.22.22.22:3306/sztpd