Architecture Guide for Customer Systems Teams
Audience
IT Operations Team: Systems Administrators with an understanding of servers, OSs, networking, applications, security, etc. People comfortable with technical details.
Goal
This document provides a high level overall understanding of the Pliant architecture at every level, including:
Kubernetes Platform
Pliant microservices and their interaction
Pliant platform interaction with other systems
Scope
This document includes:
Kubernetes deployment models (K3s single node, K3s multi-node, Managed Kube Hosted)
Names and descriptions of each Pliant microservice
Communications diagram showing internal microservice interactions
Description and diagram of how Pliant interacts with third party systems
Mention remote worker and reference another document
Not covered in this document:
Backup and recovery
Troubleshooting
Kubernetes Platform
Pliant is a containerized microservices application that runs on top of a Kubernetes platform. The exact platform varies depending on the needs of the deployment. Pliant supports running on a single node Kubernetes K3s cluster, a three node cluster running K3s, or a hosted Kubernetes service such as Google GKE, Amazon EKS, or Azure AKS. Kubernetes provides the container management layer that enables scaling and availability of the Pliant microservices.
Regardless of the underlying Kubernetes flavor, the Pliant instance uses the same installation scripts and configuration files for deployment and upgrade.
K3s Single Node
In this Kubernetes deployment model, the Pliant instance and all of its storage and components are run from a single host. K3s is packaged as a single <40MB binary that reduces the dependencies and steps needed to install, run and auto-update a production Kubernetes cluster. This lightweight Kubernetes version leverages the built in containerd container runtime that is built in to Ubuntu, CentOS and other host operating systems. Persistent storage is provided to containers using the local-path-provisioner service, which maps pieces of the host file system into containers. This deployment model is used in proof of concept deployments along with Pliant Starter and Professional platform packages.
K3s Multi-Node
In this Kubernetes deployment model, a minimum of three dedicated Linux hosts are required for the Kubernetes cluster. The nodes are ideally placed in different locations (Separate hosts, racks, or availability zones) but interconnected with less than 10 milliseconds of network latency. An internal cluster Certificate Authority is established and used to generate a PKI infrastructure used to secure intra-cluster communications. The etcd distributed database is used to store and sync Kubernetes state configuration across the cluster. Persistent storage for pods is provided by Rancher Longhorn, which distributes data storage across the nodes and provides storage high availability. This deployment model is used with Pliant Enterprise platform packages when deployed on a customer premise. External load balancing is required to route inbound requests to the cluster nodes.
Cloud Hosted Kubernetes
Cloud service providers, such as Amazon Web Services, Microsoft Azure, and Google Cloud Platform, provide Kubernetes cluster hosting as a service. In this model, the Kubernetes cluster management, compute nodes, storage class, and inbound load balancing are managed by the cloud provider, leaving the admins only with the work of managing the containers that run on top. Storage is typically provided using a native cloud storage service, such as AWS Elastic Block Store. This deployment model is used with Pliant Enterprise platform packages when Pliant is hosted.
The Pliant Microservices
Regardless of the underlying Kubernetes configuration, Pliant comprised of the same microservices containers.
Data Storage
Persistent data is stored in three different locations depending on the type of data. These containers run as SatefulSets in Kubernetes, and each one maps to a Persistent Volume Claim(PVC) that is provided by the storage subsystem.
mysqldb
The MySQL database is the central data storage system for a Pliant instance. Information regarding users, permissions, authKeys, workflows, and other aspects of Pliant configuration are stored here. All sensitive information (authentication credentials) is encrypted. This container is not accessible from outside of the Pliant Kubernetes namespace.
Depends on: N/A
Dependency of: pliant-api
object-storage
A minio instance provides an internal object storage service. Configuration files and worker builds are stored here. This container is not accessible from outside of the Pliant Kubernetes namespace.
Depends on: N/A
Dependency of: pliant-api
pliant-stats
An InfluxDB instance stores statistical time-series data about workflow executions for reporting purposes.
Depends on: N/A
Dependency of: pliant-api
rabbitmq
Pliant uses an instance of the RabbitMQ message broker deployed as a Kubernetes StatefulSet. Workflow execution requests are queued here for the pliant-worker instances and execution results are queued here for handling by the pliant-api. This container is accessible externally to allow for communication from remote workers.
Depends on: N/A
Dependency of: pliant-api, pliant-worker
pliant-api
The pliant-api service is the central hub of Pliant. All communications from the user interface, databases, workers, and messagebus pass through the API. Configured as a Kubernetes Deployment, the pliant-api is scalable to serve an instance of any size.
Depends on: mysqldb, object-storage, pliant-stats, rabbitmq
Dependency of: pliant-worker, pliant-scheduler, pliant-app-gateway
pliant-app-gateway
The app gateway hosts the endpoints for custom APIs
Depends on: pliant-api
Dependency of: N/A
pliant-compiler
The compiler generates executable code from given user flows. The pliant-worker service communicates with the compiler at flow execution time.
Depends on: N/A
Dependency of: pliant-worker
pliant-db-migration
This service handles any MySQL database schema changes that need to be executed during a Pliant software upgrade.
Depends on: mysqldb
Dependency of: N/A
pliant-front
This service hosts the the Pliant web interface. It is transparently accessed through the pliant-proxy service and uses the pliant-api service to perform all actions.
Depends on: N/A
Dependency of: N/A
pliant-kv-store
This is an instance of the Redis memory based key-value store. It is used to store volatile and frequently accessed statistics such as the count of flow executions.
Depends on: N/A
Dependency of: N/A
pliant-proxy
The proxy is an Nginx reverse proxy that handles SSL offload and controls access to pliant-front, pliant-api, and rabbitmq services.
Depends on: N/A
Dependency of: N/A
pliant-scheduler
Pliant workflows can be scheduled to run periodically. The pliant-scheduler service handles triggering flow executions per the defined schedule.
Depends on: pliant-api
Dependency of: N/A
pliant-worker
The pliant-worker is the execution engine for the flows. Positioned behind a message bus, workers are both replaceable and unique at the same time. The workflow execution is orchestrated with the help of a persistent message bus with exactly-one delivery (within a reasonable time frame). Any flow trigger events and any other flow-related events are delivered through the message bus. The only exception is the flow itself as well as any other sensitive information (i.e. credentials for integrations) which is available to the worker nodes through a secure and authorized channel to the API. Worker nodes are also special users from the perspective of the API. The role given the workers allows them to access the sensitive information needed to run users' flows. Strict measures are taken to ensure the security of such information as well as hiding it from the business logic of user-defined flows.
Depends on: pliant-api, rabbitmq
Dependency of: N/A
pliant-worker-nodejs-config
This service builds configuration files for pliant-worker based on the installed integration packages. When an integration package is added or removed, this service is triggered and creates a new configuration build for the workers and places it in object-storage. The worker will automatically restart when a new build becomes available and then it is able to use the new integration blocks.
Depends on: N/A
Dependency of: N/A
pliant-worker-nodejs-remote
This service recompiles remote worker binaries when integration packages are added or deleted. The compiled remote worker binaries are stored in object-storage and accessed via pliant-api.
Depends on: N/A
Dependency of: N/A
Add-ons / Plugins
addon-ansible
This addon allows Pliant workflows to execute Ansible runbooks.
Depends on: N/A
Dependency of: N/A
addon-git
This addon allows Pliant to interact with git based version control systems.
Depends on: N/A
Dependency of: N/A
addon-mqws
This is a web socket proxy that allows RabbitMQ connections from remote workers to operate over an HTTPS connection using port 443 instead of 5671.
Depends on: rabbitmq
Dependency of: N/A
addon-jinja
This addon works with the Jinja integration to provide processing of Jinja2 templates.
Depends on: N/A
Dependency of: N/A
addon-napalm
This addon provides an integration with network devices via NAPALM (Network Automation and Programmability Abstraction Layer with Multivendor support), a Python library that implements a set of functions to interact with different network device Operating Systems using a unified API.
Depends on: N/A
Dependency of: N/A
addon-textfsm
This addon provides for the use of templates for the TextFSM Python module in workflows.
Depends on: N/A
Dependency of: N/A
addon-themes
This addon allows users to customize the appearance of the Pliant web interface.
Depends on: N/A
Dependency of: N/A
addon-trigger
This add-on provides a simple web interface for executing Pliant workflows with custom inputs and viewing the output. It is reachable at the URL: https://<pliant fqdn or ip>/plugin/trigger (Soon to be deprecated)
Depends on: N/A
Dependency of: N/A
Pliant Communication With Third Parties
Inbound
Pliant listens on the following TCP ports for inbound requests:
80 (HTTP): Used for a redirect to 443 (HTTPS)
443 (HTTTPS): All web interface, API, inbound webhook, and trigger connections are over this connection
5671 (AMPQS): Remote worker connections to the RabbitMQ message bus (optional after Pliant release 2021.9 and Remote Worker OVA version 1.2.N+, only required for binary remote workers)
Outbound
Pliant is capable of establishing outbound connections on an ever increasing number of protocols. Associated TCP or UDP ports are configured in the respective authKeys
SMTP over plaintext, SSL, or TLS
SSH
SFTP
HTTP
HTTPS (Most ReST and SOAP API integrations will use this protocol)
MongoDB
MySQL
MSSQL
Syslog