--- title: "Installation and Setup v1.4.0" canonical: "https://atomwatch.kitepipe.com/space/CS/212795410/Installation%20and%20Setup%20v1.4.0" format: markdown --- # 1 Overview AtomWatch comes with four dashboards. You are encouraged to take these steps immediately upon installing AtomWatch: 1. Clone the dashboards so you can begin customizing them. The originals are read-only. 2. Install all recommended Monitors that are part of AtomWatch. 3. Subscribe to alerts on these monitors as desired. AtomWatch by Kitepipe is an Agent-based integration that collects metrics from Boomi processes, cluster nodes, and related infrastructure to inform both Datadog and Boomi customers about the health of the integration. AtomWatch version 1.4 contains 7 dashboards, 17 custom metrics, and 16 monitors that report on Boomi execution statistics, cluster status, JMX monitoring, and infrastructure health. These metrics are available to Datadog and Boomi customers for extended time-trending analysis (over the standard of 30 days for Boomi Process Reporting availability). Datadog customers who purchase AtomWatch must manage the Boomi Java Runtime in either an Atom or Molecule configuration. Kitepipe includes a one hour set-up and configuration session with the initial 14 Day Free Trial. # 2 [About Kitepipe](https://www.kitepipe.com/) Kitepipe is a Boomi Platinum Implementation Partner, and is the premier Boomi integration development team in North America. Kitepipe was founded in 2011 in response to the need for a Boomi-focused services team that could deliver all the promises of this powerful integration platform. Today, the Kitepipe team of certified Boomi on-shore developers help dozens of Boomi customers quickly achieve business value with the industry-leading Boomi integration platform. The Datadog service AtomWatch is a new offering from Kitepipe with a focus on Boomi managed services in AWS. Kitepipe is the leader in a number of integration areas, verticals, and domains, including AWS migrations of Boomi processes, AWS managed Boomi, Biotech vertical solutions built on Boomi, NetSuite, SAP, Coupa, Oracle, Workday, and HRIS, Data Mart/BI, and more endpoints. # 3 Dashboards The following dashboards are available to AtomWatch customers. | Name | Purpose | | --- | --- | | **AtomWatch Overview for Boomi** | A dashboard displaying a high-level summary of Compute, Cluster, and Workload metrics. | | **Boomi Workload Monitoring** | A dashboard displaying enhanced Process Reporting with associated graphs. | | **Boomi Compute Monitoring** | A dashboard displaying infrastructure metrics to check resource health. | | **Boomi Cluster Monitoring** | A dashboard about [Boomi-recommended cluster status inspection](https://help.boomi.com/bundle/integration/page/r-atm-Cluster_Status_panel.html). | # 4 Log Collection This integration makes API calls to the Boomi Platform on your behalf, retrieving execution records and sending them to Datadog as logs. This integration also optionally monitors in-progress executions and JVM telemetry via JMX and sends this information to Datadog as logs. You can see which Boomi processes are running in which JVMs, and the associated memory, garbage collection, thread count, and other information. ### CONTAINER LOGS: To configure the Datadog Agent to ship Boomi container logs to your Datadog account, perform these steps *on each molecule node*: - Create the following file at `/etc/datadog-agent/conf.d/BoomiContainerLog.d/conf.yaml` (or equivalent Windows location), replacing with appropriate values for your setup. --- ``` --- logs: - type: file path: /logs/*.container..log service: BoomiContainerLog source: kitepipe-boomi log_processing_rules: - type: multi_line name: multi_line_rule pattern: \w+ \d+, \d+ \d+:\d+:\d+ (AM|PM) ``` - In the `datadog.yaml` file, ensure the following line is present and uncommented: ``` logs_enabled: true ``` ### HTTP LOGS: To configure the Datadog Agent to ship Boomi HTTP logs to your Datadog account, perform these steps *on each molecule node*: - Create the following file at `/etc/datadog-agent/conf.d/BoomiHTTPLog.d/conf.yaml` (or equivalent Windows location), replacing with appropriate values for your setup. --- ``` --- logs: - type: file path: /logs/*.shared_http_server..log service: BoomiHTTP source: kitepipe-boomi log_processing_rules: - type: exclude_at_match name: exclude_healthcheck pattern: _admin/status ``` - In the `datadog.yaml` file, ensure the following line is present and uncommented: ``` logs_enabled: true ``` # 5 List of Recommended Monitors The following monitors are provided. You can subscribe to alerts on each of these monitors, and create others as needed for your use cases: | Name | Purpose | | --- | --- | | **Execution Duration Anomaly** | Detect long- and short-running executions. | | **AtomWatch is Down** | Detect if AtomWatch has stopped working. | | **Boomi "View File" is missing** | Detect if Boomi is not running on a cluster node. | | **Boomi “View File” is too old** | Detect if a cluster node is offline or operating with lag. | | **Boomi “View File” reports a problem** | Detect if a cluster node has reported a problem. | | **Infrastructure - Molecule Node Disk Usage High** | Detect if a molecule node is running out of disk space. | | **Infrastructure - Molecule Node CPU Usage High** | Detect if a molecule node is at high CPU usage. | | **Infrastructure - Molecule Node Memory Usage High** | Detect if a molecule node is at high RAM usage. | | **Infrastructure - API Gateway Node Disk Usage High** | Detect if an API gateway node is running out of disk space. | | **Infrastructure - API Gateway Node CPU Usage High** | Detect if an API gateway node is at high CPU usage. | | **Infrastructure - API Gateway Node Memory Usage High** | Detect if an API gateway node is at high RAM usage. | | **Runtime Online Status** | Detect if runtimes are reporting heartbeat to Boomi datacenter | | **Failure calling Boomi Platform API** | When AtomWatch cannot reach the Boomi Platform API, then some dashboard widgets are not up-to-date and some monitors will fail to alert as desired.  This monitor is designed to tolerate a chosen length of Boomi Platform API downtime. | | **Boomi API Calls From More Than One Node** | If you have accidentally configured more than one node to make Boomi Platform API calls, this monitor will go into an alert state. If you have configured AtomWatch to let the “head node” make API calls, and your molecule enters a state of multiple “head nodes”, this monitor can also go into an alert state. While in alert state, there may be double-reporting of your Boomi executions into Datadog. | | **Runtime Low Memory** | This monitor detects when a runtime self-reports as being in an Low-Memory state. Requires JMX monitoring to be active. | | **Runtime Out Of Memory** | This monitor detects when a runtime self-reports as being in an Out-Of-Memory state. Requires JMX monitoring to be active. | # 6 Events This integration retrieves AuditLog records from the Boomi API, and sends them to Datadog as events. The events are visible in filtered form in the Boomi Workload Monitoring Dashboard or in the [Events Explorer](https://app.datadoghq.com/event/explorer). You can build your own monitors to inspect the unfiltered AuditLog records. # 7 Metrics You can explore a list of metrics in the **Data Collected** tab. Metrics cover the following categories: | Type | Categories | Definition | | --- | --- | --- | | Workload Metrics | - Executions - Documents | Enhanced version of Boomi Process Reporting data. | | Cluster Metrics | - File Existence - File Age - Reported Problems | Enhanced versions of Boomi Cluster Status data. | | Infrastructure Metrics | - CPU - Network - Disk - RAM | Standard Datadog infrastructure metrics in these categories. | | Boomi JMX Metrics | - CPU Time - Heap Memory - Runtime LoMem and OutOfMem | Information from your JVMs that are running or managing Boomi processes. | # 8 Setup Follow the instructions below to install and configure this integration for an Agent running on a host server. Containerized environments are not supported. Kitepipe support engineers are available to assist customers in configuring and installing the Boomi Monitoring suite. Several advanced features require mapping of server or node to Boomi resources in the config file. See the [Support](https://github.com/DataDog/marketplace/tree/kitepipe_boomiwatch/kitepipe_boomiwatch#support) section for more details. ## 8.1 Prerequisites 1. Boomi Atom or Molecule running on Linux server or virtual server (not containerized). 1. NOTE: Molecule must NOT be installed in any `/home` directory due to Linux permissions issues. There is a workaround for Atoms installed in `/home` directory; please contact us for details. 2. Optionally, Boomi API Gateway running on server or virtual server. 3. Ability to remote into each server with admin permissions and edit files using the command line. 4. Admin privileges in your Boomi account. 5. Knowledge of how to create a user in Boomi, assign permissions, and [generate an API token](https://help.boomi.com/bundle/atomsphere_platform/page/int-Adding_API_tokens.html). 6. An email address to use as the login ID for a newly created Boomi user. ## 8.2 Boomi User Creation This Datadog integration makes calls to the Boomi API on your behalf. You must supply Boomi user credentials to be used in these API calls. It is recommended to create a service user identity for this purpose. Follow these steps: 1. In your Boomi account, create a service user for use with AtomWatch. If your Boomi account has Custom Roles, create a Custom Role with the following permissions. Otherwise, use the built-in role “Boomi API Role” which has these permissions: - API Access - Atom Management Read Access - Build Read Access - Execute - View Audit Logs - View Data - View Results 2. Apply this role to the newly created service user. 3. [Add this role](https://help.boomi.com/bundle/integration/page/t-atm-Attaching_a_role_to_an_Environment.html) to every Boomi Environment that you want to monitor. NOTE: If you apply the role to unwanted environments, metrics and logs from those environments will show in the Datadog dashboards. This will not increase your AtomWatch per-node cost. However, you may not wish to see this data in Datadog. In this case, you will have to add filters in your Datadog dashboards to hide the unwanted environments. 4. Log in to Boomi as this user and generate an API token. 5. Save this user's login ID and the API token for later use. ## 8.3 Installing the Agent > ⚠️ The Datadog Agent must be installed on each node of the Molecule or API Gateway. > ⚠️ The Datadog Agent must be the latest version. Upgrade your existing Datadog Agents using the [in-app instructions](https://app.datadoghq.com/fleet/install-agent/latest?platform=overview). 1. If not already installed and upgraded, [download and install the Datadog Agent](https://app.datadoghq.com/fleet/install-agent/latest?platform=overview) on the node. 2. Install AtomWatch with this command: `sudo -u dd-agent datadog-agent integration install --third-party datadog-kitepipe-atomwatch==1.4.0` ## 8.4 Prepare the "Last Datetime File" As part of making Boomi AtomSphere API calls, the integration stores a timestamp in a text file on your Molecule shared disk or Atom installation directory. You must create this file and set permissions manually. You only need to do this once because only one server or node will be making API calls. - Remote into a Molecule node or your Atom server. - From the [Boomi Atom Management - Startup Properties panel](https://help.boomi.com/bundle/integration/page/r-atm-Startup_Properties_panel.html), determine your Atom Installation Directory. You will use this in later steps. - Create an empty file owned by `dd-agent` using this command: `sudo touch /work/kitepipe-atomwatch-last-end-datetime.txt && sudo chown dd-agent:dd-agent /work/kitepipe-atomwatch-last-end-datetime.txt` - NOTE: Known problems arise if running Boomi as root user. As a workaround, if AtomWatch gets “permission denied” errors when reading/writing the “Last Datetime File”, ensure that the ownership of the “work” folder allows read and write access to “other” users. As a best practice, run Boomi as non-root user. In any case, you can review the [“Event Explorer” in the Datadog console](https://app.datadoghq.com/event/explorer?query=%22AtomWatch%20Datetime%20Range%22) to see errors or warnings regarding the “Last Datetime File”. Further error messages can be seen in the Datadog Agent Logs at ## 8.5 Configuring the Agent ### 8.5.1 Edit datadog.yaml In the `# tags` section, add the following values: ``` - env: - role: - boomi-hostname:"" - integration:kitepipe-atomwatch ``` Uncomment `# process_config` and configure as follows: ``` process_config: enabled: true ``` ### 8.5.1 Edit conf.yaml An example configuration file can be found at `/conf.d/kitepipe_atomwatch.d/conf.yaml.example`. Copy this to `conf.yaml` in the same directory using these commands: ``` cd /conf.d/kitepipe_atomwatch.d sudo cp conf.yaml.example conf.yaml sudo chown dd-agent:dd-agent conf.yaml ``` Then, edit the `conf.yaml` as follows: ``` instances: - dd_api_key: site: ## Only one runtime should call the Boomi Platform API. The API calls will return information on all permitted runtimes in the Boomi account. ## Therefore, please uncomment and configure the following on all nodes of *ONLY ONE* runtime, i.e. one Atom, Molecule, or API Gateway: ## # boomi_api_url: https://api.boomi.com # make_api_calls_from_head_node: true # boomi_api_userid: # boomi_api_token: # boomi_account_id: ## Required ## boomi_atom_or_molecule_install_dir: ## Optional - UNCOMMENT AND SUPPLY VALUES IF APPLIABLE ## # boomi_api_gateway_install_dir: # boomi_molecule_node_id: '' # boomi_api_gateway_node_id: '' ## Optional - UNCOMMENT AND SUPPLY VALUES IF MONITORING JMX ## You must also configure and enable JMX as noted in the relevant Boomi ## help pages. ## See Kitepipe AtomWatch documentation ("Further Reading" section below) ## for example of supported configurations. ## IMPORTANT: Do not allow too large a port range; this will impact Datadog ## billing by sending a large number of unique tag values. # # path_to_jmxterm_jar: # lowest_jmx_port: # highest_jmx_port: ``` - Restart the Datadog Agent: `sudo systemctl restart datadog-agent`. - Verify that Datadog Agent is running the `kitepipe_atomwatch` check by running: `sudo -u dd-agent datadog-agent status`. If successful, you should see a recent "Last Successful Execution Date" as in the example below: ``` kitepipe_atomwatch (1.4.0) -------------------------------- Instance ID: kitepipe_atomwatch:3dcc48da6d736c4a [OK] Configuration Source: file:/etc/datadog-agent/conf.d/kitepipe_atomwatch.d/conf.yaml Total Runs: 1 Metric Samples: Last Run: 12, Total: 12 Events: Last Run: 0, Total: 0 Service Checks: Last Run: 7, Total: 7 Average Execution Time : 55ms Last Execution Date : 2023-12-17 15:18:21 UTC (1671290301000) Last Successful Execution Date : 2023-12-17 15:18:21 UTC (1671290301000)` ``` # 9 Uninstallation - Remote into each node where AtomWatch was installed. - Uninstall the Kitepipe AtomWatch Integration from the Datadog Agent: `$ sudo -u dd-agent datadog-agent integration remove datadog-kitepipe_atomwatch`. - Delete the `/conf.d/kitepipe_atomwatch.d/conf.yaml` file. - Restart the Datadog Agent: `sudo systemctl restart datadog-agent`. - Verify that AtomWatch integration is no longer running: `sudo -u dd-agent datadog-agent status`. You should no longer see a "kitepipe_atomwatch" section in the output. - Log on to the Boomi console. - Revoke the API token you created during AtomWatch installation. - Delete the Boomi user you created during AtomWatch installation. # 10 Support AtomWatch customers may send a troubleshooting request to `AtomWatch.Support@kitepipe.com`. Kitepipe support hours for AtomWatch are designated during the business hours of 9AM to 3PM across US and Canadian time zones. AtomWatch troubleshooting requests will be answered within 24 to 48 hours from the notification receipt to the AtomWatch email alias. For best response results, include the customer name, Boomi configuration, and a brief description of the event or troubleshooting question. Enhanced support programs are available from Kitepipe upon request. # 11 AtomWatch JMX Monitoring *NOTE: The Datadog agent supports general JMX monitoring; this behavior is outside the scope of AtomWatch. Please see the **[relevant Datadog help pages](https://docs.datadoghq.com/integrations/java/?tab=host)** to configure general JMX monitoring, separate from AtomWatch, if desired.* To use AtomWatch JMX Monitoring, you must obtain the **[JMXTerm open-source JAR file](https://github.com/jiaqi/jmxterm/releases/tag/v1.0.4)**[ ](https://github.com/jiaqi/jmxterm/releases/tag/v1.0.2)and place it where each node can access it. AtomWatch has been tested with JMXTerm v1.0.4. Uncomment and configure the `path_to_jmxterm_jar`, `lowest_jmx_port`, and `highest_jmx_port` values in the Datadog AtomWatch conf.yaml file. To begin AtomWatch JMX Monitoring, first configure and enable JMX on your runtime by following the [relevant Boomi help pages](https://help.boomi.com/docs/Atomsphere/Integration/Integration%20management/t-atm-Enabling_remote_JMX_on_an_Atom_1a1625d0-330d-43c6-a765-42502d7768ec). Only the following configurations are supported: - JMX port must be on localhost. - JMX must be configured for no SSL, no authentication - You may specify any port number that does not conflict with other processes running on the host. - You may specify port 0 which causes JMX to listen at an ephemeral port; however, this will incur higher Datadog charges due to a large number of tag values. You may customize your JVM launch scripts to restrict the JMX port range as desired. - Forked and non-Forked Execution are supported. - Atoms and Molecules are supported. Notes: - AtomWatch JMX Monitoring will inspect “ExecutionManager” JVMs and “ProcRunner” JVMs only. It has not been tested with “Browse” or ”AtomWorker” JVMs. You may use native Datadog JMX monitoring for those JVMs if desired. - AtomWatch JMX records a single timestamp after performing multiple JMXTerm queries per JMX port. This single timestamp is used when delivering logs to Datadog. It can be used to correlate log entries to visualize which Boomi processes were running in which JVMs at a given moment. # 12 Additional Setup Some dashboard widgets will require additional setup steps. Follow the instructions below to set up these widgets. ## Execution Duration Anomaly Widget On the AtomWatch Overview dashboard, edit the “Execution Duration Anomaly” widget by clicking the pencil icon: ![image](media://e7683fef-d837-49c5-938c-45f72ca43497) You will see this screen below. Edit the dropdown that says “Something went wrong…”: ![image](media://afc4411e-f531-4db4-8dbf-1c9072841251) Change the dropdown to look like the following: ![image](media://c79d1840-96a4-4ca2-8d14-2c2888869e28) At the bottom right of the screen, click “Save”: ![image](media://48deba08-f2c4-4899-ac56-511eb7301821) The widget will start showing data. It is OK if there is a message about “not enough historical data”. It will take up to a week for the algorithm to gather enough weekly seasonality data to detect anomalies: ![image](media://e4987883-2c59-4235-bf3c-84fc3326d1ac) *NOTE: Be sure to make this same edit on the Execution Duration Anomaly widget on the Boomi Workload Monitoring dashboard.* ## “View File age” widget On the AtomWatch Overview dashboard, edit the “View File age” widget by clicking the pencil icon: ![image](media://ec4a890d-1d18-49da-99a2-ea24f344a309) You will see this screen below. Edit the dropdown that says “Something went wrong…”: ![image](media://484c17bb-a601-48f1-a0e5-9dc594adca3f) Change the dropdown to look like the following: ![image](media://47c75376-c28f-4072-bc79-848e86657606) At the bottom right of the screen, click “Save”: ![image](media://48deba08-f2c4-4899-ac56-511eb7301821) The widget will start showing data: ![image](media://9d16e3fe-018f-4c38-b0b9-99a5a1c29c06)