New TheHive and Cortex AMIs now available with improved built-in operation tools

The latest TheHive and Cortex AMIs are now available on the AWS marketplace. In addition to the usual application and OS updates, we included a few new features to make the operations side of things even easier.

Built-in CloudWatch agent

First off is the CloudWatch agent which is now baked into both the Cortex and TheHive AMIs. The agent is installed and provided with a default configuration file located at: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json

The agent can be activated and started using the following commands in your cloud-init bootstrap script:

runcmd:
    — [ systemctl, daemon-reload ] 
    — [ systemctl, enable, amazon-cloudwatch-agent.service ] 
    — [ systemctl, start, amazon-cloudwatch-agent.service ]

To opt for a different configuration, you can either replace the configuration file entirely before starting the service or you can import an additional configuration file using the append-config option instead of fetch-config. The complete documentation for using multiple configuration files can be found here.

Remember that:
- Your instance role must include the CloudWatchAgentServerPolicy to allow pushing logs to CloudWatch
- By default, log data is stored in CloudWatch Logs indefinitely! You probably want to set a retention period that fits your requirements.

Backup / snapshot scripts

While there are plenty of available options to backup your EBS data volumes, not all of them will allow snapshotting multiple data volumes in a consistent way. Some native backup services will start the snapshots of multiple EBS volumes within a given time window (usually around 15 minutes). This can be problematic for TheHive where we have the database, the database indexes and the attachments storage located on three different volumes. We definitely want them to be systematically backed up in a consistent way.

We now provide a simple script to easily perform consistent snapshots of your TheHive or Cortex EBS data volumes. The script will stop all relevant services, perform the snapshots and restart the services. It can also stop and restart a monitoring service if needed to avoid any false alarms.

The script for TheHive is located at: /opt/thehive/ops/scripts/ops-thehive4-data-backup.sh

The script for Cortex is located at: opt/cortex/ops/scripts/ops-cortex-data-backup.sh

Both scripts accept two inputs:
- An optional snapshot retention period (in days): every time the script is executed, it will locate previous snapshots and tag them as Key:Expired, Value:True if they were started prior to the retention period. If no retention period is provided, the older snapshots are not tagged.
- An optional monitoring service name to stop / restart.

The scripts are ready to use and can be scheduled when launching a TheHive or Cortex instance.

Here is some sample code to add to your cloud-init bootstrap script to schedule a daily backup of TheHive volumes, keep snapshots for 7 days (after which they are flagged as Key:Expired, Value:True but not deleted) and stop / restart the monitoring-service-name service:

write_files:
    - owner: root:root
      path: /etc/cron.d/thehive-data-backup
      content: |
           @daily root /opt/thehive/ops/scripts/ops-thehive4-data-backup.sh 7 monitoring-service-name

Here is the same sample code to add to your cloud-init bootstrap script to schedule a daily backup of Cortex volumes, keep snapshots for 7 days (after which they are flagged as Key:Expired, Value:True but not deleted) and stop / restart the monitoring-service-name service:

write_files:
    - owner: root:root
      path: /etc/cron.d/cortex-data-backup
      content: |
           @daily root /opt/cortex/ops/scripts/ops-cortex-data-backup.sh 7 monitoring-service-name

Note: While the required AWS CLI is already baked into both the TheHive and Cortex AMIs, your instance role must of course allow starting EBS snapshots and managing their tags.

NTP sync on AWS internal time source

TheHive and Cortex AMIs use chrony for time synchronisation. Prior to this release, the ntp.org pools were used as time sources. We opted to include the internal AWS time servers as the primary sources and keep the ntp.org sources as backup.

Webhooks configuration template

In compliance with our own recommended configuration, TheHive AMI now includes a webhooks configuration file template. This file can be used as a template to configure webhooks whenever you need to in the future. But, as with all configuration file templates, you can use it on brand new TheHive instances to automatically set up webhooks at first launch. Just replace the file content before launching the init script.

Here is some sample code to add to your cloud-init bootstrap script when initialising a new TheHive instance (with a new database and no existing data):

write_files:
    - path: /opt/thehive/ops/templates/webhooks.conf
      content: |
      	// ## Webhook notification
        // notification.webhook.endpoints = [
        //   {
        //    name: WEBHOOK_NAME
        //    url: "WEBHOOK_URL"
        //    version: 0
        //    wsConfig: {}
        //    auth: {type: "none"}
        //    includedTheHiveOrganisations: ["*"]
        //    excludedTheHiveOrganisations: []
        //  }
        // ]
runcmd:
    - [ /opt/thehive/ops/scripts/ops-thehive4-init.sh, /dev/sdh, /dev/sdi, /dev/sdj ]

We hope you enjoy these new AMI updates and as always, we look forward to your feedback at aws@strangebee.com