Forward AWS Logs via Lambda Shipper with Terraform

Overview

Our latest AWS integration provides the easiest way to connect with Coralogix. By using a predefined Lambda function, you can seamlessly send AWS logs and events to your Coralogix subscription for detailed analysis, monitoring, and troubleshooting.

This integration guide walks you through completing the predefined Lambda function template using Terraform. You’ll need to provide specific configuration parameters based on the service you want to connect. A reference list for these parameters is provided below.

Requirements

Providers

Modules

Universal configuration

You need to use an existing Coralogix Send-Your-Data API key to make the connection. Also, make sure your integration is region-specific. You should always deploy the AWS Lambda function in the same AWS region as your resource (e.g. the S3 bucket).

Using a single S3 bucket for multiple integrations

If you're deploying multiple integrations through the same S3 bucket, you'll need to specify parameters for each integration separately using integration_info.

!!! note\n If you already have a Lambda function with an S3 trigger set up, this Terraform deployment will remove the trigger. This applies to the following integration types within the same S3 bucket: S3, CloudTrail, VpcFlow, S3Csv, and CloudFront. To prevent this issue, you can explore alternative deployment methods: - Deploy the integration using CF Quick Create or SAR, as explained in this document. - Migrate your existing integrations to Terraform and use the integration_info variable.

Name	Description	Type	Default	Required
coralogix_region	The Coralogix location region, available options: [EU1, EU2, AP1, AP2, AP3, US1, US2, Custom]	string	n/a	yes
custom_domain	If using a custom domain name for your private cluster, Coralogix will send telemetry from the specified address (e.g. custom.coralogix.com). There is no need to add ingress. to the domain.	string	n/a	no
integration_type	The AWS service to integrate with Coralogix. Possible values: S3, CloudTrail, VpcFlow, CloudWatch, S3Csv, SNS, SQS, Kinesis, CloudFront, MSK, Kafka, EcrScan.	string	S3	yes
api_key	The Coralogix Send Your Data - API key validates your authenticity. This value can be a direct Coralogix API key or an AWS secret manager ARN containing the API key.	string	n/a	yes
store_api_key_in_secrets_manager	Enable this to store your API key securely. Otherwise, it will remain exposed in plain text as an environment variable in the Lambda function console.	bool	true	no
application_name	The name of your application. For a dynamic value, use $.my_log.field. This option is not supported since version 1.1.0 for the source code	string	n\a	yes
subsystem_name	The name of your subsystem. For a dynamic value, use $.my_log.field for CloudWatch log group leave empty. This option is not supported since version 1.1.0 for the source code	string	n\a	yes

S3, CloudTrail, Vpc Flow, S3Csv configuration

Additional parameters for integration_info

CloudWatch configuration

!!! note\n The log_group variable will get a list of log groups and then add them to the Lambda as triggers, each log group will also add permission to the Lambda, in some cases when there are a lot of log groups this will cause an error because the code tries to create too many permissions for the Lambda (AWS have a limitation for the number of permission that you can have for a Lambda), and this is why we have the log_group_prefix parameter, this parameter will add only permission to the Lambda using a wildcard( * ). for example, in case I have the log groups: log1,log2,log3 instead that the code will create for each of the log group permission to trigger the shipper Lambda then you can set log_group_prefix = ["log"], and then it will create only 1 permission for all of the log groups to trigger the shipper Lambda, but you will still need to set log_groups = ["log1","log2","log3"]. When using this parameter, you will not be able to see the log groups as triggers for the Lambda. If you need to add multiple log groups to the Lambda function using regex, refer to our Lambda manager

SNS configuration

SQS configuration

Kinesis configuration

MSK configuration

Kafka configuration

Generic configuration (optional)

Custom S3 bucket

Use the custom_s3_bucket variable only when deploying the integration in an AWS region where CX does not provide a public bucket (e.g., GovCloud). When using this variable, you must create an S3 bucket in the desired region for the integration. After that, pass the bucket name as custom_s3_bucket. The module will download the integration file to your local workspace, upload it to the custom_s3_bucket, and then remove the file from your local workspace once the process is complete.

When using this variable you will need to create an S3 bucket in the region where you want to run the integration. Then, pass this bucket name as custom_s3_bucket. The module will download the integration file to your local workspace, and then upload these files to the custom_s3_bucket. At the end, remove the file from your local workspace once the process is complete.

Lambda configuration (optional)

Metadata

The metadata features decribed below are only available in coralogix-aws-shipper v1.1.0 and later. The add_metadata parameter allows you to add metadata to the log message. The metadata is added to the log message as a JSON object. The metadata is specific to the integration type. For example, for S3, the metadata is s3.object.key and s3.bucket. For CloudWatch, the metadata is cw.log.group and cw.log.stream. See the table below for full metadata list.

Integration Type	Metadata Key	Description
S3	s3.bucket	The name of the S3 bucket
S3	s3.object.key	The key/path of the S3 object
CloudWatch	cw.log.group	The name of the CloudWatch log group
CloudWatch	cw.log.stream	The name of the CloudWatch log stream
Cloudwatch	cw.owner	The owner of the log group
Kafka	kafka.topic	The name of the Kafka topic
MSK	kafka.topic	The name of the Kafka topic
Kinesis	kinesis.event.id	The Kinesis event ID
Kinesis	kinesis.event.name	The Kinesis event name
kinesis	kinesis.event.source	The Kinesis event source
kinesis	kinesis.event.source_arn	The Kinesis event source ARN
Sqs	sqs.event.source	The SQS event source/queue
Sqs	sqs.event.id	The SQS event ID
Ecr	ecr.scan.id	The EXR scan ID
Ecr	ecr.scan.source	The ECR scan source

!!! note\n The metadata is not added by default. You must specify the metadata keys in the add_metadata parameter. For example, to add the bucket name and key name to the log message, set the add_metadata parameter to s3.object.key,s3.bucket. Some metadata keys will overlap as some integrations share the same metadata. For example, both Kafka and MSK have the same metadata key kafka.topic or both Kinesis and Cloudwatch metadata will be added in cases where a Cloudwatch log stream is being ingested from a Kinesis stream.

Dynamic subsystem or application name

As of v1.1.0 for the source code, you can use dynamic values for the application and subsystem name parameters based on the internal metadata defined above. To do accomplish this, use the following syntax:

{{ metadata.key | r'regex' }}

{{ s3.bucket }}

{{ cw.log.group }}

{{ s3.object.key | r'AWSLogs\\/.+\\/(.*)$' }}

!!! important\n - The regex must be a valid regex pattern. - The regex must define a capture group for part of the string you want to use as the value - The metadata key must exist in the list defined above and be a part of the integration type that is deployed.

Dynamic values are only supported for the application_name and subsystem_name parameters, the custom_metadata parameter is not supported.

To use a json key value as the application name, set the application_name parameter to:

{{ $.json_key_path }}

"json_log" : {
  {
    "application_key": "application_value"
  }
}

{{ $.json_log.application_key }}

VPC configuration (optional)

DLQ

A Dead Letter Queue (DLQ) is a queue where messages are sent if they cannot be processed by the Lambda function. This is useful for debugging and monitoring.

To enable the DLQ, provide the required parameters listed below.

Parameter	Description	Default Value	Required
enable_dlq	Enable the Dead Letter Queue for the Lambda function.	false	yes
dlq_s3_bucket	An S3 bucket used to store all failure events that have exhausted retries.		yes
dlq_retry_limit	The number of times a failed event to be retried before being saved in S3	3	yes
dlq_retry_delay	The delay in seconds between retries of failed events	900	yes

AWS PrivateLink

If you want to bypass using the public internet, you can use AWS PrivateLink to facilitate secure connections between your VPCs and AWS services. This option is available under VPC Configuration. For additional instructions on AWS PrivateLink, follow our dedicated tutorial.

Cloudwatch metrics streaming via PrivateLink (beta)

As of Lambda source code version v1.3.0 and Terraform module version v2.6.0, the Coralogix AWS Shipper supports streaming Cloudwatch metrics to Coralogix via Firehose over a PrivateLink.

This workflow is designed for scenarios where you need to stream metrics from a CloudWatch metrics stream to Coralogix via a PrivateLink endpoint.

Why to use this workflow

AWS Firehose does not support PrivateLink endpoints as a destination because Firehose cannot be connected to a VPC, which is required to reach a PrivateLink endpoint. To overcome this limitation, the Coralogix AWS Shipper acts as a transform function. It is attached to a Firehose instance that receives metrics from the CloudWatch metrics stream and forwards them to Coralogix over a PrivateLink.

When to use this workflow

This workflow is designed to bypass the limitations of using Firehose with the Coralogix PrivateLink endpoint. If PrivateLink is not required, we recommend using the default Firehose integration for CloudWatch Stream Metrics, available here.

How does it work?

To enable CloudWatch metrics streaming via Firehose (PrivateLink), you must provide the necessary parameters listed below.

Parameter	Description	Default Value	Required
telemetry_mode	Specify the telemetry collection modes, supported values (metrics, logs). Note that this value must be set to metrics for the Cloudwatch metric stream workflow	logs.
api_key	The Coralogix Send Your Data - API key validates your authenticity. This value can be a direct Coralogix API key or an AWS secret manager ARN containing the API key.	string	n/a
application_name	The name of the application for which the integration is configured. Metadata specifies dynamic value retrieval options.	string	n\a
subsystem_name	The name of your subsystem. For a dynamic value, refer to the Metadata section. For CloudWatch, leave this field empty to use the log group name.	string	n\a
coralogix_region	The Coralogix location region, possible options are [EU1, EU2, AP1, AP2, AP3, US1, US2, Custom]	string	n/a
s3_bucket_name	The S3 bucket to be used to store records that have failed processing.
subnet_ids	The ID of the subnet for the integration deployment.	list(string)	n/a
security_group_ids	The ID of the security group for the integration deployment.	list(string)	n/a
include_metric_stream_filter	List of inclusive metric filters. If you specify this parameter, the stream sends only the conditional metric names from the specified metric namespaces. Leave empty to send all metrics	llist(object({namespace=string, metric_names=list(string)})	n/a

The include_metric_stream_filter example:

module "coralogix_aws_shipper" "coralogix_firehose_metrics_private_link" {
  source = "coralogix/aws-shipper/coralogix"
  telemetry_mode = "metrics"
  api_key = <your private key>
  application_name = "application_name"
  subsystem_name = "subsystem_name"
  coralogix_region = <coralogix region>
  s3_bucket_name = <s3 bucket name>
  subnet_ids = <subnet ids>
  security_group_ids = <security group ids>

  include_metric_stream_filter = [
    {
      namespace    = "AWS/EC2"
      metric_names = ["CPUUtilization", "NetworkOut"]
    },
    {
      namespace    = "AWS/S3"
      metric_names = ["BucketSizeBytes"]
    },
  ]
}

Outputs

Previous Forward AWS Logs via Lambda Shipper

Next Send Logs using Amazon Data Firehose

Theme

Light

light

dark