AWS Resource Metadata Collection

Deploy the AWS Resource Metadata Collection AWS Lambda function in your AWS account. The function collects metadata of EC2 instances and AWS Lambda functions in the region of your AWS account and sends them to Coralogix.

What you'll find

This tutorial presents:

• Configuration of the AWS Resource Metadata integration for standard users

• Configuration of the AWS Resource Metadata (High Volume) integration for users with more than 5,000 Lambda functions

Overview

AWS resources can be vast and interconnected. To better understand log data and troubleshoot issues, it's important to have context about which AWS resources are involved. The AWS Resource Metadata Collection integration collects information about AWS resources that are associated with log events. This contextual information can include details about the AWS service, resource tags, AWS region, timestamps for resource creation or modification, and any relevant custom tags specific to the organization's AWS environment.

Benefits

The collection of EC2 instance and Lambda function metadata serves as a foundation for better AWS resource management, optimization, security, and efficient troubleshooting. It helps you make informed decisions and take actions based on a more comprehensive understanding of your AWS resources and their interactions.

• Improve troubleshooting. The metadata can provide valuable operational insights into your AWS environment. You can analyze resource-specific patterns, performance trends, and utilization to ensure your applications run smoothly. Having resource context helps in identifying the source of issues more quickly and accurately.

• Resource optimization. By collecting metadata, you can understand the relationships between different AWS resources, such as how Lambda functions interact with specific EC2 instances or other services. This can be essential for monitoring and managing complex AWS architectures, and making informed decisions about resource usage and optimization.

• Security and compliance. Resource metadata can assist in security monitoring, compliance reporting, and auditing.

• Cost management. Understanding resource attributes can be valuable for cost management and allocation.

Prerequisites

• AWS account

• Permissions to create Lambda functions

• If you are using Secret Manager you should first deploy the SM Lambda layer. Note that you should only deploy one layer per region.

Standard configuration

STEP 1. In your navigation pane, click Data Flow > Integrations. View the list of available integrations.

STEP 2. Select AWS Resource Metadata.

STEP 3. Click + ADD NEW.

STEP 4. Input the integration details.

• Input a name for your integration.

• Select the authentication type, either APIKey or Existing Secret.

  • If using an API key, input an existing Coralogix Send-Your-Data API Key or click CREATE NEW KEY.

  • If using an existing secret, enter the AWS Secret Name.

• Mark the Collect Aliases checkbox if you want to collect the aliases of the resources.

• Select your AWS Region from the dropdown list.

• If you want to use AWS PrivateLink, click Advanced Settings and mark the Use AWS PrivateLink checkbox. AWS PrivateLink is a service that facilitates secure and private connections between VPCs and AWS services, bypassing the need for the public internet. It is worth noting that the integration might not succeed if AWS PrivateLink is not properly set up.

STEP 5. Click NEXT.

STEP 6. View the instructions for your integration, then click CREATE CLOUDFORMATION.

STEP 7. You will be rerouted to the AWS website. Verify that all of the auto pre-populated values are correct, then click the acknowledgement checkboxes, and click Create Stack.

STEP 8. Go back to the Coralogix application and click COMPLETE to ensure your deployment is successful. This triggers a test to verify the deployment, the result of which can be seen on the next page as either Failed or Connected.

STEP 9. View your integration information.

STEP 10. Upon successful deployment, leverage the Coralogix APM Serverless Monitoring feature to access detailed insights into the Lambda functions operating within the deployed region.

High Volume configuration

For scenarios where you have more than 5,000 Lambda functions, you need to use the AWS Resource Metadata (High Volume) integration. It's a more advanced version of the Resource Metadata integration, designed to:

1. Handle larger volumes of metadata from 5,000 up to 100,000 Lambda functions in the target AWS region.
2. Support cross-account and multi-region collection of metadata from multiple AWS accounts.

For deployment, follow the same steps, but select the AWS Resource Metadata (High Volume) integration in STEP 2.

Cross-Region Collection

To enable cross-region collection within an account, simply add the SourceRegions parameter. You don't need to do anything else.

Cross-Account Collection

This function supports cross-account collection of metadata from multiple AWS accounts.

This feature is disabled by default. Enable it in the Integration Details section.

All options for cross-account metadata collection:

• Disabled
• StaticIAM: IAM Cross-account roles mode, loops over account IDs (AccountIds), assuming IAM role in each account. IAM role should have the same name in each account (CrossAccountIAMRoleName).
• Config: AWS Config Resource Aggregator mode, runs a Query on AWS Config Aggregator (ConfigAggregatorName) deployed in the account where the resource-metadata is running. It also requires IAM cross-account roles setup.

There are many other ways to collect the metadata about Lambda functions and EC2 instances from different accounts. See the cross-account solutions doc for more details. The rest of those options can be implemented on demand.

Cross-account permissions

Here is the set of required IAM permissions that should be set on the target roles:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
            "ec2:DescribeInstances",
            "lambda:ListFunctions",
            "lambda:ListVersionsByFunction", 
            "lambda:GetFunctionConfiguration",
            "lambda:GetFunctionConcurrency",
            "lambda:ListTags",
            "lambda:ListAliases",
            "lambda:ListEventSourceMappings",
            "lambda:GetPolicy",
            "tag:GetResources"
      ],
      "Resource": "*"
    }
  ]
}

As you will know the exact functions' role ARNs only after the template is deployed, you need to follow these steps to make it work:

1. Create the roles in the target accounts, setting necessary IAM permissions, but without setting the trust relationship to the source account, since we don't know Lambda functions role ARNs yet.
2. Deploy the template with the CrossAccountIAMRoleName parameter, mentioning the target roles' name.
3. After the template is deployed, set the trust relationship to the source account, using the Lambda functions' role ARNs. If you're using a StaticIAM mode, make sure to include both collector and generator functions' role ARNs. If you're using a Config mode, you need to include only generator role ARN.

Here is an example of a trust relationship policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::123456789012:role/mystackname-GeneratorLambdaFunctionRole-randomid",
                    "arn:aws:iam::123456789012:role/mystackname-CollectorLambdaFunctionRole-randomid"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

After setting the trust relationship, the generator and collector functions will be able to assume the target roles and collect metadata from those accounts.

Event Mode

The High Volume integration supports the Event Mode feature. Event Mode allows you to create Lambda and EC2 resources in Coralogix on a near-real-time basis, starting metadata collection as soon as a new function or instance is created. It usually takes 3-5 seconds for the resource to appear in Coralogix after being created in AWS.

This feature is disabled by default. Enable it in the Integration Details section.

All options for Event Mode:

• Disabled
• EnabledWithExistingTrail – Skips the creation of a CloudTrail trail and S3 bucket. This option is used if there is already one trail running in the target AWS region.
• EnabledCreateTrail – Creates all resources, including CloudTrail trail and S3 bucket.

Events from other accounts/regions

You can route CloudTrail events from other accounts and regions to the function's SQS queue.

To make it work:

1. cross-account mode needs to be enabled
2. OrganizationId parameter has to be set
3. the function needs to have access to the source account by using IAM Assume Role
4. the Eventbridge rules have to use the following event pattern:

Lambda + EC2:

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "source": [
    "aws.ec2",
    "aws.lambda"
  ],
  "detail": {
    "eventSource": [
      "ec2.amazonaws.com",
      "lambda.amazonaws.com"
    ],
    "eventName": [
      "RunInstances",
      "CreateFunction20150331"
    ],
    "errorCode": [
      {
        "exists": false
      }
    ]
  }
}

Lambda only:

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "source": [
    "aws.lambda"
  ],
  "detail": {
    "eventSource": [
      "lambda.amazonaws.com"
    ],
    "eventName": [
      "CreateFunction20150331"
    ],
    "errorCode": [
      {
        "exists": false
      }
    ]
  }
}

EC2 only:

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "source": [
    "aws.ec2"
  ],
  "detail": {
    "eventSource": [
      "ec2.amazonaws.com"
    ],
    "eventName": [
      "RunInstances"
    ],
    "errorCode": [
      {
        "exists": false
      }
    ]
  }
}

Parameters and descriptions

Additional parameters for high-volume mode

Event Mode Parameters

Cross-account and cross-region parameters

Integration parameters

Support

Need help?

Our world-class customer success team is available 24/7 to walk you through your setup and answer any questions that may come up.

Feel free to reach out to us via our in-app chat or by sending us an email at support@coralogix.com.

Previous AWS VPC Flow Logs Terraform Module

Next AWS Resource Metadata Collection Terraform Module

Theme

Light

light

dark