If you want to check Elastic Beanstalk (EB) logs, there are a few ways you can do it. You can use CloudWatch or send your application logs to an observability platform. Some might even download the logs directly from EB and check them, although this method is not very intuitive. Of course, you can also check via CloudWatch. In our case, we use OpenSearch. Since we are using it for some of our ECS applications, our logs are centralized and easy to observe. To achieve this, I was initially thinking of using Filebeat. However, I discovered that Filebeat is not compatible with AWS IAM roles. If you want to use an AWS IAM role, you have to sign every request to OpenSearch. Fortunately, I realized that we are already using Fluent Bit in our ECS, which includes AWS Sigv4 signature out of the box. So, we don’t have to manage anything. You can check the documentation here: link to Fluent Bit documentation on OpenSearch

To set up Fluent Bit, we need to prepare our .ebextensions folder. Inside the folder, I am setting it up as follows:

fluent.config

files:
  "/etc/fluent-bit/fluent-bit.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      [SERVICE]
          Flush        1
          Daemon       Off
          Log_Level    info
          Parsers_File /etc/fluent-bit/parsers.conf

      [INPUT]
          Name              tail
          Path              /var/log/web.stdout.log
          Parser            custom_parser
          Tag               web.stdout.log
          Refresh_Interval  5
          Rotate_Wait       5
          Mem_Buf_Limit     5MB
          Skip_Long_Lines   On

      [OUTPUT]
          name       es
          match      *
          host       es.us-west-1.es.amazonaws.com
          port       443
          tls        On
          tls.verify Off
          index      fluentbit-eb-logs
          Suppress_Type_Name On
          aws_auth   On
          aws_region us-west-1

  "/etc/fluent-bit/parsers.conf":
    mode: "000644"
    owner: root
    group: root
    content: |
      [PARSER]
          Name        custom_parser
          Format      regex
          Regex       ^(?<time>[^ ]* [^ ]* [^ ]*) (?<ip>[^ ]*) (?<app>[^:]*): (?<message>.*)$
          Time_Key    time
          Time_Format %b %d %H:%M:%S

commands:
  01_install_fluentbit:
    command: "curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh"
    cwd: "/tmp"
  02_start_fluentbit:
    command: "sudo systemctl start fluent-bit"

Let me explain what I am doing in the above code. Inside the /etc/fluent-bit/fluent-bit.conf file, I am setting up parsers and the OpenSearch host, among other things. I believe you can understand it by reading the code. There are a few issues that I encountered while setting up, which you should be aware of. You don’t need to put https:// in front of your OpenSearch domain. You need to set up Suppress_Type_Name. You can find more information about it here: link to Fluent Bit documentation on Suppress_Type_Name. It’s a common error.

I am using a custom parser for my logs and naming it custom_parser. You can check the available out-of-the-box parsers in the Fluent Bit documentation: link to Fluent Bit documentation on parsers

Now that our Fluent Bit configuration is ready, we need to set up our IAM role for OpenSearch. Go to OpenSearch Security and add your EC2 IAM role as follows. In our example, my EC2 instance IAM role is ec2-role. I have also added my IP to the whitelist.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::12345678:role/ec2-role"
        ]
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-1:12345678:domain/es/*"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "es:*",
      "Resource": "arn:aws:es:us-west-1:12345678:domain/es/*",
      "Condition": {
        "IpAddress": {
          "aws:SourceIp": [
            "{your_ip}/32"
          ]
        }
      }
    }
  ]
}

After finishing the setup of all necessary configurations, we can deploy and test it out. Once the deployment is complete, you will be able to see your logs coming into OpenSearch.

I hope this helps! Let me know if you have any further questions.