AWS DevOps Professional is one of the toughest AWS examination as it not only tests knowledge but also requires extensive concentration during the 3 hours of your exam. It has got 6 domains, during this blog, I will cover the first domain which is Software Development Lifecycle.

Topics and AWS Services

Code Commit, triggers, notifications and integration with lambda functions Code Build Code Pipeline

Code Commit

Code commit - Version control with ability to understand changes to the code. Enabled by using a version control system such as Git AWS Code commit offers private git repository solution, no limit on repositories, fully managed and scalable. Secure and can be integrated with Jenkins, CodeBuild and other CI tools

Question: How would you restrict permissions to Junior developers to push and merge branches to AWS code commit Answer: You can do this using IAM policy and applying it to IAM group or role, here is an AWS blog on the same

Code commit integration with Lambda

I personally find this feature very cool, you can have a lambda function and setup a trigger based on Code Commit events! It's very simple,

• On AWS Console, open Lambda console and create a function, select python 3.
• Once you create a function, under configurations, Add a trigger
• Configure trigger as shown in screenshot below

For Python 3, add the following code to your Lambda function -

import json
import boto3

codecommit = boto3.client('codecommit')

def lambda_handler(event, context):
    #Log the updated references from the event
    references = { reference['ref'] for reference in event['Records'][0]['codecommit']['references'] }
    print("References: "  + str(references))

    #Get the repository from the event and show its git clone URL
    repository = event['Records'][0]['eventSourceARN'].split(':')[5]
    try:
        response = codecommit.get_repository(repositoryName=repository)
        print("Clone URL: " +response['repositoryMetadata']['cloneUrlHttp'])
        return response['repositoryMetadata']['cloneUrlHttp']
    except Exception as e:
        print(e)
        print('Error getting repository {}. Make sure it exists and that your repository is in the same region as this function.'.format(repository))
        raise e

Here is a link to the full article

Code Build

Fully Managed Build service which is alternative to build tools like Jenkins Integration with KMS for encryption of build artifacts, IAM for build permission and VPC for network security, Cloud Trail for API calls logging

Points to remember -

1) Integration with source code from GitHub, Code Commit, S3 etc 2) Build instructions can be defined in buildspec.yml file 3) Logs can be sent to Cloudwatch or S3 4) Metrics to monitor Code Build statics 5) Cloud Watch events can detect failed builds and trigger notifications

Build spec.yml file -

This is a hear of Code Build, You can specify different phases of install, prebuild, build, post build, environment variable, parameter store, finally block. Here is an example of buildspec.yml file

version: 0.2

env:
  variables:
    JAVA_HOME: "/usr/lib/jvm/java-8-openjdk-amd64"
  parameter-store:
    LOGIN_PASSWORD: /CodeBuild/dockerLoginPassword

phases:
  install:
    commands:
      - echo Entered the install phase...
      - apt-get update -y
      - apt-get install -y maven
    finally:
      - echo This always runs even if the update or install command fails 
  pre_build:
    commands:
      - echo Entered the pre_build phase...
      - docker login -u User -p $LOGIN_PASSWORD
    finally:
      - echo This always runs even if the login command fails 
  build:
    commands:
      - echo Entered the build phase...
      - echo Build started on `date`
      - mvn install
    finally:
      - echo This always runs even if the install command fails
  post_build:
    commands:
      - echo Entered the post_build phase...
      - echo Build completed on `date`

reports:
  arn:aws:codebuild:your-region:your-aws-account-id:report-group/report-group-name-1:
    files:
      - "**/*"
    base-directory: 'target/tests/reports'
    discard-paths: no
  reportGroupCucumberJson:
    files:
      - 'cucumber/target/cucumber-tests.xml'
    discard-paths: yes
    file-format: CUCUMBERJSON # default is JUNITXML
artifacts:
  files:
    - target/messageUtil-1.0.jar
  discard-paths: yes
  secondary-artifacts:
    artifact1:
      files:
        - target/artifact-1.0.jar
      discard-paths: yes
    artifact2:
      files:
        - target/artifact-2.0.jar
      discard-paths: yes
cache:
  paths:
    - '/root/.m2/**/*'

Environment Variable and it's type :

The type of environment variable. Valid values include:

PARAMETER_STORE: An environment variable stored in Systems Manager Parameter Store. To learn how to specify a parameter store environment variable, see env/parameter-store in the AWS CodeBuild User Guide.

PLAINTEXT: An environment variable in plain text format. This is the default value.

SECRETS_MANAGER: An environment variable stored in AWS Secrets Manager. To learn how to specify a secrets manager environment variable, see env/secrets-manager in the AWS CodeBuild User Guide.

Code Build Integration with cloud watch events - you can create a schedule in cloud watch event where you can set the target to code build Recommended Blog : https://aws.amazon.com/blogs/devops/validating-aws-codecommit-pull-requests-with-aws-codebuild-and-aws-lambda/

Code Deploy

EC2 instances can be grouped by a deployment group (dev/test/prod) Lots of flexibility to define any kind of deployments Code Deploy can be chained with Code Pipeline and use artifacts from there. Code Deploy can re-use any existing setup tools, work on application, auto scaling B/G only works for EC2 instances, not on premise Support for AWS Lambda deployments, EC2

AppSpec - BeforeInstall – Use to run tasks before the replacement task set is created. One target group is associated with the original task set. If an optional test listener is specified, it is associated with the original task set. A rollback is not possible at this point.

AfterInstall – Use to run tasks after the replacement task set is created and one of the target groups is associated with it. If an optional test listener is specified, it is associated with the original task set. The results of a hook function at this lifecycle event can trigger a rollback.

AfterAllowTestTraffic – Use to run tasks after the test listener serves traffic to the replacement task set. The results of a hook function at this point can trigger a rollback.

BeforeAllowTraffic – Use to run tasks after the second target group is associated with the replacement task set, but before traffic is shifted to the replacement task set. The results of a hook function at this lifecycle event can trigger a rollback.

AfterAllowTraffic – Use to run tasks after the second target group serves traffic to the replacement task set. The results of a hook function at this lifecycle event can trigger a rollback.

Code Deploy integration with Lambda

Working with Code Deploy Configurations

Deployment configuration for different EC2/on-premise compute platform

CodeDeployDefault.AllAtOnce: In-place Deployments: Attempts to deploy an application revisions to as many instances possible at a time; For Blue/green deployments, Deployments will replace the environment

CodeDeployDefault.HalfAtATime: Deploys upto half of the instances at a time (with fractions rounded down)

Deployment configuration on an AWS lambda compute platform

Systems Manager also helps you configure and maintain your managed instances. Supported machine types include Amazon Elastic Compute Cloud (Amazon EC2) instances, on-premises servers, and virtual machines (VMs), including VMs in other cloud environments. Supported operating system types include Windows Server, macOS, Raspberry Pi OS (formerly Raspbian), and multiple distributions of Linux.

In this short blog, we are going to cover registering an ubuntu on-prem instance to AWS Systems Manager

Installation Steps:

1) Go to AWS console and open SSM

2) Under "Node Management" click on "Hybrid Activations" & then "Create an activation"

3) Fill out the required details, you can give it a name and setup an expiration time

4) Once you generate the activation, you will get activation-code and activation-id. This will be used for our on-prem instance

5) Login to your onprem instance and execute the following commands after you replace activation-code and activation-id -

mkdir /tmp/ssm
curl https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm -o /tmp/ssm/amazon-ssm-agent.rpm
sudo yum install -y /tmp/ssm/amazon-ssm-agent.rpm
sudo systemctl stop amazon-ssm-agent
# edit the code, id and region in the command below
sudo amazon-ssm-agent -register -code "activation-code" -id "activation-id" -region "region"
sudo systemctl start amazon-ssm-agent

6) Go to SSM dashboard and check your registered instance, since this is on-prem it should start with "mi-"

With the boom of industry 4.0 and IoT, connectivity between edge IoT devices and Cloud is rapidly gaining importance. There are billions of devices out in the market and not all computation can be done at the edge. Talking specifically about edge IoT devices, MQTT is a widely used protocol for communication and there are cloud solutions like GreenGrass that would send logs to IoT Core. Sending Data to IoT Core it's just the start, the data can be leveraged in endless possible ways like sending it to Athena, New Relic and many more. We are going to cover a very simple use case here which is sending it to S3 bucket. Let's get started with understanding of IoT core and kinesis firehose.

IoT Core:

AWS IoT Core lets you connect IoT devices to the AWS cloud without the need to provision or manage servers. AWS IoT Core can support billions of devices and trillions of messages, and can process and route those messages to AWS endpoints and to other devices reliably and securely. With AWS IoT Core, your applications can keep track of and communicate with all your devices, all the time, even when they aren’t connected. %[https://aws.amazon.com/iot-core/][1]

Kinesis Firehose:

Amazon Kinesis Data Firehose is the easiest way to reliably load streaming data into data lakes, data stores, and analytics services. It can capture, transform, and deliver streaming data to Amazon S3, Amazon Redshift, Amazon OpenSearch Service, generic HTTP endpoints, and service providers like Datadog, New Relic, MongoDB, and Splunk. It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration. It can also batch, compress, transform, and encrypt your data streams before loading, minimizing the amount of storage used and increasing security.

Implementation

Step 1:

On your AWS console, navigate to AWS Core Services When you are in IoT core, on the left hand side expand on "Act" and click on Rules.

Step 2:

Provide a rule name and a query attribute in the statement . In the below screenshot, my rule name is Demo_Rule and the MQTT topic I have subscribed is iot/topic. If you want to subscribe to all topics, your query should be SELECT * FROM '#'

Step 3:

Click on Add action, and select Send a message to an Amazon Kinesis Firehose Stream

Step 4:

Kinesis Firehose delivery Stream. You can select an existing Kinesis Firehose delivery stream or you can create a new one, in this blog I am creating a new one.

Source: Direct PUT Destination: S3 Delivery Stream Name: iot_demo_test (please customize as required) Destination Settings: Provide details on the name of S3 bucket and prefix. (You can also adjust the buffer size as required) & Click on Create delivery stream.

Step 5:

Assign Role to your IoT Rule and complete creation of IoT Rule

Testing :

Under Test with demo data, choose Start sending demo data to generate sample stock ticker data. Follow the onscreen instructions to verify that data is being delivered to your S3 bucket. Note that it might take a few minutes for new objects to appear in your bucket, based on the buffering configuration of your bucket. When the test is complete, choose Stop sending demo data to stop incurring usage charges.