AWS Learning Notes

Posted on | Edited on

AWS Regions

AWS has regions globally

  • each region has availability zones
  • each availability zone is a physical center in the region
  • AWS Consoles are region scoped

1. Identity and Access Management

  • users
  • groups contain users
  • roles should be given to machines\
  • policies define what each of above can and cannot do

IAM has global view
Give user minimal permissions to perform their job
One IAM user person, do not share
One IAM role per application

2. EC2

ssh -i key ec2-user@ipaddress

2.1 Security Group

Security Group controls how traffic is allowed in and out of EC2 machine

allow, inbound and outbound

a security group can be attached to multiple instances

lock down to a region/vpc combination

good to maintain one separate security group for SSH access

timeout —- security group issue

connection refused —- application issue

2.2 Bootstrap EC2 instance using EC2 User Data

  • only run once at the instance first start
  • EC2 user data is used to automate boot tasks such as:
    • installing updates;
    • installing software
    • downloading common files from the internet
  • The EC2 User data run with the root account

  • 3. Load Balancer

    There are 3 types of load balancers:
  • Classic Load Balancer v1
  • Application Load Balancer v2

  • Network Load Balancer v2

    With Application Load Balancers and Network Load Balancers, you register targets in target groups and route traffic to the target groups. With Classic Load Balancers, you register instances with the load balancer.

Elastic Load Balancing uses a service-linked role for the permissions that it requires to call other AWS services on your behalf.

It is recommended to use v2 version load balancer as it provides more features

You can set up internal and external ELBs

Elasticcache supports Redis and Memcached. redis is recommended.

3.1 Health check

Application Load Balancer

Load balancing based on the route in URL

Load balancing based on hostname in DNS

3.2 Target group

When creating a load balancer

  1. create an ALB
  2. setup security group
  3. setup target group
  4. add an instance to the target group

4. ASG Auto Scaling Group

5. AWS S3

  • AWS S3 is a regional service, but need to have a globally unique name, no uppercase, no underscore, not ip, must start with a lowercase letter or a number

    • In s3 there is no directory, but the s3 UI will make you think there are different directories levels.

  • Each object in a bucket has a key, the key is the full path of the object.

  • Maximum size is 5TB, if you wanna upload file greater than 5GB, need to use multiple uploads.

5.1 S3 versioning

Versioning is a bucket level setting

Each version will have a version ID

Any file not versioned will have the version ‘null’

If a versioned file is deleted, it can be recovered from version control.

5.2 S3 Encryption

  • There are 4 methods of encrypting objects in S3:
  • SSE-S3 uses key managed by S3, AES-256. Put x-amz-server-side-encrypt:‘sse-s3’ to enable
  • SSE-KWS uses key managed by KWS. Put x-amz-server-side-encrypt:‘aws-kws’ to enable
  • SSE-C uses client-side provided key, every time upload data to S3, need to provide the key in the header. This time HTTPS must be used.
  • Client Side Encryption. Encryption happened in the client, before sending data to S3.

Set default encryption to enable auto encryption for every file.

5.3 S3 Websites

S3 can host a static website.

Need to set up permission and make sure the bucket is public by using bucket policy

5.4 S3 CORS

Cross-Origin Resource Sharing allows you to limit the number of websites that can request your files in S3.

5.5 Consistency Model

  • Read after writes consistency for PUTS of new objects.
    • As soon as the object is written, we can retrieve it.
    • This true, except if we did a GET before to see if the object existed. GET->PUT->GET eventually consistent
  • Eventually Consistency for DELETES and PUTS of existing objects.

5.6 Performance

It is recommended to have 4 random characters in front of your key name to optimize performance. It will force S3 to involve multiple partitions.

Current State: You don’t need to prefix. coz 2018-07-17 support prefix.

Multiple upload for file > 5 GB

  • parallelized PUTS
  • maximum network bandwidth
  • decrease time to retry in case a part fails

if you do a lot of read around S3, use CloudFront to cache S3 objects around the world

S3 Transfer Acceleration - just need to change the endpoint you write to, not the code.

If using SSE-KMS, you may be limited to your AWS limits for KMS usage (100s - 1000s downloads/uploads per second). So you may see a performance decrease.

5. AWS CLI, IAM Roles & Policies

Never put your personal cred on EC2, user IAM Roles instead.

  • IAM Roles can be attached to EC2 instance.
  • IAM Roles can come with policy what defines the instance should be able to do

5.1 Instance Metadata

allows EC2 instance know themselves without an IAM Role

The URL is http://169.254.169.254/latest/meta-data

You can retrieve the IAM Role name but not the IAM Policy

6. Elastic Beanstalk

Elastic Beanstalk is a developer-centric view of deploying an application on AWS

Managed Service

Just the application code is the responsibility of the developer

Three architecture models

  • Single Instance deployment: good for dev
  • LB + ASG: great for production or pre
  • ASG only: greate for non-web apps in production

Three components

  • Application
  • Application version
  • Environment name

Deploy application version to environments and can promote application versions to the next environment

6.1 Beanstalk Deployment Modes

Single Instance: great for dev

High availability with LB: greate for prod

  • All at once (deploy all in one go)

  • Rolling: update a few, then move the next one until the first one is healthy

    • the application is running below capacity
    • can set the bucket size
    • no additional cost
    • long deployment
    • the application is running both versions simultaneously

  • Rolling with addition batches: like rolling, use the new instance to move the batch

    • The application is running at capacity
    • Can set the bucket size
    • The application is running both versions simultaneously
    • Small additional cost
    • Additional batch is removed at the end of the deployment
    • longer deployment
    • Good for prod

  • Immutable: spin up new instances in a new ASG, deploy and then swaps when all the instance is healthy

    create only one instance first, once the first instance is healthy, then launch all the remaining ones

    • zero downtime
    • the new code is deployed to new instances on a temporary ASG
    • high cost, double capacity
    • longest deployment
    • quick rollback in case of failures
    • great for prod

6.2 Beanstalk Extension

  • zip file containing our code must be deployed to Beanstalk
  • All the parameters set in the UI can be configured with code using files
  • Requirements:
    • in the .ebextensions/ directory in the root of the source code
    • yaml or JSON format
    • .config extensions
    • able to modify some default setting using: option_settings
    • ability to add resources such as RDS, ElastCache, DynamoDB, etc..

6.3 ElasticBeanStalk Deploy Mechanism

  • Describe dependencies
  • Package code as a zip
  • The zip file is uploaded to each EC2 machine
  • Each EC2 machine resolves dependencies
  • Optimization in case of long deployment: Package dependencies with source code to improve deployment performance and speed.

All the configuration file will be deleted when the environment is gone

7. CICD

7.1 Continuous Integration

  • Developer push code to repo
  • Test/Build server checks the code as soon as it’s pushed
  • The developer gets feedback about the tests and checks that have passed / failed

CodeCommit Security

  • Interaction are done using Git
  • Authentication in Git:
    • SSH key: AWS Users can configure SSH keys in their IAM Console
    • HTTPS: Done through the AWS CLI Authentication helper or Generatin HTTPS credentials
    • MFA
  • Authorization in Git:
    • IAM Policies
  • Encryption:
    • Repo are automatically encrypted at rest using KMS
    • Encrypted in transit

CodeCommit Notifications

  • Trigger notifications in CodeCommit using AWS SNS or AWS Lambda or AWS CloudWatch Event Rules
  • Use case for SNS or Lambad:
    • deletion
    • trigger for pushes
    • notify external build system
    • trigger aws lambda function to perform codebase analysis
  • Use case for CloudWatch Event Rules
    • Trigger for pull request
    • Commit comments events
    • CloudWatch Event Rules goes into an SNS topic

7.2 Continuous Delivery

CodePipeline

  • continuous delivery
  • visual workflow
  • source: Github / CodeCommit
  • Build: CodeBuild
  • Deploy: CodeDeploy / BeanStalk / Cloudfount …
  • Made of stages:
    • Each stage can have sequentail actions or parallel actions
    • Stages Example: build / test / deploy / loadtest / etc
    • Manual approval can be defined at any stage

Each pipeline stage can generate artifacts and the next stage will get these artiacts

CodePipline Troubleshooting

  • CodePipline stage changes happen in AWS CloudWatch Event, which can in return create SNS notifications.
  • Ensure that the software can be released reliably whenever needed
  • Ensure deployments happen often and are quick
  • Shift away from one release every 3 months to 5 releases a day

Technology Stack for CICD:

code : aws codecommit
build/test : aws codebuild
deploy/ provision: elastic beanstalk

code build

  • fully managed build service
  • alternative to other build tools such as Jenkins
  • Continuous scaling ( no server to manage or provision - no build queue)
  • Pay for usage ( the time it takes to complete the builds)
  • leverages Docker under the hood for reproducible builds
  • possibility to extend capabilites leveraging our own based Docker images
  • Secure: Integration with KMS for encryption of artifacts, IAM for build permissions, and VPC for network security, CloudTrail for API calls logging

Build instruction can be defined in code (buildspec.yml file) must be at root of you code

Output logs to Amazon S3 & AWS Cloudwatch

Metrics to monitor CodeBuild statstics

User CloudWatch Alarms to detect failed build and trigger notification

Ability to reproduce CodeBuild locally to debug

CodeDeploy

  • we want to deploy our application automatically to many EC2 instance
  • there instances are not managed by Elastic Beanstalk
  • There are serveral ways to handle deployments using open source tools
  • we can use the managed Service AWS CodeDeploy

Steps to make it work:

  • Each EC2 Machine must be running the CodeDeploy Agent
  • The agent is continuously polling AWS CodeDeploy for work to do
  • CodeDeploy sends appspec.yml file
  • Application is pulled from Github or S3
  • EC2 will run the deployment instructions
  • Codedeploy Agent will report of success / failure of deployment on the instance

Components:

  • application: unique name
  • compute platform: EC2/On-Premise or Lambda
  • deployment configuration: Deploy rules for success/failure
    • EC2/On-premise: can specify minimum number of healthy instances for the deployment
    • AWS Lambda: specify how traffic is routed to your updated Lambda function version
  • deployment group: group of tagged instances
  • deployment type: in-place deployment or blue/green deployment
  • IAM instance profile: need to give EC2 the permissions to pull from S3 /Github
  • Application Revision: application code + appspec.yml file
  • Service role: Role for CodeDeploy to perform what it needs
  • Target Revision: Target deployment application version

AWS CodeDeploy AppSpec

  • file secion: how to source and copy from S3
  • Hooks: set of instruction to do to deploy the new version. The order is:
    • ApplicationStop
    • DownloadBundle
    • BeforeInstall
    • AfterInstall
    • ApplicationStart
    • ValidateService: really important

Cloudformation

CloudWatch

  • cloudwatch provides metrics for every services in AWS
  • Metric is a variable to monitor
  • Metrics belong to namespaces
  • Dimension is an attribute of a metric
  • Up to 10 dimensions per metric
  • Metrics have timestamps
  • Can create CloudWatch dashboards of metrics

AWS CloudWatch EC2 Detailed monitoring

  • EC2 instance metrics have metrics ‘every 5 minutes’
  • With detailed monitoring, you get data every 1 minute
  • User detailed monitoring if you want to more prompt scale your ASG

AWS CloudWatch Custome Metrics

  • Possibility to define and send your own custom metrics to CloudWatch
  • Ability to use dimensions to segment metrics
    • instance.id
    • Environment.name
  • Metric resolution:
    • standard: 1mintue
    • High Resolution: up to 1 second – Higher cost
  • Use API call PutMetricData
  • Use exponential back off in case of throttle errors

CloudWatch Alarm

  • are used to trigger notification for any metrics
  • can go to Auto Scaling, EC2 and SNS notification
  • various options
  • states:
    • ok
    • insufficient_data
    • alram
  • Period
    • lenght of tiem in seconds
    • high resolution custome metrics can only choose 10 sec or 30 sec

X-Ray