Skip to content

Instantly share code, notes, and snippets.

@abantej

abantej/aws-ec2-notes.md

Last active March 9, 2024 17:21
Show Gist options
  • Save abantej/01e54d0f93c8f544d0b9d9e67d396d49 to your computer and use it in GitHub Desktop.
Save abantej/01e54d0f93c8f544d0b9d9e67d396d49 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
aws-ec2-notes.md

Elastic Compute Cloud (EC2)

  • a web service that provide the resizable compute capacity in the cloud.
  • reduces the time required to obtain and boot new server instances to minutes, allowing you to scale capacity, both up and down, as your computing requirements change
  • allows you to pay only for capacity that you actually use
  • it provides developers the tools to build failure resilient applications and isolate themselves from common failure scenarios

EC2 Options

  • On Demand
    • allow you to pay a fixed rate by the hour with no commitment
  • Reserved
    • provde you a capacity reservation, and offer a significant discount on the hourly charge for an instance
    • 1 year or 3 year terms
  • Spot
    • enable you to bid whatever price you want for instance capacity, providing for even greater savings if your applications have flexible start and end times
    • if you terminate the instance, you pay for the hour
    • if AWS terminate the instance, you get the hour it was terminated in for free
  • Dedicated Hosts
    • physical EC2 server dedicated for your use.
    • it can help you reduce costs by allowing you to use your existing server-bound software licenses

EC2 Intance Types

  1. D2 - Dense Storage (Fileservers/Data Warehousing/Hadoop)
  2. R4 - Memory Optimized (Memory Intensive Apps/DBs)
  3. M4 - General Purpose (Application Servers)
  4. C4 - Compute Optimized (CPU Intensive Apps/DBs)
  5. G2 - Graphics Intensive (Video Encoding/3D Appliacation Streaming)
  6. I2 - High Speed Storage (NoSQL DBs, Data Warehousing etc)
  7. F1 - Field Programmable Gate Array (Hardware Acceleration for your code)
  8. T2 - Lowest Cost General Purpose (Web Servers/Small DBs)
  9. P2 - Graphics/General Purpose GPU (Machine Learning, Bit Coin Mining)
  10. X1 - Memory Optimized (SAP HANA/Apache Spark etc.)

DR Mc GIFT PX

  • D for Density
  • R for RAM
  • M - main choice for general purpose apps
  • C for Compute
  • G for Graphics
  • I for IOPS
  • F for FPGA
  • T for T2 Micro, cheap general purpose
  • P Graphics (think Pics)
  • X for eXtreme memory

Amazon EBS

  • allows you to create storage volumes and attach them to Amazon EC2 instances
  • once attached, you can create a file system on top of these volumes, run a database, or use them in any other way you would use a block device
  • EBS volumes are placed in a specific Availability Zone, where they are automatically replicated to protect you from the failure of a single component

EBS Volume Types

  • General Purpose SSD (GP2)
    • General purpose, balances both price and performance
    • Ratio of 3 IOPS per GB with up to 10000 IOPS and the ability to burst up to 3000 IOPS for extended periods of time for volumes at 3334 GiB and above
  • Provisioned IOPS SSD (IO1)
    • Designed for I/O intensive applications such as large relational or NoSQL databases
    • Use if you need more than 10,000 IOPS
    • Can provision up to 20,000 IOPS per volume
  • Throughput Optimized HDD (ST1)
    • Big Data
    • Data warehouses
    • Log processing
    • Cannot be a boot volume
  • Cold HDD (SC1)
    • lowest cost storage for infrequently accessed workloads
    • file server
    • cannot be a boot volume
  • Magnetic (Standard)
    • lowest cost per gigabyte of all EBS volume types that is bootable
    • Magnetic volumes are ideal for workloads where data is accessed infrequently, and applications where the lowest storage cost is important

EBS Volume Types Summary

  1. SSD, General Purpose - GP2 - (Up to 10,000 IOPS)
  2. SSD, Provisioned IOPS - IO1 - (More than 10,000 IOPS)
  3. HDD, Throughput Optimized - ST1 - frequently accessed workloads
  4. HDD, Cold - SC1 - less frequently accessed data
  5. HDD, Magnetic - Standard - cheap infrequently accessed storage

Note: You cannot mount 1 EBS volume to multiple EC2 instances, instead us EFS

EC2 - Lab

  • Termination Protection is turned off by default, you must turn it on
  • On an EBS-backed instance, the default action is for the root EBS volume to be deleted when the instance is terminated
  • EBS Root Volumes of your default AMI's cannot be encryted. You can also use a third party tool (such as bit locker) to encrypt the root volume, or this can be done when creating AMI's in the AWS console or using the API
  • Additional volumes can be encrypted

EC2 Linux Commands

  • sudo su (root access)
  • yum update -y (install updates)
  • yum install httpd (install apache web server)
  • service start httpd (start apache web server)
  • chkconfig httpd on

EC2 Security Groups

  • All inbound traffic is blocked by default
  • All outbound traffic is allowed
  • Changes to security groups take effect immediately
  • Security groups are stateful, anything you allow in (inbound) will go out (outbound) as well
  • You can have any number of EC2 instances within a Security Group.
  • You can have multiple security groups attached to EC2 instances
  • You cannot block specifi IP addresses using Security Groups, instead use Network Access Control Lists.
  • You can specify allow rules, but not deny rules

EBS Volume Types

  • always have the EC2 instance and EBS volume in the same availability zone
  • no downtown when changing storage types
  • standard type volume cannot be changed to a different volume type
  • to create another EC2 instance/volume in another availability zone, must create snapshot
  • only root volume is terminated when EC2 instance is terminated

EBS Volumes and Snapshots

  • virtual hard disks
  • Snapshots exist on S3
  • Snapshots are point in time copies of volumes
  • Snapshots are incremental - only blocks that have changed since your last snapshot are moved to S3

Snapshots of Root Device Volumes

  • To create a snapshot for Amazon EBS volumes that serve as root devices, you should stop the instance before taking the snapshot
  • However you can take a snapshot while the instance is running
  • You can create AMI's from both volumes and snapshots
  • You can change EBS volume sizes on the fly, including changing the size and storage type
  • Volumes will always be in the same availability zone as the EC2 instance
  • To move an EC3 volume from one AZ/Region to another, take a snap or an image of it, then copy it to the new AZ/Region
  • snapshots of encrypted volumes are automatically encrypted
  • volumes restored from encrypted snapshots are encrypted automatically
  • you can share snapshots, but only if they are unencrypted
    • these snapshots can be shared with other AWS accounts or made public

RAID

  • RAID = Redundant Array of Independent Disks
    • RAID 0 - Striped, No Redundancy, Good performance
    • RAID 1 - Mirrored, Redundancy
    • RAID 5 - Good for reads, bad for writes, AWS does not recommend putting RAID 5's on EBS
    • RAID 10 - Striped and Mirrored, Good Redundancy, Good Performance

RAID Snapshot Problem

  • take a snapshot, the snapshot exludes data held in the cache by applications and the OS
  • this tends not to matter on a single volume
  • however using multiple volumes in a RAID array, this can be a problem due to interdependencies of the array

RAID Snapshot Solution

  • take an application consistency snapshot

How to take an application consistency snapshot?

  • stop the application from writing to disk
  • flush all caches to disk
    • freeze the file system
    • unmount the RAID array
    • shutting down the associated EC2 instance

AMI Selection basis:

  • Region
  • Operating System
  • Architecture (32 or 64)
  • Launch permissions
  • Storage for the root device (root device volume)
    • Instance store (EPHEMERAL STORAGE)
      • instance cannot be stopped
    • EBS backed volumes

EBS vs Instance Store

  • All AMI's are categorized as either backed by Amazon EBS or backed by instance store.
  • For EBS volumes: The root device for an instance launched from the AMI is an Amazon EBS volume created from an Amazon EBS snapshot
  • For Instance Store Volumes: The root device for an instance launched from the AMI is an instance store volume created from a template stored in Amazon S3
  • Instance Store Volumes are sometimes called Ephemeral Storage.
  • Instance Store Volumes cannot be stopped. If the underlying host fails, you will lose your data.
  • EBS backed instances can be stopped. You will not lose the data on this instance if it is stopped.
  • You can reboot both, you will not lose your data.
  • By default, both ROOT volumes will be deleted on termination, however with EBS volumes, you can tell AWS to keep the root device volume.

Elastice Load Balancer

  • instances monitored by ELB are reported as InService or OutofService
  • health checks check the instance health by talking to it
  • have their own dns name. you are never given an ip address

Cloud Watch

  • standard monitoring - 5 minutes
  • detailed monitoring - 1 minute
  • Dashboards - to see what is happening with you AWS environment
  • Alarms - allows you to set alarms that notify you when particular threasholds are hit
  • Events - helps you to respond to state changes in your aws resources
  • Logs - helps you aggregate, monitor, and store logs by installing agent

Exam tips

  • CloudWatch is for monitoring
  • CloudTrail is for auditing

AWS CLI and EC2

  • aws s3 ls
    • list all s3 from all regions
  • aws configure
    • need to key in access id, access password, and region which is provided while creating iAM users
  • aws s3 help
    • list all aws commands
  • aws configuration files
    • /root/.aws/config
    • /root/.aws/credentials
  • aws ec2 describe-instances
  • aws ec2 terminate-instances --instance-ids

Identity Access Mangement Roles

  • storing access ids in ec2 instances is not secure
  • all roles are global
  • create roles and associate them to your EC2 instance instead of hard coding the access id/password in the EC2 instance using 'aws configure'

S3 CLI and Regions

  • aws s3 cp --recursive s3://acloudguru-nv-01 /home/ec2-user
  • if you get an error, pass in the region like the following:
    • aws s3 cp --recursive s3://acloudguru-nv-01 /home/ec2-user --region eu-west-2

Sample bash script for initializing EC2 instance

#!/bin/bash
yum update -y
yum install httpd -y
service httpd start
chkconfig httpd on
aws s3 cp s3://abantej-s3-nv/index.html /var/www/html

EC2 Instance Metadata

Launch Configuration and Autoscaling Groups

  • before you can create an autoscaling group, you need to create a launch configuration

Placement Group

  • a logical grouping of instances within a single availability zone
  • enables applications to participate in a low-latency, 10 gbps network
  • recommend for applications that benefit from low network latency, high network throughput, or both
  • example usage: hadoop cluster, cassandra database

EC2 Placement Group

  • a placement group cannot span multiple availability zones
  • placement group name must be unique within your AWS account
  • only certain types of instances can be launched in a placement group (Compute optimized, GPU, Memory Optimized, Storage Optimized)
  • AWS recommend homogenous instances within placement groups. (instances of the same size and family)
  • you can't merge placement groups
  • you can't move an existing instance into a placement group. you can create an AMI from your existing instance, and then launch a new instance from the AMI into a placement group

Amazon Elastic File System (EFS)

  • a file storage service for EC2
  • easy to use and provides a simple interface that allows you to create and configure file systems quickly and easily
  • elastic storage capacity, growing and shrinking automatically as you add and remove files
  • allows you to mount a single EFS to multiple EC2 instance
  • supports the Network File System version 4 (NFSv4) protocol
  • you only pay for the storage you use (no pre-provisioning required)
  • 30 cents per gig
  • can scale up to petabytes
  • can support thousands of concurrent NFS connections
  • data is stored across multiple availability zones within a region
  • block based storage
  • read after write consistency
  • mount by: sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 fs-80f75fc8.efs.us-east-1.amazonaws.com:/ efs

Serverless Webpage with API Gateway and Lambda

Polly - A Serverless Approach

  • text to speech recognition service
  • the same technology that powers alexa

CORS

  • Cross-Origin Resource Sharing

Exam tips on EC2

  • Know the differencees between:
    1. On Demand
    2. Spot
    3. Reserved
    4. Dedicated Hosts
  • Remember with spot instances:
    1. if you terminate the instance, you pay for the hour
    2. if AWS terminates the spot instance, you get the hour it was terminated in for free
  • Remember EC2 Instance types
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment