Published: 31. 8. 2021   Category: Programming

Manage AWS from the shell — awscli

Configure authentication

The best way is to keep your credentials/AWS keys is in ~/.aws/credentials, especially if you are managing more accounts and need to switch between profiles (production/development/staging/etc.). Ideally, you can have some federated account management with session credentials. I am receiving session via SAML2AWS after password and MFA authentication and the session timeout is 8 hours, so I need to enter my password usually just once a day. In case that your box is compromised, an attacker will hopefully do not compromise also your AWS account, because he will find only invalid tokens of the old sessions. Also, use very well defined IAM keys for automation tasks with just very limited permissions only for things needed to be done in such task.

If you need to use more accounts/region

For example, if you are dealing with 3 accounts and 5 regions. I have them listed in two environment variables, defined in my ~/.bashrc:

export AWS_ACCOUNTS=(production development staging)
export AWS_REGIONS=(us-east-1 us-east-2 eu-west-1 eu-west-2 eu-central-1)

When I need to obtain some information from all accounts I will use a for-cycle over all accounts and regions, this way:

for profile in ${AWS_ACCOUNTS[*]}; do
    for region in ${AWS_REGIONS[*]}; do
        aws --profile=$profile --region=$region ...
    done
done

Get region locally from EC2 metadata

curl -s http://169.254.169.254/latest/meta-data/placement/region

curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region

Check your identity

aws sts get-caller-identity --region=us-east-1

Get informations about EC2 instances

Find Instance-Id based on value in Tags

aws ec2 describe-instances --region=$region \
    --filters "Name=tag-value,Values=NameTagValue" \
    --query="Reservations[].Instances[].[InstanceId]" --out text

Find Tag and Private IP for given Instance-Id

aws ec2 describe-instances --instance-id=i-InstanceId
    --region=$region \
    --query='Reservations[].Instances[].[PrivateIpAddress,Tags[?Key==`Name`]|[0].Value]' \
    --out text

Get table of running EC2 instances with basic info


aws ec2 describe-instances --region=$region \
    --filters Name=instance-state-name,Values=running \
    --query 'Reservations[*].Instances[*].{
        VPC:VpcId,
        ImageId:ImageId,
        Instance:InstanceId,
        Type:InstanceType,
        AZ:Placement.AvailabilityZone,
        Subnet:SubnetId,
        Name:Tags[?Key==`Name`]|[0].Value}' \
    --output table

Terminate list of EC2 instances

Instance-Ids are given as a simple list behind the --instance-ids:

aws ec2 terminate-instances --region=$region \
   --instance-ids i-0b24eb27 i-0fa97b1b i-347ca930

Find if all hosts in private subnets are up (a.k.a test VPN)

aws ec2 describe-instances --region=us-east-1 \
    --query='Reservations[].Instances[].[PrivateIpAddress]' \
    --filters Name=instance-state-name,Values=running --out=text | \
    xargs nmap -sn -oG -
Note: nmap -sn does not scan ports, just discovers what is up.

List unassigned volumes

aws --region $region ec2 describe-volumes  \
   --filters Name=status,Values=available --query \
   'Volumes[*].{a1:VolumeId,b2:AvailabilityZone,c3:Tags[?Key==`Name`]|[0].Value}' \
   --output text

Terminate volumes

VOLUMES=(vol-123 vol-345)

for vol_id in ${VOLUMES[*]}
do
    aws ec2 delete-volume --volume-id $vol_id --region $region
done

List subnets of given VPC

aws --region=$region ec2 describe-subnets \
    --filters Name=vpc-id,Values=vpc-123456  \
    --query='Subnets[].[Tags[?Key==`Name`]|[0].Value,SubnetId,AvailabilityZone,CidrBlock]' \
    --out text

List RDS instances

aws rds describe-db-instances --region=$region \
    --output=text --query \
        "DBInstances[].[DBInstanceIdentifier,DBInstanceClass,
         Engine,EngineVersion,AllocatedStorage,AvailabilityZone]"

Get items from AWS Parameter Store

Get names and description, filter given value

aws ssm describe-parameters --region=$region \
    --parameter-filters="Key=Name,Option=Contains,Values=value" \
    --query='Parameters[].[Name,Description]' --out text

Get and decrypt stored parameter

aws ssm get-parameters --region=$region --names=parameter \
    --with-decryption --query='Parameters[].Value'

Get all parameters decrypted

for param in $(aws ssm describe-parameters --region=$region \
                   --query='Parameters[].Name' --out text )
do
    echo -ne "$param\t"
    aws ssm get-parameters --region=$REGION --names=$param \
        --with-decryption --query='Parameters[].Value'
        --out text
done

Get all records from Route53 for given hosted zone

aws route53 list-resource-record-sets \
    --hosted-zone-id Z.... \
    --query="ResourceRecordSets[*].Name"

Check status of Load Balancer's instances

for elb in $(aws elb describe-load-balancers --region=$region \
     --query='LoadBalancerDescriptions[].[LoadBalancerName]' --out text)
do
    echo $elb:
    aws elb describe-instance-health --region=$region --out text \
        --query='InstanceStates[].[InstanceId,State]' \
        --load-balancer-name $elb
done

Set encryption on all S3 buckets

for bucket in $(aws s3api list-buckets --query='Buckets[].Name' --out text)
do
    aws s3api put-bucket-encryption \
        --bucket $bucket \
        --server-side-encryption-configuration '{
                "Rules": [{"ApplyServerSideEncryptionByDefault": {
                        "SSEAlgorithm": "AES256"}}]}'

done

Check encryption on Amazon WorkSpaces

aws workspaces describe-workspaces \
  --region=$region \
  --query 'Workspaces[*].{
        User:UserName,
        "Root Encryption":RootVolumeEncryptionEnabled,
        "Disk Encryption":UserVolumeEncryptionEnabled}' \
  --output table

Check RDS encryption and multi-zone deployment

aws rds describe-db-instances --region=$region \
  --query='DBInstances[].{
              "RDS Id":DBInstanceIdentifier,
              "Storage Encrypted":StorageEncrypted,
              "KMS Key":KmsKeyId,
              Certificate:CACertificateIdentifier,
              "Multizone Deployment":MultiAZ}'

AWS Lambdas

List AWS Lambdas and filter prefix name

PREFIX=myfunc
aws lambda list-functions --region=us-east-1 \
    --query "Functions[?starts_with(FunctionName, `$PREFIX`) == `true`].FunctionName"
    --output text

Download all AWS Lambdas

REGION=us-east-1
LIST=( $(aws lambda list-functions --region=$REGION --query='Functions[].[FunctionName]' --out text) )


for fn in ${LIST[*]}
do
    aws lambda get-function --region=$REGION \
        --function-name $fn --query 'Code.Location' | \
        xargs wget -O ${fn}.zip
done