Published: 24. 10. 2018   Category: Programming

Run and check AWS OpsWorks recipe from a command line

To understand this article you need to be somehow familiar with AWS OpsWorks and bash. It may give you some ideas how to call a task(s) from command line and automate them.

Executing recipe from command line is not very easy, especially when you need to call it for the first time. In this case you need to gather a lot of information: StackId, LayerId, InstanceId, DeploymentId, etc.

When debugging recipe on specified instance, which belong to some Stack and Layer you will need to gather this informations first to execute a command. When a command is executed, you will get unique DeploymentId, then you can check its status (successful, running, pending, skipped or failed) and you will obtain also URL of the log.

Also you need to setup your AWS API credentials with aws configure or by env. variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION. And also have awscli installed: pip install awscli.

Because answer of the following commands are quite verbose JSON strings, I have used grep to see the important information. Even if awscli returns JSON strings nicely formatted, it is generally better to deal with a JSON with jq command.

AWS API philosophy also requires to deal with your infrastructure not by human readable names but with long hexadecimal identifiers. But to be able understand JSON a little bit, there are usually "Name" and "Id" somehow bonded together, so you will understand what is described here, but when you will call furthers commands you need to use "Id" as main identifier of stack, layer, instance or deployment.

  1. Get StackId:
    aws opsworks describe-stacks | grep -E 'StackId|Name'
  2. Get Stack's layers:
    aws opsworks describe-layers --stack-id STACK_ID  | grep -E "LayerID|Name"
  3. Get instance Id, notice that there are two Ids, one related to OpsWorks and second to EC2:
    aws opsworks describe-instances --layer-id  LAYER_ID | grep -E "Hostname|InstanceId"
  4. To get list of recipes use layer information, because Chef recipes names are in notation cookbook::recipe, use double colon to list just recipes.
    aws opsworks describe-layers --stack-id $ID | grep '::'
  5. Update recipes on given instance:
    aws opsworks create-deployment --stack-id STACK_ID \
      --instance-id INSTANCE_ID --command '{ "Name": "update_custom_cookbooks" }'
    The result of all commands is unique DeploymentId. In next steps, you will use this Id to get status and log of the command. Usually updating of cookbooks works without problem, the DeploymentId is more important for checking your custom recipes.
  6. Execute recipe:
    aws opsworks create-deployment --stack-id STACK_ID \
      --instance-id INSTANCE_ID --command \
      '{ "Name": "execute_recipes", "Args": {"recipes": ["cookbook::recipe"]} }'
  7. Check deployment status, you can see running during execution and when finished "successful" or "failed":
    aws opsworks describe-commands --deployment-id DEPLOYMENT_ID | grep -E "Status|LogUrl"
    When you need to see a log, you will get URL of html page with it.
  8. Display log by calling HTTP request, which is currently gzipped:
    curl https://opsworks... | gunzip 

So this was the process of executing your custom recipe on the give instance. When debugging some recipe you need to call it repetitively.

Execute recipe

And now put it together and execute an OpsWorks recipe. A recipe execution usually takes some time, so there is function wait_to_finish which checks its status periodically and waits


function get_val() {
    local id=$1
    sed -ne "/$id/{s/.*\".*\": \"\(.*\)\".*/\1/;p}"

function wait_to_finish(){
    local deployment_id=$1

    while :
        sleep 10 # duration of deployment is quite long
        RESULT=$(aws opsworks describe-commands --deployment-id $deployment_id)
        STATUS=$(get_val Status <<< "$RESULT")

        if [[ $STATUS == "pending" ]] || [[ $STATUS == "running" ]]
            echo "$RESULT"

# Update
DEPLOYMENT_ID=$(aws opsworks create-deployment --stack-id $STACK_ID \
    --instance-id $INSTANCE_ID \
    --command '{ "Name": "update_custom_cookbooks" }' | \
    get_val DeploymentId)

wait_to_finish $DEPLOYMENT_ID

# Execute
DEPLOYMENT_ID=$(aws opsworks create-deployment --stack-id $STACK_ID \
    --instance-id $INSTANCE_ID \
     --command "{
        \"Name\": \"execute_recipes\",
        \"Args\": {\"recipes\": [\"$RECIPE\"]}
    }" | get_val DeploymentId)

# Check result of command execution
RESULT=$(wait_to_finish $DEPLOYMENT_ID)
STATUS=$(get_val Status <<< "$RESULT")

if [[ $STATUS == "failed" ]] ; then
    # Display error log
    LOGURL=$(get_val LogUrl <<< "$RESULT")
    curl $LOGURL | gunzip

Upload your recipe to repository

I am using S3 as repository of Chef recipes, it contains several directories (development, staging, production) and the package configured in Stack options has always name opsworks-latest.tar.gz. I am using following script to upload recipes repository. It reads branch identifies as the first position parameter, I am usually using separate development and production branches.

DATE=$(date +%Y%m%d%H%M)
# Set name of actual git branch
BRANCH=${1:?"Branch is not set!"}

if [[ $1 == 'check' ]] || [[ $1 == 'help' ]] ; then
    echo Available branches:
    aws s3 ls $S3_BUCKET | sed -e 's/.*PRE\([^/]*\)\//* \1/g'

berks package

mv cookbooks-*.tar.gz $PACKAGE 


aws s3 cp $S3_BUCKET/$BRANCH/$PACKAGE $S3_BUCKET/$BRANCH/opsworks-latest.tar.gz


Test recipe locally on a AWS EC2 machine

Previous text allows to handle your OpsWorks task from any machine with installed AWS utils (awscli). But the development cycle may be quite long, when you change something, you need to package it, save to S3 repository and deploy. In many cases, you will fight with some errors, which can be troubleshooted locally and once solved, saved in your code managment repository.

Once cookbooks were updated, you can get list of recipes on the particular machine:

cd /var/chef/cookbooks
find . -name "*.rb" ! -path "./opsworks/*" -type f | \
    grep recipes | \
    awk -F/ '{print $2"::"substr($4,0,length($4)-3)}'

Run recipe layer::recipe and use latest attribs.json (file storing Stack's Custom JSON):

cd /var/chef/
chef-client -j attribs.json -z -o layer::recipe
It is possible to modify recipe in /var/chef/cookbooks and immediatelly repeat step chef-client command to see if a recipe works.