Tools to Conquer AWS | Narrative Dev

AWS Command-Line Tools

First create create AWS credentials via the console, or speak to one of the other devs to have some created for you.

Install awscli

brew install awscli aws-shell

Configure awscli using the generated Access Key

$ aws configure
> AWS Access Key ID [None]: <SECRET>
> AWS Secret Access Key [None]: <SECRET>
> Default region name [None]: us-east-1
> Default output format [None]: json

Make the AWS credentials accessible to Hadoop/etc

Some Hadoop/S3 libraries look for the credentials in the AWS_* environment variables instead of ~/.aws/credentials or ~/.aws_credentials, so you may want to define AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Note: these environment variables do not play well with assume-role, so they should not be automatically sourced. Source them at your convenience.

ln -s ~/.aws/credentials ~/.aws_credentials
cat  > ~/.aws/env_config << 'EOF'
export AWS_ACCESS_KEY_ID=<TO_REPLACE>
export AWS_SECRET_ACCESS_KEY=<TO_REPLACE>
EOF

Test that the config is OK

aws ec2 describe-instance
aws s3 ls narrative-artifact-releases

Configure FoxyProxy to access EMR UIs

  • Browse to the EMR console
  • Click on any cluster
  • Click on Enable Web Connections and follow the instructions to configure FoxyProxy

Install Packer

brew install packer

assume-role

assume-role will request and set temporary credentials in your shell environment variables for a given (AWS IAM) role.

We use it to temporarily acquire a role while performing certain tasks that require additional permissions.

brew install remind101/formulae/assume-role