On 2020-08-02, the Cornell Shibboleth IdP was updated to v4.01. That update broke the awscli-login package for use at Cornell. See Help! I got a 500 Server Error below.
Thank you to user-af308 and Hong Ye for troubleshooting and identifying the problem!
Use Case
This document shows how to setup and use the awscli-login tool to retrieve temporary AWS access keys using your Cornell netid credentials and Duo (i.e., Cornell Two-Step Login). Using temporary access keys associated with an AWS role to authenticate to the AWS Command Line Interface (CLI) is much safer than using fixed AWS access keys tied to an IAM user. Now that this option is available to Cornell AWS users, we recommend that fixed access keys no longer be used for humans using the AWS CLI.
Prerequisites
- If you don't have the AWS CLI installed yet, that's great. Start by checking that you have Python 3.5+, then worry about the AWS CLI.
- AWS CLI installation help: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html
- Python installation information: https://wiki.python.org/moin/BeginnersGuide/Download (for non-beginners: follow links on the page as appropriate)
- If you already have the AWS CLI installed, you'll need to make sure that it is using Python 3.5+. If it isn't, then the awscli-login plugin won't work properly and installing it may break AWS CLI installs that are NOT using Python 3.5+.
- Docker with the aws login tool with other helpful cloud utilities are available in this repo https://github.com/CU-CommunityApps/ct-cloud-utils-dockerized
Linux/Mac OS
$ python --version Python 2.7.14 $ python3 --version -bash: python3: command not found $ aws --version aws-cli/1.15.83 Python/2.7.14 Linux/4.14.77-70.59.amzn1.x86_64 botocore/1.10.82
In the above example, Python 2.x is installed and the AWS CLI is installed, but using Python 2.x. That's exactly what we don't want. It is beyond the current scope of this article to describe how to install Python 3.5+, and the AWS CLI, ensuring that the CLI is using your Python 3.5+ installation. Please note that virtualenv may make it easier to get the AWS CLI installed and using Python 3.5+.
Install and Configure awscli-login
Don't bother trying to install or use the awscli-login tool until your installed version of the AWS CLI reports that it is using Python 3.5+. You will also need to ensure that pip is using Python 3.5+. That may mean you will need to use pip3 in the commands below, instead of plain pip.
$ pip install --upgrade https://github.com/CU-CommunityApps/awscli-login/tarball/master#egg=awscli-login ... $ aws configure set plugins.login awscli_login ... $ aws login configure ECP Endpoint URL [None]: https://shibidp.cit.cornell.edu/idp/profile/SAML2/SOAP/ECP Username [None]: <YOUR NETID> Enable Keyring [False]: Duo Factor [None]: Role ARN [None]: $
You may ask...
Why is the Cornell fork version 0.1.0a5 while the "official" version at PyPi is version 0.1.0a6?
We're not sure. The master branch in the original git repo is labeled 0.1.0a5, so the master branch of our fork is too. We are looking into this, but our fork of version 0.1.0a5 seems to work.
Using awscli-login
Login using defaults setup above:
$ aws login
Password: ********
Factor: push
# Provided second factor out of band
Please choose the role you would like to assume:
Account: 000011112222
[ 0 ]: shib-admin
Account: 777788889999
[ 1 ]: shib-admin
[ 2 ]: shib-cs
[ 3 ]: shib-dba
Selection: 1
$ aws sts get-caller-identity
{
"UserId": "AROAICCPMY7VALLFYHWPA:pea1@cornell.edu",
"Account": "777788889999",
"Arn": "arn:aws:sts::777788889999:assumed-role/shib-admin/pea1@cornell.edu"
}
The options for "Factor" are "push", "sms", "phone", "auto"
Advanced Use
$ aws --profile foo login configure
ECP Endpoint URL [None]: https://shibidp.cit.cornell.edu/idp/profile/SAML2/SOAP/ECP
Username [None]: pea1
Enable Keyring [False]:
Duo Factor [None]: auto
Role ARN [None]: arn:aws:iam::111111111111:role/shib-admin
$ aws --profile bar login configure
ECP Endpoint URL [None]: https://shibidp.cit.cornell.edu/idp/profile/SAML2/SOAP/ECP
Username [None]: pea1
Enable Keyring [False]:
Duo Factor [None]: auto
Role ARN [None]: arn:aws:iam::222222222222:role/shib-admin
$ aws --profile foo login
Password: **************
# Provided second factor out of band
$ aws --profile foo sts get-caller-identity
{
"Arn": "arn:aws:sts::111111111111:assumed-role/shib-admin/pea1@cornell.edu",
"Account": "111111111111",
"UserId": "XXXXICCPMY7VALLFXXXX:pea1@cornell.edu"
}
$ aws --profile bar login
Password: **************
# Provided second factor out of band
$ aws --profile bar sts get-caller-identity
{
"Arn": "arn:aws:sts::222222222222:assumed-role/shib-admin/pea1@cornell.edu",
"Account": "222222222222",
"UserId": "YYYYICCPMY7VALLFYYYY:pea1@cornell.edu"
}
Troubleshooting
Help! I got a 401 Client Error
$ aws login Password: ************** 401 Client Error: Unauthorized for url: https://shibidp.cit.cornell.edu/idp/profile/SAML2/SOAP/ECP
This situation occurs when you provide an incorrect password.
If you have setup your awscli-login config to save your password (i.e., enable_keyring = true) then the plugin will happily save an incorrect password in the keyring. In this situation you will never be prompted for a password and you will immediately get a 401 error. To work through this edit your ~/.aws-login/config file to set enable_keyring = false, to be prompted for a password again.
Help! I got a 504 Server Error
$ aws login Password: ************** 504 Server Error: GATEWAY_TIMEOUT for url: https://shibidp.cit.cornell.edu/idp/profile/SAML2/SOAP/ECP
This situation occurs when you fail to provide your Duo second factor in time.
Help! I got a 500 Server Error
$ aws login
Password:
Traceback (most recent call last):
File "/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/awscli_login/__main__.py", line 175, in main
profile.cookies,
File "/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/awscli_login/saml.py", line 149, in refresh
soap = saml_login(url, jar)
File "/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/awscli_login/saml.py", line 92, in saml_login
r.raise_for_status()
File "/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://shibidp.cit.cornell.edu/idp/profile/SAML2/SOAP/ECP
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/awscli_login/__main__.py", line 123, in wrapper
f(profile, session)
File "/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/awscli_login/__main__.py", line 180, in main
profile.cookies, *creds)
File "/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/awscli_login/saml.py", line 118, in authenticate
soap = saml_login(url, jar, username, password, headers)
File "/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/awscli_login/saml.py", line 92, in saml_login
r.raise_for_status()
File "/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/requests/models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: 500 for url: https://shibidp.cit.cornell.edu/idp/profile/SAML2/SOAP/ECP
500 Server Error: 500 for url: https://shibidp.cit.cornell.edu/idp/profile/SAML2/SOAP/ECP
This situation occurs when you use the https://github.com/techservicesillinois/awscli-login (original) version of awscli-login instead of the Cornell fork of that tool at https://github.com/CU-CommunityApps/awscli-login.
To fix this, uninstall your current awscli-login version and use the Cornell fork:
$ pip uninstall awscli-login
Found existing installation: awscli-login 0.1.0a6
Uninstalling awscli-login-0.1.0a6:
Would remove:
/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/awscli_login-0.1.0a6.dist-info/*
/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/awscli_login/*
/home/ec2-user/environment/python37venv/lib/python3.7/site-packages/tests/*
Proceed (y/n)? y
Successfully uninstalled awscli-login-0.1.0a6
$ pip install --upgrade https://github.com/CU-CommunityApps/awscli-login/tarball/master#egg=awscli-login
Collecting awscli-login
Using cached https://github.com/CU-CommunityApps/awscli-login/tarball/master
Requirement already satisfied, skipping upgrade: awscli in ./testvenv/lib/python3.7/site-packages (from awscli-login) (1.18.110)
Requirement already satisfied, skipping upgrade: boto3 in ./testvenv/lib/python3.7/site-packages (from awscli-login) (1.14.33)
Requirement already satisfied, skipping upgrade: botocore in ./testvenv/lib/python3.7/site-packages (from awscli-login) (1.17.33)
Requirement already satisfied, skipping upgrade: daemoniker in ./testvenv/lib/python3.7/site-packages (from awscli-login) (0.2.3)
Requirement already satisfied, skipping upgrade: keyring in ./testvenv/lib/python3.7/site-packages (from awscli-login) (21.3.0)
Requirement already satisfied, skipping upgrade: lxml in ./testvenv/lib/python3.7/site-packages (from awscli-login) (4.5.2)
Requirement already satisfied, skipping upgrade: psutil in ./testvenv/lib/python3.7/site-packages (from awscli-login) (5.7.2)
Requirement already satisfied, skipping upgrade: requests in ./testvenv/lib/python3.7/site-packages (from awscli-login) (2.24.0)
Requirement already satisfied, skipping upgrade: docutils<0.16,>=0.10 in ./testvenv/lib/python3.7/site-packages (from awscli->awscli-login) (0.15.2)
Requirement already satisfied, skipping upgrade: colorama<0.4.4,>=0.2.5; python_version != "3.4" in ./testvenv/lib/python3.7/site-packages (from awscli->awscli-login) (0.4.3)
Requirement already satisfied, skipping upgrade: rsa<=4.5.0,>=3.1.2; python_version != "3.4" in ./testvenv/lib/python3.7/site-packages (from awscli->awscli-login) (4.5)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in ./testvenv/lib/python3.7/site-packages (from awscli->awscli-login) (0.3.3)
Requirement already satisfied, skipping upgrade: PyYAML<5.4,>=3.10; python_version != "3.4" in ./testvenv/lib/python3.7/site-packages (from awscli->awscli-login) (5.3.1)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in ./testvenv/lib/python3.7/site-packages (from boto3->awscli-login) (0.10.0)
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.1 in ./testvenv/lib/python3.7/site-packages (from botocore->awscli-login) (2.8.1)
Requirement already satisfied, skipping upgrade: urllib3<1.26,>=1.20; python_version != "3.4" in ./testvenv/lib/python3.7/site-packages (from botocore->awscli-login) (1.25.10)
Requirement already satisfied, skipping upgrade: importlib-metadata; python_version < "3.8" in ./testvenv/lib/python3.7/site-packages (from keyring->awscli-login) (1.7.0)
Requirement already satisfied, skipping upgrade: SecretStorage>=3; sys_platform == "linux" in ./testvenv/lib/python3.7/site-packages (from keyring->awscli-login) (3.1.2)
Requirement already satisfied, skipping upgrade: jeepney>=0.4.2; sys_platform == "linux" in ./testvenv/lib/python3.7/site-packages (from keyring->awscli-login) (0.4.3)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in ./testvenv/lib/python3.7/site-packages (from requests->awscli-login) (2.10)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in ./testvenv/lib/python3.7/site-packages (from requests->awscli-login) (3.0.4)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in ./testvenv/lib/python3.7/site-packages (from requests->awscli-login) (2020.6.20)
Requirement already satisfied, skipping upgrade: pyasn1>=0.1.3 in ./testvenv/lib/python3.7/site-packages (from rsa<=4.5.0,>=3.1.2; python_version != "3.4"->awscli->awscli-login) (0.4.8)
Requirement already satisfied, skipping upgrade: six>=1.5 in ./testvenv/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore->awscli-login) (1.15.0)
Requirement already satisfied, skipping upgrade: zipp>=0.5 in ./testvenv/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->keyring->awscli-login) (3.1.0)
Requirement already satisfied, skipping upgrade: cryptography in ./testvenv/lib/python3.7/site-packages (from SecretStorage>=3; sys_platform == "linux"->keyring->awscli-login) (3.0)
Requirement already satisfied, skipping upgrade: cffi!=1.11.3,>=1.8 in ./testvenv/lib/python3.7/site-packages (from cryptography->SecretStorage>=3; sys_platform == "linux"->keyring->awscli-login) (1.14.1)
Requirement already satisfied, skipping upgrade: pycparser in ./testvenv/lib/python3.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography->SecretStorage>=3; sys_platform == "linux"->keyring->awscli-login) (2.20)
Building wheels for collected packages: awscli-login
Building wheel for awscli-login (setup.py) ... done
Created wheel for awscli-login: filename=awscli_login-0.1.0a5-py3-none-any.whl size=37905 sha256=8ae34faf84b11a8237f0de5e3472465b22b2790997c3fafcfdf8cd0ccf6373a6
Stored in directory: /tmp/pip-ephem-wheel-cache-8234a34d/wheels/f5/2d/15/dada127d400de6504a958268b535444a9bd985f839b3c5bdfe
Successfully built awscli-login
Installing collected packages: awscli-login
Successfully installed awscli-login-0.1.0a5