Problems with AWS Linux and PIP

TL;DR

Before creating a virtual environment in an AWS Linux instance, I’m going to save my sanity and run:
unset PYTHON_INSTALL_LAYOUT

The Situation

If you read my post about using pymssql in AWS Lambda, then you know that I use an AWS Linux instance, not Ubuntu or CentOS, for deploying Python packages because it closely mirrors the Linux environment in AWS Lambda. If it works in one, then it works in the other. Maybe I’ll start getting into Cloud9 soon but I digress. After creating a virtual environment in the instance, I will use the ‘pip‘ command to install my applications dependencies in the virtual environment. However, I’ve recently come to learn that AWS Linux instances have a weird quirk with ‘pip’ that is annoying but simple to solve.

The Problem

To illustrate, I created a virtual environment called ‘small_env’ and installed the ‘Cython’ package inside using ‘pip’.

As you see from the above image, ‘Cython’ installed without issue; but, when I ask ‘pip’ to show me the installed packages, ‘Cython’ is not found.

How could the package have installed correctly and not be listed? I ran a ‘find’ command and discovered that Cython was indeed installed in my virtual environment. It was installed in the small_env/lib64/python3.6/dist-packages/ directory. I thought this was odd since ‘pip’ on my Mac always installs things in a ../site-packages/ directory. It’s not a big deal though until you realize that the default Python PATH in AWS Linux doesn’t include the former directory:

This is caused by the package ‘system-rpm-config’ which is installed as part of the “@development tools” group defining the following environment variable: PYTHON_INSTALL_LAYOUT=amzon.

The Solution

I thought I might be able to run a command to find all the paths where ‘pip’ might install something and add it to my PATH but it turns out that doesn’t fully fix the issue. What will fix the issue is a simpler command I found in a comment from a user named ‘brad-alexander’ on GitHub. Before creating the virtual environment, just run: unset PYTHON_INSTALL_LAYOUT

That’s it! Without this variable set, everything works fine, because packages are installed into the site-packages directory which is in the search path.