MacOS ships with Python 2.7 Pre-installed. This is helpful when you are just getting started with Python since you can just start using it with no setup.
Unfortunately, Python 2.7 is going out of support in 2020, so we all need to get to upgrading to Python 3. When you start upgrading Python, there is a good chance that Python will get messed up on your computer. This article will help you troubleshoot and solve some of the issues you might encounter as you start moving to Python 3.X on MacOS.
Preventing Problems
Most issues you will encounter will be related to where Python is installed vs what paths are in your system path variables vs where pip is installing your python modules too. You need to make sure all of these file system paths match up, otherwise, you will encounter issues where you install a package with pip, but python still complains that the package is missing. Or you might find that you are not running the version of python you thought you were running.
The best way to prevent problems is to install virtualenv at the system level. And use virtual environments for everything else. Instructions on virtual environments can be found here.
Next, avoid using sudo when installing python packages. When you install a package using sudo, you install as the root user in the root users environment on the entire system. This works if you are going to be running all of your code as root. But typically that is not the case.
Even if you are going to be running something as root, if you properly used virtual environments, you will have access to all the packages that you need. Even if you installed the packages without being root. This is because when you use a virtual environment, all the python binaries and packages are stored in your virtual environment folder.
If you don’t want to continue troubleshooting the Python issues you are experiencing right now, try virtual environments now and see if that resolves your issues.
Reinstalling Python
A lot of issues can be resolved by simply re-installing Python. This is easy to do if you used Homebrew. You can first run the following to uninstall the homebrew version of python 2 and python 3:
brew uninstall python2 python3
To re-install python 2 and 3, run:
brew install python2 python3
After this is done, test again and see if you are still getting errors. If you continue having trouble, try removing python2 and python3 again and only installing python3.
If you want more information on homebrew, check out our article on getting started with Tensorflow. There is information about how to install home brew.
Installation Paths
By default, MacOS uses Python installation located at:
/usr/local/bin/python
If you installed Python3 using Home-brew, your python3 install will be at this location:
/usr/local/Cellar/python/<Python Version>
If you installed Python2 using homebrew, your python2 install will be at this location:
/usr/local/Cellar/python@2/<Python Version>
Checking System and Python Path(s)
The system path variables tell your terminal which file system paths to look at when you run a command. If you have your python folder as part of the path variable, then you will be able to run the python command to execute your scripts.
If Python is not part of your path variable, then you will need to give the full path to your python binary file every time you run a command, Example: /usr/bin/python vs just typing “python”
You can check what folders are currently part of your path variable by running the following:
Echo $PATH
You should see a bunch of folders listed. Most likely you will see the /usr/bin folder listed. This is where you would expect the Python binary file(s) to be located. If you run the following, you will see all the python files in that folder:
ls -l /usr/bin | grep python
Here is screenshot showing you what the output might look like:
Notice there are a few instances of python. We see python and python 2.7. In this case, Python is simply the binary file that gets called when we run the python command. python2.7 is a symlink meaning it is just a shortcut that points to another location. If we run the python2.7 command, it will run the binary file located in the ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 location.
A shortcut to see all the different python paths on your system is to run the following two commands:
python -c "import sys; print(sys.path)"
Python3 -c "import sys; print(sys.path)"
If you are not sure the path of the python installation you are currently using (As we have seen, it is possible to have more than one installed), you can type:
which python -or- which python3
Example output might look like:
/usr/bin/python or /usr/local/bin/python3
Next, you can see which installation of pip you are using by typing:
which pip -or- which pip3
Next you can see where pip is installing the packages to by typing:
pip show
where *package name* is the name of a package you have installed. for example:
pip show virtualenv
might output:
/Library/Python/2.7/site-packages
Next, you can confirm that python is looking at the same path by running:
python -m site
Example output from the above command might look like this:
Notice /Library/Python2.7/site-packages is listed in the sys.path variable, which is the same place that pip is installing to. If this was missing from the path, then when we install a pip package, python still won’t know about it.
Adding to the system paths
In the previous section, we talked about where Python is installed to and where it looks to find installed packages. If a path is incorrect, it can cause things to not work. There are two kinds of paths to look at.
There are System paths which are where the system looks to find executable files. Then there is there are python paths, which is where python looks to find packages.
The system paths are stored in the /etc/paths file. You can manipulate this file by running:
sudo vi /etc/paths
You can add/remove paths to this file if a given path is missing. Be careful when modifying this file since it can break your whole computer.
Next, we have the python paths. In the previous section we ran this command to list everything in the python path:
python -c "import sys; print(sys.path)"
We can temprarily add and remove paths from this list by running:
sys.path.append -or- sys.path.remove
This sequence will:
- list all the paths in pythonpath
- add something to the list
- query the list again
- remove that item from the list
- show the list again
import sys
print(sys.path)
sys.path.append('/Users/sean/test')
print(sys.path)
sys.path.remove('/Users/sean/test')
print(sys.path)Using these commands you can add/remove to the pythonpath at runtime. The problem is this is not persistent. As soon as your session ends, everything goes back to normal. If you want it to be persistent, you need to modify or create a python path file
To add a path to our python3 installation, start by finding your site-packages folder by running python3 -m site
Note the line that contains site-packages. On my computer the path is: /usr/local/lib/python3.7/site-packages
next, you need to create a file with any name, but it has to end with a .pth extension. I am going to create example.pth with the following command:
sudo vi /usr/local/lib/python3.7/site-packages/example.pth
Now I enter any paths I want to include. For this example, I will enter:
/Users/sean/test
Save the file and test to see if it now shows up in your python path:
python3 -c "import sys; print(sys.path)"
You have now added the missing path to your python installation.
You can remove this path by modifying or deleting example.pth
