abstract There have been thousands of articles written about Python virtual environments. As of 2020-10-07, a quick search on Google for “python virtualenv article” returns “about 1,450,000 results”. In the majority of cases I have seen, they are partially filled with impractical advice. They gush about pip, pipx, venv, virtualenvwrapper, pipenv, poetry and others, but they don’t explain how to achieve an ergonomic execution of a Python program once the virtualenv is in place, and in all environments. To be clear, too many of these articles imply, infer, or gloss over an ugly reality of their suggested approach — that that command line used to run project components will be different between environments, thus turning something trivial into something diabolical.

This article attempts to correct that by explaining how to achieve a smooth as possible deployment experience across environments (from local development, to remote production). This is at once both pretty basic and very important, but I have found that “doing the basic stuff really well” has proved to be a profitable approach.

preliminaries / terminology

It’s important to be clear about what it is we are discussing.

versions When “Python” is mentioned here, version 3.6 or later of CPython on a Debian-based Linux installation is assumed. Win32 and Python 2, fine platforms that they are, do not come under consideration, except perhaps by way of comparison to illustrate a concept or example.

venv The "venv" module does the same job as the “virtualenv” module from Python 2. It creates and populates the .venv/ directory.

pip The “pip” module is the tool which installs modules into .venv/lib/.../site-packages/ for use by the project. By the convention described here, the packages and their versions are enumerated in the file etc/pip/requirement.txt (singular)

bash We use bash for a wrapper script to make the .venv/ environment easily accessible. [1]

structure Source code arrangement is a topic all-too-often overlooked. I suggest that the filesystem-level layout of your project is as just as important as any other design decision or ergonomic consideration. Think of it like a sign saying “Please keep the workshop and van clean and tidy.” It’s a good idea, even if we don’t manage it all the time, but so much else is easier and quicker if we do.

polyglot Python is not the only game in town. It is common for large projects to use more than one language. These languages and their associated runtime systems will have different conventions on how things work. To reduce friction in daily navigation, all languages, their libraries and tooling, must live in predictable, obvious locations in the project tree. A little consideration goes a long way.

Things that this solution does not do.

Multiple python versions The solution presented here is a minimal-but-complete solution for the common case. It does not attempt to to rival or compete with any of the capabilities of virtualenvwrapper, pipenv, poetry and others. I understand that library maintainers may wish to run their unit tests on multiple versions of the Python runtime, and this is a laudable goal.

Multiple transitive library dependencies If your project requires both library A and B, and A requires library C version 4, and B requires library C version 5 we have a problem which no virtualenv management tool can reasonably resolve unless it starts adjusting module locations at installation time, and rewriting imports of library C within libraries A and B. While this may be conceivable, it is not a robust approach, and on balance should be avoided.

installation

bin/venv-create This invokes the venv module to create the .venv/ directory, and pip to populate the latter with the modules listed in etc/pip/requirement.txt

execution

There are two wrapper scripts for convenience / practicality. (A PhD student working as a TA once asked me what a wrapper script was. I mean, had he no imagination?) The main purpose of wrapper scripts is to ensure the sanity of the execution environment, so to call them “convenience scripts” is to understate their necessity.

bin/venv-python This sets PYTHONPATH to include src/python/main/, and PYTHONDONTWRITEBYTECODE and then calls .venv/bin/python, passing along any arguments.

If invoked via a symlink, it invokes .venv/bin/python3 with the “-m” option, passing along any arguments, the first one being the module name, which it gets from the source name of the symlink. This allows us to create something the symlink

bin/jupyterlab -> venv-python

and running bin/jupyterlab does the right thing, with the expected environment, without typing a long path and all is well.

bin/run-main Purely as a matter of convention, this runs src/main/python/main.py via bin/venv-python — you will likely want to add further “run-*” scripts as copies of this to suit your requirements.

That is pretty much it — some simple execution machinery underneath to create a predictable interface above, allowing the user to run the system in the same fashion in all environments. In the end, isn’t that what it’s all about?

complaints department

Thank you for your time and attention. If you have anything to say (either good or bad) about what I have written above, I would be very grateful to hear from you. Alternatively, if you have nothing to say but you like it anyway, then please tell a friend.

[1] The GNU bash shell is to be tamed, not avoided. If you think that knowing this is beneath you, then you need to think again. A serious chef keeps sharp knives.