Key Word(s): Virtual environments, conda environments

Download Notebook

Lecture 14

Tuesday, October 22th 2019

Practice with Packaging and Deployment

Note: No need to turn lecture exercises in today.

Package Requirements

  • Come up with a basic project directory structure (this one will be pretty bare bones). For example:

       myproj\
         myproj\
               __init__.py
               module.py
               tests/
                    __init__.py
         README.md
         setup.py
         LICENSE

  • Note: I'm using the "Tests as part of application code" from the pytest documentation.

Virtual Environments

  • You've got a package ready to go
  • You probably (hopefully) developed it in the CS207 working directory
  • What if you're developing many packages with different requirements?
  • How do you deploy it?

Virtual Environments

  • A virtual environment allows the developer to develop (and deploy) a package without having to worry about dependencies
    • For example, your project may include several external dependencies
    • You don't want to muck up or pollute your global environment with all these dependencies
    • However, you need to install those dependencies to use the package!

Virtual Environments

  • You may have different packages with different dependencies and you'd like to keep those environments separate while developing
  • You may want to make it easy for a user to install dependencies so they can get up and running with your package right away

What are Virtual Environments?

A virtual environment is a directory with the following components:

  1. site_packages/ directory where third-party libraries are installed.
  2. Links [really symlinks] to the Python executables on your system.
  3. Some scripts that ensure that Python code uses the interpreter and site packages in the virtual environment.

virtualenv

virtualenv

  • virtualenv has historically been an indispensable tool for managing virtual environments
  • Nowadays, virtualenv has been incorporated into broader tools such as pipenv
  • virtualenv is still a great way of illustrating the basic ideas
  • virtualenv is shipped with Anaconda

Getting Going with virtualenv

  1. You probably already have virtualenv, but just in case type sudo easy_install virtualenv
  2. Now, navigate to the top level of your project directory and type virtualenv env
    • What do you see when you do ls?
  3. Next type source env/bin/activate
    • What do you see?

Comments on virtualenv

  • You created and activated a virtual environment called env
  • Now everything you install will be in that virtual environment
  • Within the virtual environment type pip install sparsegrad. (See sparsegrad)
    • Type python to start the python interpreter
    • In the python interpreter type import sparsegrad
    • It works!

More comments on virtualenv

  • Now exit the python interpreter
  • And exit the virtual environment (Just type deactivate)
  • Open the python interpreter again
    • Notice that you're going to run python outside of the virtual environment
  • Type import sparsegrad
  • It's not there!

Summary so Far

Hopefully you have the basic idea now of the utility of virtual environments for development purposes.

They can also be useful for developers to share codes (deployment).

Suppose you have a package that you want to share with me.

You want to make it easy for me to install the dependencies, but otherwise you trust that I more or less know what I'm doing.

A Scenario

Here's a possible scenario:

  • You send me the package (probably in an archive file like .tar.gz) and the requirements.txt file
  • I create a directory called myproj and unpack the project into that directory
  • Then I do the following:
    • virtualenv env
    • source env/bin/activate
    • pip install -r requirements.txt

conda

conda environments

virtualenv is great, but conda environments are a bit more powerful.

  • pip is a package manager for Python.
  • virtualenv is an environment manager for Python.
  • conda is a package and environment manager and is language agnostic

conda vs. pip vs. virtualenv

  • Create an environment with the command
conda create --name env_name python
  • env_name is the name of the environment.
  • You can specify a different Python version with
conda create --name env_name python=3.7
  • You can even install other packages:
conda create -n env_name python=3.7 numpy
  • Note: You can also specify which versions of the packages you'd like to install.

Activating the environment

  • You can try conda activate env_name. This will work on many platforms.

    • It is possible that conda will complain that your shell hasn't been configured to use conda activate.
    • You can run conda init, but this may modify your bash_profile to always use a conda base environment.

  • Alternatively, you may simply use source activate env_name.

  • You can get out of the environment (deactivate it) by typing conda deactivate.

Where conda environments live

  • When working with virtualenv, we saw that our environment lives directly in the package directory.
  • conda environments live, by default, in /Users/user-name/anaconda3/envs (or similar).
  • Note: You can override the default path by using the --prefix (or -p) flag: conda create -p /path/to/env_name.
    • This can complicate things. See the docs for more information.
  • You can list your environments with conda env list.

Installing Packages in a conda environment

  • Inside the environment simply type conda install package_name=version.
  • List the installed packages in your environment with conda list.
  • You can install packages with pip as well.
    • You should try conda install first.

Note: To remove a conda environment, use conda remove --name env_name --all.

Exercise (Part 1)

  • Clone the course playground repo.
  • Go into playground.
  • Create a conda python virtual environment
    • You can name it anything you like.
    • Don't install numpy or pytest for the moment.
  • Activate the virtual environment.
  • Try to run the tests with pytest.

Exercise (Part 2)

  • Create a new conda python virtual environment
    • Install numpy or pytest when you build the environment.
  • Activate the virtual environment.
  • Try to run the tests with pytest.

Exercise (Part 3)

  • Create a new conda python virtual environment using conda env create -n env_name -f environment.yml
  • Activate the virtual environment.
  • Try to run the tests with pytest.

Exercise Summary

  • The virtual environment doesn't have the correct dependencies in general --- you need to customize this.
  • In this exercise, the package came with an environment.yml file.
  • Using the environment.yml file, you can create a virtual environment with the exact same setup that the package developers use.
  • Note: If you want to generate the environment.yml file for your own packages, simply use conda env export -f environment.yml.

Notes

  • You may should probably remove the three environments that you just created.
  • If you want to add the packages in an environment.yml file you can use conda env update -n conda-env -f environment.yml.

Concluding Remarks on Virtual Environments

Advantages

  • Pretty useful stuff if you're working with a fellow developer.
  • To remove all the things that we don't want, just remove the virtual environment directories.

Disadvantages

  • Not so great if we want to deploy our package to a wider audience
  • The process can be streamlined a fair bit (e.g. Pipenv)

Additional Reading

Deployment and Exercise 2

Now let's say you want to make your Complex package more widely available.

You should follow the tutorial at Packaging Python Projects.

You can work with the tutorial practice package (rather than the Complex package).

Notes:

  • Might need to use conda install -c conda-forge twine to get twine
  • You will need to verify your email address after you sign up for TestPyPI
  • Please use a conda environment rather than virtualenv.

Further notes on Deployment