Environments & Installing a Python Package in Editable Mode

Some basic commands of environments. And how to edit the behavior of the Python library as you need

Installing a Python Package in Editable Mode

1. Clone the GitHub repository of the package you want to edit, and cd into that directory.

2. conda deactivate to make sure you're not in a current environment.

3. Create a dedicated conda env for that package you want to edit: conda create --name=MYENV python=3.11 -y or shorthand the --name to -n I like to add the Python version to the env name as a reminder, something like myenv-py311

4. Install the package you want with pip install PACKAGE or use python -m pip install PACKAGE it's cleaner.

5. Install the editing package in the folder you're at, which is the cloned repo of the package you want to edit pip install -e . the dot at the end means "here" i.e. in this folder.

Now when you want to edit and test the library; in terminal, you activate the conda environment you created, cd into the repo directory where you installed, and every change you make, the -e library tells Conda to run the edited package from this directory.

IMPORTANT: Don't forget to create a branch first in the GitHub repo before you edit anything! That's always recommended practice, even if it's your own project and not collaborating with anyone on it.

Notes

• If you further want your newly created conda environment to be available for a Jupyter Notebook, do pip install ipykernel and for an overkill do python -m ipykernel install --user --name MYENV

• If you want to use interactive Python from within the terminal, install ipython in the conda env.

• You can run inline Python commands with python -c separate commands with a ; and wrap the whole thing in double quotes. e.g. python -c "import pandas; print(pandas.__version__)" You can also use python3 if you still have Python2 installed as well.  

• You can use python -m pip install PACKAGE wherever you use pip install

• You can choose to install a package without its dependencies if you want more control for editing and testing. python -m pip install PACKAGE --no-deps

• You can install a bunch of libraries at once if you have them in a requirements.txt file or a YAML file etc. See https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html for all things Conda envrionments.

• python -m is a Mac/Linux command, for Windows it's py -m

• All a Conda env really is, a directory to install packages in; and when you "activate" a conda environment, all you're doing is pointing your Python path to that directory. That's why if you downloaded, say a version of scikit-learn, in a certain conda environment, then you went back to where this scikit-learn was installed and modified something in the code, that conda env is going to pull that modified version of the library. Installing editable mode mentioned above python -m pip install -e . helps editing a package from an IDE instead of manually going into paths; it tells Python to use the package in this exact folder not the one in site-packages folder where conda installs packages by default. In editing a package from GitHub repo, the package we want to edit and run is not the one in site-packages, but in the current folder in our GitHub repositories folder.

Useful pip commands

To check a library's version and main info, use pip show PACKAGE or pip freeze | grep PACKAGE though the latter may not always work. Or conda list PACKAGE

Pip official docs is a treasure trove as well https://pip.pypa.io/en/stable/user_guide/

Removing Conda environments and ghost Jupyter kernels

This can happen if you run your notebooks on some cloud virtual machine or sandbox, like a Docker container image. You create conda and Jupyter environments, and it's possible that they stick but be inaccessible when you shut down the instance and relaunch it again. To get rid of those leftovers and ghost environments. You need to remove the Jupyter ones first, then the conda ones.

Remove Jupyter kernels

jupyter kernelspec list
jupyter kernelspec uninstall <unwanted-env> -y

Repeat for every one you want to remove. You won't find a kernel for the conda environments which you haven't installed ipykernel in, so they won't show up here.

Source: https://discourse.jupyter.org/t/remove-environments-from-the-jupyter-lab-launcher/762/4 and https://stackoverflow.com/a/42647666/11381214

Remove a Conda env

conda env remove -n <env-name>