API Shape for the Environments Discovery Component

Requirements

The environment component  API should support following:

1. Discover installed python interpreters: This includes identifying interpreter
   details like name, version, bit-ness, interpreter paths
2. Find globally created using conda/poetry/pipenv/pyenv etc: This is another thing
   we support currently is to find all the globally created environments.
3. Find installed modules using the right package manager: This is to light up
   the various tooling that we support via python package.
4. Able to install modules using the right package manager: This is to allow us to
   do the right thing when installing a python package.
5. Search environments in a given location or locally installed environments: This
   is to find any environments, in the current directory, or any other user provided
   location.
6. Activatable environment details: If an environment requires activation, we should
   be able to get the command to activate that environment.
7. Get command to execute interpreter: Get the right command for executing python
   code, example ?python spam.py?, or ?conda run spam.py? etc.
8. Bonus:
    a. API to get module details (like installed version): This is so we can figure
       out if we support the current version of the python package. Say pytest
       (ancient version), we should be able to say this is too old for us to use.
       Instead of failing when user tries to use it.
    b. API to get published module version: This is so we can suggest they could
       upgrade to the latest.

Functionality

discovery
- identify all environments, across various distros
- search $PATH, home dir, CWD, specified "root", etc.
"current" / selected
- getSelected()
- setSelected()
caching
- lazy/background discovery (incl. identify when running or never done)
activation
- for a given env, provide the argv to activate it
execution
- for a given env, provide the argv to run Python
package management
- use the correct tool
- run relative to a given env
- identify if a package is installed
- install a package

Info

environment:

name
version
architecture
executable filename

Distributions

"global":

PSF
system
pyenv
(Windows registry)
(Windows store)

"virtual":

venv
virtualenv
conda
pipenv

Package Management Tools

poetry
pip
conda

Extras

discovery
- notify when FS changes indicate a possible need to discover (also per-distro?)
- support plugging in new distros ("locators")
package management
- report metadata (e.g. version) for an installed package
- report metadata (e.g. version) for a published package
daemon
- ...

Status Quo

Actions

discover all
set current
open REPL
run in REPL
run code
run file
run module
install module
get module version

VS Code Connections

status bar: displays env/interpreter info
quick pick (for select command): displays env/interpreter info (incl. path), shows current env
pop-ups:
- reminder to select interpreter
- "use detected venv?"
- bad pythonPath in debug config
- ...
code lens: shbang
logging (console)
logging (Python output panel)
commands:
- "Python: Select Interpreter"
- python.setInterpreter
- python.execInTerminal
- python.execInTerminal-icon
- python.execSelectionInDjangoShell
- python.execSelectionInTerminal
- python.createTerminal
data store (cached info)

Usage

"pythonPath" setting:

one-off
- internal tools
  - jedi
  - ...
- external tools
  - test discovery
  - test execution
  - linter
  - formatter
  - debugger
  - terminal: REPL
  - terminal: run selection
  - terminal: run file
- datascience
  - ...
daemon
- ...

Dependencies

shell
telemetry
...

Settings

pythonPath
condaPath
pipenvPath
poetryPath
venvFolders
venvPath
globalModuleInstallation
(envFile)

in debug configs:

pythonPath
(env)
(envFile)

References

"python.pythonPath" setting:

discovery / config
- src/client/interpreter/interpreterService.ts
- src/client/interpreter/locators/services/currentPathService.ts
- src/client/interpreter/locators/services/hashProviderFactory.ts
- src/client/interpreter/locators/services/pipEnvServiceHelper.ts
- src/client/interpreter/configuration/interpreterSelector.ts
- src/client/application/diagnostics/checks/invalidPythonPathInDebugger.ts
- src/client/application/diagnostics/checks/macPythonInterpreter.ts
- src/client/extension.ts (hasUserDefinedPythonPath())
- src/client/interpreter/display/shebangCodeLensProvider.ts
activation:
- src/client/common/terminal/environmentActivationProviders/baseActivationProvider.ts
- src/client/common/terminal/environmentActivationProviders/condaActivationProvider.ts
execution:
- src/client/common/process/pythonExecutionFactory.ts
- src/client/common/process/pythonProcess.ts
- src/client/common/process/windowsStorePythonProcess.ts
- src/client/common/process/pythonDaemon.ts
- src/client/common/process/pythonDaemonPool.ts
terminal:
- src/client/common/terminal/helper.ts
- src/client/common/terminal/service.ts
- src/client/terminals/codeExecution/terminalCodeExecution.ts
tools:
- src/client/linters/linterManager.ts
- src/client/providers/jediProxy.ts
degugger:
- src/client/debugger/extension/adapter/factory.ts
- src/client/debugger/extension/configuration/resolvers/base.ts
- src/client/debugger/extension/configuration/resolvers/launch.ts
- src/client/debugger/debugAdapter/main.ts
- src/client/debugger/debugAdapter/DebugClients/LocalDebugClient.ts
module installation:
- src/client/common/installer/condaInstaller.ts
- src/client/common/installer/moduleInstaller.ts

Locators

CurrentPathService:

"pythonPath" setting
execute to see if they work (i.e. are on $PATH):
- python
- python2
- python3
- python3.6
- python3.7
- py -2
- py -3
- py -3.6
- py -3.7
- (no 3.8, etc.!!!)

KnownPathsService:

regex for files in directories (/^python\d(\.\d*)?(\.exe)?$/)
- (no double-digit Python releases!!!)
- $PATH
- on non-Windows:
  - ['/usr/local/bin', '/usr/bin', '/bin', '/usr/sbin', '/sbin', '/usr/local/sbin']
  - $HOME + ['/usr/local/bin', '/usr/bin', '/bin', '/usr/sbin', '/sbin', '/usr/local/sbin']
  - (no $HOME + '/.local/*'!!!)
  - $HOME/anaconda/bin
  - $HOME/python/bin

GlobalVirtualEnvService:

same regex on files in bin/Scripts subdir of the following
- $HOME/ + ['envs', '.pyenv', '.direnv', '.virtualenvs']
- $HOME/ + "venvFolders" setting
- $WORKON_HOME
- $PYENV_ROOT
- $(pyenv root)
- $HOME/.pyenv
- $PYENV_ROOT/versions
- $(pyenv root)/versions
- $HOME/.pyenv/versions

WorkspaceVirtualEnvService:

same, but look in the following:
- "venvPath" setting
- current workspace root
- current workspace root + '/.direnv'

PipEnvService:

use the result of pipenv --py ${workspace root}

CondaEnvService:

look in each env provided by conda info

WindowsRegistryService:

look up any in the registry ('\\Software\\Python')
- see https://www.python.org/dev/peps/pep-0514/

WindowsStoreInterpreter

look in known install locations

API

...

Proposal

high-level API (vertical layers)
low-level API
external dependencies (horizontal layers)
adapter to connect horizontal & veritical layers (i.e. a top-level driver)
- exposes extension-specific API
- used by other components of the extension

Concepts

interpreter
- the actual Python executable
env (Python environment)
- has a single interpreter
  - may be shared with other envs
  - at the least has a unique filename for the env (e.g. symlink)
- has own sys.modules, etc.
distro (Python distribution)
- "meta" - effectively, generic properties of a distro, including its source
  - identified below as "(meta) distro"
- "install" - a single installation of a distro
  - identified below as just "distro"
- provides envs
- may provide interpreters or depend on other distros
- encapsulates the behavior for a class of envs
- may be tied to a specific package manager
- may be tied to a specific package repository
package manager
- installs packages from a repository
- looks up metadata for installed packages and those in a repository

Initial Support

Distributions

"global":

PSF
system
pyenv
(Windows registry)
(Windows store)

"virtual":

venv
virtualenv
conda
pipenv

Package Management Tools

poetry
pip
conda

Lookup

REGEX (Python executable basename): /^python\d(\.\d+)?(\.exe)?$/

"pythonPath" setting
"global / system distro
- try py -X and py -X.X
- REGEX on files in each directory in $PATH
- on non-Windows, REGEX on files in each of the following:
  - ['/usr/local/bin', '/usr/bin', '/bin', '/usr/sbin', '/sbin', '/usr/local/sbin']
  - ? $HOME + ['/usr/local/bin', '/usr/bin', '/bin', '/usr/sbin', '/sbin', '/usr/local/sbin']
  - $HOME + '/.local' + ['/usr/local/bin', '/usr/bin', '/bin', '/usr/sbin', '/sbin', '/usr/local/sbin']
  - $HOME/anaconda/bin
  - $HOME/python/bin
global "virtual" envs -- REGEX on files in bin/Scripts subdir of the following:
- $HOME/ + ['envs', '.pyenv', '.direnv', '.virtualenvs']
- $HOME/ + "venvFolders" setting
- $WORKON_HOME
- $PYENV_ROOT
- $(pyenv root)
- $HOME/.pyenv
- $PYENV_ROOT/versions
- $(pyenv root)/versions
- $HOME/.pyenv/versions
workspace virtual envs -- same, but look in the following:
- "venvPath" setting
- current workspace root
- current workspace root + '/.direnv'
pipenv
- use the result of pipenv --py ${workspace root}
conda
- look in each env provided by conda info
Windows registry
- look up any in the registry ('\\Software\\Python')
  - see https://www.python.org/dev/peps/pep-0514/
Windows store
- look in known install locations

API

info

distro
- id: string
  - unique from all other distros
  - format: "://"
  - meta ID is unique from other (meta) distros
  - meta ID is common to installs of the same (meta) distro
  - meta ID is typically a name (e.g. "conda", "PSF")
  - install ID is unique within the same (meta) distro
  - install ID may be a name or directory or some encoded info
- orgName: string
- version: Version
interpreter
- id: string // unique within same distro
- filename: string // executable
- version: Version
- arch: Architecture
- implementation
  - name: string // e.g. cpython, pypy
  - version: Version
env
- id: string // unique within same distro
  - may be name, root dir, etc.
- kind: 'system' | 'global' | 'virtual'
- distro: distro
- interpreter: interpreter
- parent?: env // e.g. if virtual
- rootDir?: string // i.e. sys.prefix
package
- name: string
- version: Version
- ...

discovery (basic)

available resources:

homedir
filesystem API (incl. watcher)
events triggered by config changes
(optionally) process execution API

interface IPythonDistroMeta {
    readonly id: string;  // usually a name (e.g. "conda", "PSF")
    readonly orgName: string;

    // Indicates that findInstalls() should be called.
    readonly onChangeToEnvs: Event<void>;

    findInstalls(rootDirs?: string[]): Promise<IPythonDistro[]>;
}

interface IPythonDistro {
    // all operations fail early if the distro is no longer installed.

    readonly info: PythonDistroInfo;

    // Indicates that findEnvs() should be called.
    readonly onChangeToEnvs: Event<void>;

    // Return all discoverable Python environments.
    findEnvs(rootDirs?: string[]): Promise<IPythonEnv[]>;

    // Return all available interpreters for the distro.
    getInterpreters(): Promise<PythonInterpreterInfo[]>;
}

interface IPythonEnv {
    // all operations fail early if the env no longer exists.

    readonly info: PythonEnvInfo;
    readonly commands: IPythonEnvCommands;
    readonly packages: IPythonEnvPackages;
}

commands

type ProcessExecutionInfo = {
    argv: string[];
    envVars?: NodeJS.ProcessEnv;
};
type ProcessExecutionResult = {
    [string, string][];  // [stdout, stderr] (one or the other)
};
type ProcessExecution = ProcessExecutionInfo | ProcessExecutionResult;

interface IPythonEnvCommands {
    runModule(module: string, ...args: string[]): Promise<ProcessExecution>;
    runScript(filename: string, ...args: string[]): Promise<ProcessExecution>;
    runCode(text: string): Promise<ProcessExecution>;
    getREPLCommand(): Promise<ProcessExecutionInfo>;
    getActivationCommand(): Promise<ProcessExecutionInfo | undefined>;
}

package management

// Operations on a Python environment's installed packages.
interface IPythonEnvPackages {
    readonly toolName: string;

    find(name: string): Promise<PythonPackageInfo[] | ProcessExecutionInfo>;
    findInstalled(name: string): Promise<PythonPackageInfo | ProcessExecutionInfo>;
    install(package: string, version?: Version): Promise<PythonPackageInfo | ProcessExecutionInfo>;
    parseOutput(text: string): PythinPackageInfo;
}

high-level

additional available resources:

data store
config
workspace roots

// discovery
interface IPythonEnvs {
    readonly selected: IPythonEnvSelected;

    // Indicates that findEnvs(), getEnvs(), or refresh() should be called.
    readonly onChangeToEnvs: Event<void>;

    // Add the distro to the list used to find environments.
    registerDistro(distro: IPythonDistroMeta): void;

    // Return all discoverable Python environments.
    findEnvs(): Promise<IPythonEnv[]>;

    // caching
    readonly hasCache: boolean;
    getEnvs(): Promise<IPythonEnv[]>;
    refresh(): Promise<void>;
}

// "current"
interface IPythonEnvSelected {
    readonly onChanged: Event<IPythonEnv | undefined>;

    get(): Promise<IPythonEnv | undefined>;
    set(env: IPythonEnv | undefined): Promise<IPythonEnv | undefined>
    default(): Promise<IPythonEnv | undefined>;
}

adapter layer

// Generate the objects (apply deps), plug in to VS Code, and wrap
// in an API object.
function install(...): Promise<PythonEnvs> { ... }

helpers

// via node module or via extension object (from VS Code API):
// environments: VSCodePythonEnvironments;

// i.e. displayName()
function summarize(python: PythonEnvironmentInfo): string { ... }

//
async function getBestPythonEnvironment(envs: IPythonEnvironments): Promise<PythonEnvironment> { ... }

//
async function isModuleAvailable(python: IPythonEnvironment, module: string): Promise<boolean> { ... }

//
async function findInWindowsRegistry(...): Promise<...> { ... }

possible

interface IPythonEnvCommands {
    runREPL(terminal: Terminal): Promise<IPythonREPL>;
    activate(terminal: Terminal): Promise<DeactivateFunc>;
}

interface IPythonREPL {
    readonly env: IPythonEnv;

    send(text: string): Promise<void>;
    exit(): Promise<void>;
}

interface IPythonDistros {
    // Indicates that findEnvs() should be called.
    readonly onChangeToEnvs: Event<IPythonDistro>;

    // Add the distro to the list used to find environments.
    register(distro: IPythonDistroMeta): void;

    // Get all the Python distros under the registered meta distros.
    findDistros(): Promise<IPythonDistro[]>;

    // Return all discoverable Python environments.
    findEnvs(rootDirs?: string[]): Promise<IPythonEnv[]>;
}

ericsnowcurrently/envs-api-design.md

Requirements

Functionality

Info

Distributions

Package Management Tools

Extras

Status Quo

Actions

VS Code Connections

Usage

Dependencies

Settings

References

Locators

API

Proposal

Concepts

Initial Support

Distributions

Package Management Tools

Lookup

API

info

discovery (basic)

commands

package management

high-level

adapter layer

helpers

possible