smhanov

Pipeline multiprocessing in Python with generators

Similar to mpipe, this short module lets you string together tasks so that they are executed in parallel.

The difference is that it lets you use generators, functions which yield results instead of returning them. Each yielded item gets passed to the next stage.

You can specify that one or more copies of the workers operate on the input queue.

Things that can be in a stage

danoneata

import numpy as np
import pdb

from sklearn.datasets import make_classification
from sklearn.mixture import GaussianMixture as GMM


def fisher_vector(xx, gmm):
    """Computes the Fisher vector on a set of descriptors.