A framework is specified as a name, version, along with names of the containers the container can run in. This information is stored within a “manifest” file. The following shows an example of a manifest file for (MXNet)[mxnet.io]
name: MXNet # name of the framework
version: 0.1 # framework version
container: # containers used to perform model prediction
# multiple platforms can be specified
amd64: # if unspecified, then the default container for the framework is used
gpu: raiproject/carml-mxnet:amd64-cpu
cpu: raiproject/carml-mxnet:amd64-gpu
ppc64le:
cpu: raiproject/carml-mxnet:ppc64le-gpu
gpu: raiproject/carml-mxnet:ppc64le-gpuOut of the box, MLModelScope has built-in support for Caffe, Caffe2, CNTK, MXNet, TensorFlow, and TensorRT. MLModelScope uses “vanilla” unmodified versions of the frameworks and uses facilities within the framework to enable layer-level profiling — this allows \carml to work with binary versions of the frameworks (version distributed through Python’s pip, for example) and support customized or different versions of the framework with no code modifications.
A predictor needs to implement the following interface
type Predictor interface {
// Gets framework and model manifests
Info() (dlframework.FrameworkManifest, dlframework.ModelManifest, error)
// Downloads model from manifest
Download(ctx context.Context, model dlframework.ModelManifest, opts ...options.Option) error
// Load model from manifest
Load(ctx context.Context, model dlframework.ModelManifest, opts ...options.Option) (Predictor, error)
// Returns the prediction options
GetPredictionOptions(ctx context.Context) (*options.Options, error)
// Returns the preprocess options
GetPreprocessOptions(ctx context.Context) (PreprocessOptions, error)
// Returns the handle to features
Predict(ctx context.Context, data [][]float32, opts ...options.Option) error
// Returns the features
ReadPredictedFeatures(ctx context.Context) ([]dlframework.Features, error)
// Clears the internal state of a predictor
Reset(ctx context.Context) error
io.Closer
}A mock predictor is provided. As discussed, the predictor interface section section, the code needs to implement a few methods to satisfy the predictor interface and be compatible with MLModelScope. This section walks through how to implement a simple predictor (called mock). This predictor does not require any resources, and just returns “mock” as the label for each input with 100% probability.
The mock predictor demonstrates that 100 lines is enough to have code that works. It also shows that the concept of framework and model is arbitrary, in fact this mock predictor uses a mock framework (that does nothing) and a mock model (that also does nothing). This is useful, since allows one to define arbitrary functions that do not follow the framework/model heirarchy (e.g. SVM classifier, sentiment analysis, PageRank, etc…).
As with other predictors, this mock predictor is accessible from within the command line, REST API, website, etc…
Since this a mock predictor, the major code that needs to be implemented is one that creates the labels. For each element in the batch, we create a new classification prediction which has the “mock” label with 100% probability.
func (p *Predictor) ReadPredictedFeatures(ctx context.Context) ([]dlframework.Features, error) {
batchSize := int(p.options.BatchSize())
output := make([]dlframework.Features, batchSize)
for ii := 0; ii < batchSize; ii++ {
output[ii] = []*dlframework.Feature{
feature.New(
feature.ClassificationIndex(0),
feature.ClassificationLabel("mock"),
feature.Probability(1),
),
}
}
return output, nil
}MLModelScope relies on some wrapper code to provide a consistent interface to create and load ML models
func New(model dlframework.ModelManifest, opts ...options.Option) (common.Predictor, error) {
framework, err := model.ResolveFramework()
if err != nil {
return nil, err
}
return &Predictor{
Framework: framework,
Model: model,
options: options.New(opts...),
}, nil
}
func (p *Predictor) Load(ctx context.Context, model dlframework.ModelManifest, opts ...options.Option) (common.Predictor, error) {
framework, err := model.ResolveFramework()
if err != nil {
return nil, err
}
return &Predictor{
Framework: framework,
Model: model,
options: options.New(opts...),
}, nil
}
func (p *Predictor) GetPreprocessOptions(ctx context.Context) (common.PreprocessOptions, error) {
return common.PreprocessOptions{
Context: ctx,
}, nil
}
func (p Predictor) GetPredictionOptions(ctx context.Context) (*options.Options, error) {
return p.options, nil
}
func (p Predictor) Info() (dlframework.FrameworkManifest, dlframework.ModelManifest, error) {
return p.Framework, p.Model, nil
}There are quite a few methods that do not apply for the mock predictor, for those cases we do nothing an return a success flag
func (p *Predictor) Download(ctx context.Context, model dlframework.ModelManifest, opts ...options.Option) error {
return nil
}
func (p *Predictor) Predict(ctx context.Context, data [][]float32, opts ...options.Option) error {
return nil
}
func (p *Predictor) Reset(ctx context.Context) error {
return nil
}
func (p *Predictor) Close() error {
return nil
}Finally, if we need to use the system in a distributed fashion, then we need to register the predictor as an agent.
func init() {
config.AfterInit(func() {
framework := mock.FrameworkManifest
agent.AddPredictor(framework, &Predictor{
Framework: framework,
})
})
}