The current Deep Learning (DL) landscape is fast-paced and is rife with non-uniform models, hardware/software (HW/SW) stacks, but lacks a DL benchmarking platform to facilitate evaluation and comparison of DL innovations, be it models, frameworks, libraries, or hardware. Due to the lack of a benchmarking platform, the current practice of evaluating the benefits of proposed DL innovations is both arduous and error-prone — stifling the adoption of the innovations.
MLModelScope is a framework- and hardware-agnostic distributed platform for benchmarking and profiling DL models across datasets/frameworks/systems. MLModelScope offers a unified and holistic way to evaluate and inspect DL models, making it easier to reproduce, compare, and analyze accuracy or performance claims of models or systems.
More specifically, MLModelScope:
Note that MLModelScope and CarML are used interchangeably within these documents. CarML (Cognitive Artifacts for Machine Learning) is the internal code name for MLModelScope.