CoreML Namespace

Important

Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.

The CoreML namespace provides relatively high-level access to machine learning model runtimes.

Classes

Name	Description
MLArrayBatchProvider	An IMLBatchProvider backed by an array.
MLComputePlan
MLComputePlanCost
MLComputePlanDeviceUsage
MLCpuComputeDevice
MLCustomLayer_Extensions	Extension methods to the IMLCustomLayer interface to support all the methods from the MLCustomLayer protocol.
MLCustomModel	Interface defining a custom CoreML model.
MLCustomModel_Extensions	Extension methods to the IMLCustomModel interface to support all the methods from the MLCustomModel protocol.
MLDictionaryConstraint	Contains a value that constrains the type of dictionary keys.
MLDictionaryFeatureProvider	An implementation of IMLFeatureProvider that is backed by a NSDictionary.
MLFeatureDescription	A developer-meaningful description of a MLModel feature.
MLFeatureValue	An immutable value and MLFeatureType for a feature.
MLFeatureValueImageOption
MLGpuComputeDevice
MLImageConstraint	Contains constraints for an image feature.
MLImageSize	Describes one acceptable image size for the CoreML model inputs.
MLImageSizeConstraint	Description of the constraint on image sizes for a CoreML model.
MLKey
MLMetricKey
MLModel	Encapsulates a trained machine-learning model.
MLModel_MLState
MLModelAsset
MLModelCollection
MLModelCollection.Notifications	Notifications posted by the MLModelCollection class.
MLModelCollectionEntry
MLModelCompilationLoadResult	This class holds the return values for an asynchronous operation.
MLModelCompilationResult	This class holds the return values for an asynchronous operation.
MLModelConfiguration
MLModelDescription	A developer-meaningful description of the MLModel.
MLModelErrorExtensions	Extension methods for the MLModelError enumeration.
MLModelMetadata	A DictionaryContainer that holds metadata related to a MLModel.
MLModelStructure
MLModelStructureNeuralNetwork
MLModelStructureNeuralNetworkLayer
MLModelStructurePipeline
MLModelStructureProgram
MLModelStructureProgramArgument
MLModelStructureProgramBinding
MLModelStructureProgramBlock
MLModelStructureProgramFunction
MLModelStructureProgramNamedValueType
MLModelStructureProgramOperation
MLModelStructureProgramValue
MLModelStructureProgramValueType
MLMultiArray	Represents an efficient multi-dimensional array.
MLMultiArrayConstraint	Contains constraints for a multidimensional array feature.
MLMultiArrayDataPointer	This class holds the return values for an asynchronous operation.
MLMultiArrayMutableDataPointer	This class holds the return values for an asynchronous operation.
MLMultiArrayShapeConstraint	Describes the constraints on the shape of the multidimensional array allowed by the model.
MLNeuralEngineComputeDevice
MLNumericConstraint
MLOptimizationHints
MLParameterDescription
MLParameterKey
MLPredictionOptions	Contains a value that indicates whether to restrict prediction computations to the CPU.
MLSequence	Encodes a sequence as a single input.
MLSequenceConstraint	A constraint on sequences of features.
MLState
MLStateConstraint
MLTask
MLUpdateContext
MLUpdateProgressHandlers
MLUpdateTask

Interfaces

Name	Description
IMLBatchProvider	Interface defining the protocol for providing data in batches to the model.
IMLComputeDeviceProtocol	This interface represents the Objective-C protocol `MLComputeDeviceProtocol`.
IMLCustomLayer	Interface defining methods necessary for a custom model layer.
IMLCustomModel	Interface defining a custom CoreML model.
IMLFeatureProvider	An interface that defines input or output features and allows access to their values.
IMLWritable	This interface represents the Objective-C protocol `MLWritable`.

Enums

Name	Description
MLComputeUnits
MLFeatureType	Enumerates the kinds of features supported by CoreML.
MLImageSizeConstraintType	Enumerates the form of a MLImageSizeConstraint.
MLModelError	Enumerates errors that may occur in the use of Core ML.
MLMultiArrayDataType	Enumerates the types of values stored in a MLMultiArray.
MLMultiArrayShapeConstraintType	Enumerates the form of a MLMultiArrayShapeConstraint.
MLReshapeFrequencyHint
MLSpecializationStrategy
MLTaskState
MLUpdateProgressEvent

Delegates

Name	Description
MLModelAssetGetFunctionNamesCompletionHandler
MLModelAssetGetModelDescriptionCompletionHandler
MLStateGetMultiArrayForStateHandler
MLStateGetPredictionCompletionHandler

Remarks

The CoreML namespace allows runtime querying of a broad variety of machine-learning models produced by frameworks such as scipy-learn, TensorFlow, and Azure Custom Vision cognitive services. CoreML does not support on-device modification of models or weights, however it does support loading a model from a URL, so developers could use that to download updated models.

CoreML relies on a "model" that is distributed as a single .mlmodel file. This model is compiled into a usable form by either using integrated tools in Xcode, Visual Studio, or at the command-line. At the command line, an .mlmodel file can be compiled with xcrun coremlcompiler compile model.mlmodel outputfolder . A compiled model takes the form of a directory called modelname.mlmodelc . The model is then loaded at runtime with code similar to:

var bundle = NSBundle.MainBundle;
var assetPath = bundle.GetUrlForResource("MarsHabitatPricer", "mlmodelc");
NSError mlErr;
model = MLModel.Create(assetPath, out mlErr);

Models in CoreML are fairly "opaque box" and do not have an extensive API. Rather, the iOS developer must know the input and output names and types the model expects. For instance, an image-recognition model might expect a CVPixelBuffer of size 227x227 identified as "image" and might have two outputs: a string identified as "classLabel" and a NSDictionary with NSString keys and double values in the range [0,1] representing the confidence of the prediction.

Developers must convert their native representations into CoreML-compatible instances of IMLFeatureProvider . The easiest way to do this is with a MLDictionaryFeatureProvider :

CVPixelBuffer pixelBuffer = // from image or video stream
var imageValue = MLFeatureValue.Create(pixelBuffer);

var inputs = new NSDictionary<NSString, NSObject>(new NSString("image"), imageValue);

NSError error, error2;
var inputFp = new MLDictionaryFeatureProvider(inputs, out error);
if(error != null)
{
	ErrorOccurred(this, new EventArgsT<string>(error.ToString()));
	return;
}
var outFeatures = model.GetPrediction(inputFp, out error2);
if(error2 != null)
{
	ErrorOccurred(this, new EventArgsT<string>(error2.ToString()));
	return;
}

var predictionsDictionary = outFeatures.GetFeatureValue("classLabelProbs").DictionaryValue;

A more complex, but more flexible, way is to implement IMLFeatureProvider :

public class MarsHabitatPricerInput : NSObject, IMLFeatureProvider
{
	public double SolarPanels { get; set; }
	public double Greenhouses { get; set; }
	public double Size { get; set; }

	public NSSet<NSString> FeatureNames => new NSSet<NSString>(new NSString("solarPanels"), new NSString("greenhouses"), new NSString("size"));

	public MLFeatureValue GetFeatureValue(string featureName)
	{
		switch (featureName)
		{
			case "solarPanels":
				return MLFeatureValue.Create(SolarPanels);
			case "greenhouses":
				return MLFeatureValue.Create(Greenhouses);
			case "size":
				return MLFeatureValue.Create(Size);
			default:
				return MLFeatureValue.Create(0);
		}
	}
}

Getting a prediction occurs synchronously, with a call to GetPrediction :

IMLFeatureProvider outFeatures = model.GetPrediction(pricerInput, out var prErr);
double result = outFeatures.GetFeatureValue("price").DoubleValue;

CoreML currently supports:

Type	Variants	Produced by:
Neural networks	Convolutional, feed-forward, recurrent	Caffe, Keras, Azure Custom Vision
Tree ensembles	Random forests, boosted trees, decision trees	scikit-learn, XGBoost
SVMs	Scalar and multiclass	scikit-learn, LIBSVM
Generalized linear models	Linear and logistic regression	scikit-learn
Pipeline models	Sequentially chained models	scikit-learn

Apple has open-sourced (3-clause BSD License) python tools to create CoreML models: https://pypi.python.org/pypi/coremltools