Andy in the Cloud

From BBC Basic to and beyond…

Image Recognition with the Salesforce Einstein API and an Amazon Echo


AI services are becoming more and more accessible to developers than ever before. Salesforce acquired Metamind last year and made some big announcements at Dreamforce 2016. Like many developers, i was keen to find out about its API. The answer at the time was “check back with us next year!”.

pipaWith Spring’17 that question has been answered. At least thus far as regards to image recognition, with the availability of Salesforce Einstein Predictive Vision Service (Pilot). The pilot is open to the public and is free to signup.

True AI consists of recognition, be that visual or spoken, performing actions and the final most critical peace, learning. This blog explores the spoken and visual recognition peace further, with the added help of Flow for performing practically any action you can envision!

You may recall a blog from last year relating to integrating Salesforce with Amazon Echo. To explore the new Einstein API, I decided to leverage that work further. In order to trigger recognition of my pictures from Alexa. Also the Salesforce Flow usage enabled easy extensibility via custom Apex Actions. Thus the Einstein Apex Action was born! After a small bit of code and some configuration i had a working voice activated image recognition demo up and running.

The following diagram breaks down what just happened in the video above. Followed by a deeper walk through of the Predictive Vision Service and how to call it.


  1. Using Salesforce1 Mobile app I uploaded an image using the Files feature.
  2. Salesforce stores this in the ContentVersion object for later querying (step 6).
  3. Using the Alexa skill, called Einstein, i was able to “Ask Einstein about my photo”
  4. This  NodeJS skill runs on Amazon and simply routes requests to Salesforce Flow
  5. Spoken terms are passed through to a named Flow via the Flow API.
  6. The Flow is simple in this case, it queries the ContentVersion for the latest upload.
  7. The Flow then calls the Einstein Apex Action which in turn calls the Einstein REST API via Apex (more on this later). Finally a Flow assignment takes the resulting prediction of what the images is actually of, and uses it to build a spoken response.

Standard Example: The above example is exposing the Einstein API in an Apex Action, this is purely to integrate with the Amazon Echo use case. The pilot documentation walks you through an standalone Apex and Visualforce example to get you started.

How does theEinstein Predictive Vision Service API work?

revaflintsilverThe service introduces a few new terms to get your head round. Firstly a dataset is a named container for the types of images (labels) you want to recognise. The demo above uses a predefined dataset and model. A model is the output from the process of taking examples of each of your data sets labels and processing them (training). Initiating this process is pretty easy, you just make a REST API call with your dataset ID. All the recognition magic is behind the scenes, you just poll for when its done. All you have to do is test the model with other images. The service returns ranked predictions (using the datasets labels) on what it thinks your picture is of. When i ran the pictures above of my family dogs, for the first time i was pretty impressed that it detected the breeds.


While quite fiddly at times, it is also well worth the walking through how to setup your own image datasets and training to get a hands on example of the above.

How do i call the Einstein API from Apex?

Salesforce saved me the trouble of wrapping the REST API in Apex and have started an Apex wrapper here in this GitHub repo. When you signup you get private key file you have to upload into Salesforce to authenticate the calls. Currently the private key file the pilot gives you seems to be scoped by your org users associated email address.

public with sharing class EinsteinAction {

    public class Prediction {
        public String label;
        public Double probability;

    @InvocableMethod(label='Classify the given files' description='Calls the Einsten API to classify the given ContentVersion files.')
    public static List<EinsteinAction.Prediction> classifyFiles(List<ID> contentVersionIds) {
        String access_token = new VisionController().getAccessToken();
        ContentVersion content = [SELECT Title,VersionData FROM ContentVersion where Id in :contentVersionIds LIMIT 1];
        List<EinsteinAction.Prediction> predictions = new List<EinsteinAction.Prediction>();
        for(Vision.Prediction vp : Vision.predictBlob(content.VersionData, access_token, 'GeneralImageClassifier')) {
            EinsteinAction.Prediction p = new EinsteinAction.Prediction();
            p.label = vp.label;
            p.probability = vp.probability;
            break; // Just take the most probable
        return predictions;

NOTE: The above method is only handling the first file passed in the parameter list, the minimum needed for this demo. To bulkify you can remove the limit in the SOQL and ideally put the file ID back in the response. It might also be useful to expose the other predictions and not just the first one.

The VisionController and Vision Apex classes from the GitHub repo are used in the above code. It looks like the repo is still very much WIP so i would expect the API to change a bit. They also assume that you have followed the standalone example tutorial here.


This initial API has made it pretty easy to access a key part of AI with what is essentially only a handful of simple REST API calls. I’m looking forward to seeing where this goes and where Salesforce goes next with future AI services.

24 thoughts on “Image Recognition with the Salesforce Einstein API and an Amazon Echo

  1. Andrew, awesome post as always. Thank you for leading us to these newer & exciting features. You are true guru.

  2. Nice work, Andy. You’ll be delighted when I publish my fully-fledged PV API on Friday at London’s Calling… 😉

  3. Amazon Echo system was really nice inventions, its works good. This show the important of sales force cloud and in this APEX is also used. Thank You.

  4. Did you ever get the Model training from a zip file? It almost seemed like the zip file needed to be hosted on a publicly accessible site.

    I pulled all the Apex classes, Visualforce pages, and Remote Site settings needed to get the basic functionality together into a single unmanaged package. Just add your own predictive_services.pem into a file and go.

  5. That’s really cool!

    Do you know of a way to get the Echo to read a google spreadsheet?

  6. Is it possible to get the image name with probability and Label?

  7. Hi Andrew,

    I have tried creating the modelID but training status is getting failed(Several times).
    Error Message: failureMsg : java.lang.RuntimeException: Training failed to complete.

    Have trained several datasets model and used them in predictive vision service.Just wondering is there any limitation on model creation under one user. Even i tried using other predictiveservice key to generate new token but still same failure message coming. Can you please assist and guide us to discuss this issue.


    • {“datasetId”:1003289,”datasetVersionId”:1417,”name”:”Diper and Wipes Model”,”status”:”FAILED”,”progress”:0.17,”createdAt”:”2017-05-23T12:59:35.000+0000″,”updatedAt”:”2017-05-23T12:59:55.000+0000″,”failureMsg”:”java.lang.RuntimeException: Training failed to complete”,”learningRate”:0.001,”epochs”:3,”object”:”training”,”modelId”:”I5EUGVFFCM5FHOPJNMHTOAQRNI”,”trainParams”:null,”trainStats”:{“labels”:2,”examples”:213,”totalTime”:null,”trainingTime”:null,”lastEpochDone”:0,”modelSaveTime”:null,”testSplitSize”:22,”trainSplitSize”:188,”datasetLoadTime”:”00:00:01:282″},”modelType”:”image”}

      Response getting on train command execution.

      • Ah ok that answers my earlier question then! 👍🏻 So this looks like an internal bug and needs raising with Salesforce via a case. You might also try posting on the GitHub repo for the API wrappers, as the devs on that might be able to help with debug. Let me know how you get on, sorry I cannot be of much help in this case.

    • I am not aware of this limit. I do wonder if more detailed error info is not being shown by java. Can you repo via curl? If so this will also give you a good basis for a support case.

  8. Great article, thank you, Andrew! We have also made a research on different Image Recognition APIs, you may find it useful:

  9. Pingback: Highlights from TrailheaDX 2017 | Andy in the Cloud

  10. I would like to see comparison to

  11. Pingback: Einstein Vision – Real Estate App – ForceOlympus

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s