Hi, my name is Matthijs Hollemans.
I am an independent consultant who specializes in machine learning and AI on mobile and edge devices.
To get a taste for the sort of things I do, check out my blog where I write about the practical aspects of doing ML and AI on smartphones and microcontrollers.
How to do deep learning on the edge
Mobile devices are finally powerful enough to take advantage of advanced machine learning and artificial intelligence algorithms, so it makes sense that developers will now want to put deep learning into their apps.
Sounds great, but how do you get started? Here’s what it takes to add machine learning to your app and your business:
- you need to design a model using an architecture that is suited to the problem that you’re trying to solve
- you need to collect and clean up the data for training the model
- you need to train the model so that it works well on new, previously unseen data
- and then you need to make the trained model work reliably in a production environment.
Building a machine learning model in a research lab is one thing — but when you put the model into an actual app… that is where theory meets the real world.
You need an experienced engineer to help make this transition from lab to app.
What can I do for you?
The scope of the projects I get hired for varies a lot. Sometimes it’s a simple fix to a Core ML model file, sometimes it’s building a complete end-to-end solution for data collection, model training, and deployment of the model in the app.
Here is a selection of the work I commonly do for my clients:
Searching for an existing machine learning model.
Creating your own machine learning model can be a lot of work. Sometimes you get lucky and someone else has already done the hard work for you! There are many trained models available for free online already.
However, many freely available models will not work well on mobile, or have license restrictions. I can evaluate existing models to see if they are suitable for your app — and if not, how they can be adapted to your needs.
Converting models to run on iOS devices.
Even though there are automatic conversion tools for translating a trained model to Core ML, often the conversion process requires manual intervention. Typically this involves creating custom layers and GPU compute kernels, writing a dedicated converter, hacking the coremltools source code, or tweaking your original PyTorch or TensorFlow code. I’ve done a lot of this, so I can usually get the job done quickly.
Sometimes Core ML is not the right choice. I’ve also helped clients with converting their models to Metal, to make sure they run on the GPU as fast as possible. This is especially important for older devices (iPhone X and below). For certain kinds of models, such as RNNs, a hand-optimized CPU-based implementation can be the best approach.
(I also occasionally help with converting models to Android, but I’m not an Android expert.)
Debugging issues with models.
Even if model conversion runs without errors, you can still get unexpected results. Parts of the model may not have been converted correctly and may require model surgery afterwards. Sometimes the model needs to be re-achitected.
Fixing these requires deep knowledge of Core ML internals. I’ve written a book about this, Core ML Survival Guide, which already should put you well on your way. But instead of reading 400 pages, you can also simply hire me. 😃
I can also help with getting models to run on other runtimes, such as TensorFlow Lite, PyTorch Mobile, Metal, Accelerate framework, Eigen, and others.
Pre-processing and post-processing.
The machine learning model is often only a small piece of the picture. The data that goes into the model usually needs some kind of pre-processing, which can range from simple cropping to advanced computer vision techniques.
It’s also not always obvious how to use the output from the model. Often this includes a post-processing step, which ranges from trivial code to super tricky math. I’ve helped clients by converting the required post-processing code from Python, MATLAB, or C++ into high-performance Swift code. I don’t just convert this code, I also optimize it for speed and battery usage.
Integrating the model into the app.
The Core ML API is pretty simple, but how do you get the most performant results and a seamless user experience?
Especially for real-time video, you want a pipeline that works as efficiently as possible. For many of the apps I’ve worked on, this means taking the output from the ML model and using it to render graphics using Metal.
Integration with other libraries, such as Vision or OpenCV, is also a common task. The model itself may run as part of a larger algorithm that uses preprocessing and postprocessing steps, and may even use multiple models.
Advising on the design of models.
Too often I receive email from people who designed and trained their model without taking the capabilities and limitations of the iPhone or iPad in mind. Their models are too big to fit in an app, are too slow to be usable, or require too much RAM and crash the app when trying to do inference.
Just because a certain model architecture achieves state-of-the-art performance, does not mean it’s suitable for mobile. You can avoid these costly mistakes by involving me early on in the model selection process.
For many tasks, such as object detection, the smart thing is to take a proven model architecture such as YOLOv3 or SSD, and train it on your own data. There are tools that make this easy, such as Create ML, but they don’t always give the best possible results. I’ve helped clients by training their models in TensorFlow and PyTorch.
Designing new models.
For many tasks that you might want to use machine learning for, there isn’t an established architecture yet — or existing techniques are too slow or expensive. I can help with the research & development of an appropriate machine learning architecture for these tasks.
Personally, I find this the most exciting because it involves inventing something new. However, research is unpredictable by nature and you should only undertake this if you have a budget for it. Clients sometimes underestimate this, despite my best efforts to coach them.
Optimizing / compressing trained models.
Let’s say you have a machine learning model that works well on your server, but it’s too big or too slow on mobile. I can compress and optimize such models to make them small enough to fit on the device, but still perform up to your standards.
Creating specialized tools for building models.
Training a completely new model usually also involves collecting training data and annotating it. I’ve written tools that help with the curation and annotation process.
Even though Core ML now supports on-device training, it is still rather limited. I’ve helped clients to implement custom backpropagation and other optimization algorithms, both on GPU and CPU.
A little more about me…
I’ve been coding for 35 years, 26 of those professionally.
In the past I’ve worked on games, graphics, programming languages, enterprise systems, back-ends, front-ends, process control systems, mobile apps, and many other types of software.
I’ve worked at large companies, at startups, and as an independent developer and contractor. Because of these varied experiences, it’s safe to say I am an all-round developer. I’m comfortable with many different programming languages and tools.
Recently I’ve been focusing exclusively on machine learning — and especially deep learning. I try to stay up-to-date with the latest research and practical techniques.
Some career highlights:
I’m the original author of The iOS Apprentice, a book for complete beginners that teaches iOS and Swift development from the ground up. It is published through the popular tutorial website raywenderlich.com.
I also founded Swift Algorithm Club, an open source project that implements many algorithms and data structures in Swift with detailed explanations — It currently has over 23 thousand stars on GitHub!
My Core ML Helpers library is a popular open source library that makes Core ML a bit easier to use.
Until recently I was licensing an iOS and macOS source code library with fast Metal-based implementations of MobileNet V1 and V2, as well as SSDLite and DeepLabv3+.
As an independent developer, I’ve published several iPhone and iPad apps over the years. Currently I’m selling the game Mahjong Cards for iPad.
I also mentor young developers.
My focus is on doing high quality work. When you hire me, you’ll be guaranteed a good working relationship, attention to detail, and a product we can both be proud of. I am only happy if the software is the best that it could possibly be!
Contact me to talk about your product and your plans for it. I’d be happy just to chat about your idea and the possibilities for turning it into an awesome mobile app. I hope to hear from you soon!