Agenda

Time Event
19:00 - 19:05 Introduction from Andrew Ng
19:05 - 20:00 Talk + Q&A
20:00 - 21:00 Networking
Pie & AI: Grenoble - Running machine learning models in the browser

Introduction from Andrew Ng

Pie & AI: Grenoble - Running machine learning models in the browser

Running machine learning models in the browser

Summary

  1. 🧠 ML workflow
  2. 🤝 ONNX format
  3. 🌍 WebAssembly
  4. ⚙️ Deploy in the browser
  5. 🧑‍💻 Examples
  6. 🏁 Final notes
Pie & AI: Grenoble - Running machine learning models in the browser

😸 It's a cat!

What are we trying to achieve?

Quick demo

Pie & AI: Grenoble - Running machine learning models in the browser

🧠 ML workflow

Two distinct phases:

  1. Model training
  2. Model inference
Pie & AI: Grenoble - Running machine learning models in the browser

🧠 ML workflow

Model training

  • Offline
  • Data collection
  • Compute intensive (CPU, GPU)
  • Python, R, Julia, MATLAB, etc.
Pie & AI: Grenoble - Running machine learning models in the browser

🧠 ML workflow

Model inference

  • Online runtime
  • Predictions from trained model
  • Less compute intensive
  • Python, C++, Go, Rust, etc.
Pie & AI: Grenoble - Running machine learning models in the browser

🤝 ONNX format

Open Neural Network Exchange

  • Generic ML model representation
  • Common file format
  • Training / inference loose coopling
  • Interoperability
  • Inference in any language
  • Inference on multiple backends
Pie & AI: Grenoble - Running machine learning models in the browser

🤝 ONNX format

Export models from favourite framework:

⚠️ Some models or layer types might not be supported by generic operators yet!

Pie & AI: Grenoble - Running machine learning models in the browser

🤝 ONNX format

Using Netron to visualize an ONNX model

Pie & AI: Grenoble - Running machine learning models in the browser

🤝 ONNX format

Available runtimes:

  • C/C++
  • Python
  • ...
  • Web!
Pie & AI: Grenoble - Running machine learning models in the browser

🌍 WebAssembly

WASM

  • Portable compilation target
  • Client and server applications
  • Major browsers support (desktop, mobile)
  • Fast, safe and open
  • Privacy
Pie & AI: Grenoble - Running machine learning models in the browser

🌍 WebAssembly

Famous usage in Data Science ecosystem:

Pie & AI: Grenoble - Running machine learning models in the browser

⚙️ Deploy in the browser

Training (with Scikit-Learn)

# Train model on training dataset
model.fit(X_train, y_train)

# Convert to ONNX
onnx_model = skl2onnx.to_onnx(model, X_train[:1].astype(np.float32))

# Save ONNX model to ONNX format
onnx_model_path.write_bytes(onnx_model.SerializeToString())
Pie & AI: Grenoble - Running machine learning models in the browser

⚙️ Deploy in the browser

Inference (with ONNX Runtime Web)

<script type="module">
// import ONNXRuntime Web
import * as ort from "https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/esm/ort.min.js"

// Create an inference session and load the model
const session = await ort.InferenceSession.create('./model.onnx')

// prepare inputs
const area = new ort.Tensor('float32', Float32Array.from([areaInput]), [1, 1])
const rooms = new ort.Tensor('float32', Float32Array.from([roomsInput]), [1, 1])
const latitude = new ort.Tensor('float32', Float32Array.from([latitudeInput]), [1, 1])
const longitude = new ort.Tensor('float32', Float32Array.from([longitudeInput]), [1, 1])
const inputs = { area, rooms, latitude, longitude }

// feed inputs and run
const results = await session.run(inputs)

// read from results
const outputVariable = results.variable.data
</script>
Pie & AI: Grenoble - Running machine learning models in the browser

🧑‍💻 Examples

GitHub Repository

pieandai-grenoble/2024-02-23-ml-models-web

Pie & AI: Grenoble - Running machine learning models in the browser

🧑‍💻 Examples

🏠 Housing Value Estimation

Pie & AI: Grenoble - Running machine learning models in the browser

🧑‍💻 Examples

🍿 Sentiment Analysis

Pie & AI: Grenoble - Running machine learning models in the browser

🧑‍💻 Examples

🌉 Image Classification

Pie & AI: Grenoble - Running machine learning models in the browser

🏁 Final notes

✨ Pros

  • Inference at the edge
  • No server required for inference
  • Easier app integration
  • Support for WebGPU API (experimental)

☢️ Cons

  • Lack of documentation
  • Memory limitations
  • Model size
Pie & AI: Grenoble - Running machine learning models in the browser

📚 References

Pie & AI: Grenoble - Running machine learning models in the browser

🙋 Q&A

Thank you for your attention!

Any questions?

Pie & AI: Grenoble - Running machine learning models in the browser

Thank you for attending the first Pie & AI event in Grenoble, France!

🔥 Tip

You will receive a promo code for a 50% off first month subscription on Coursera after filling the form sent by DeepLearning.ai

Pie & AI: Grenoble - Running machine learning models in the browser

Quick demo of image classification with a cat

Note: while it may be less compute intensive than training, it can become power hungry when used at scale with lots of users (cf. OpenAI with ChatGPT)

Note: while the name implies NN, it can be used for any type of model not only NNs

Note: runtime reference implementation in C++, bindings for most languages

Note: backends available for CPU, CUDA, OpenVINO, proprietary HW accelerators, etc.

Source: https://azure.microsoft.com/fr-fr/blog/onnx-runtime-for-inferencing-machine-learning-models-now-in-preview/

Note: supported by all major browsers since 2017

Privacy: no personal information leaking server-side (cf. OpenAI)

We'll send out a survey to the list of registrants after the event, and we’ll offer course promo code to the people who complete the survey. Please remind your event attendees during the event to keep an eye out for our email if they’d like to receive the code. The code is for 50% off first month subscription for any of our courses on Coursera.