HomeWinBuzzer NewsMicrosoft Open Sources BERT for ONNX Runtime

Microsoft Open Sources BERT for ONNX Runtime

Microsoft says ONNX Runtime now helps to power BERT for open source development on the Bing search platform.


In December, open sourced its ONNX Runtime inference engine. Now, the company says it also open-sourced an optimized version of BERT, a natural language model from , for ONNX.

By using ONNX Runtime with BERT, users can lower latency for language representation on the Bing search platform. Microsoft has previously said BERT brings “the largest improvement in search experience” for Bing.

With ONNX Runtime support, developers can use BERT to scale to as low as 1.7 milliseconds latency (alongside a V100 GPU). Microsoft told VentureBeat that this capability has previously only been available to major tech companies.

“Since the BERT model is mainly composed of stacked transformer cells, we optimize each cell by fusing key sub-graphs of multiple elementary operators into single kernels for both CPU and GPU, including Self-Attention, LayerNormalization and Gelu layers. This significantly reduces memory copy between numerous elementary computations,” Microsoft senior program manager Emma Ning said today in a blog post.


Open Neural Network Exchange (ONNX) creates a standard open platform for AI models that will work across frameworks.

ONNX Runtime is a high-performance inference engine for machine learning creations across Windows, Linux, and Mac. Developers can use the service to train AI models in any framework and turn these models to production in the cloud and edge.

Developed with and , the platform is growing in popularity. By its full launch in 2017 Facebook said several major tech companies have joined. Among them are AMD, ARM, IBM, , , NVIDIA, and Qualcomm.

Luke Jones
Luke Jones
Luke has been writing about all things tech for more than five years. He is following Microsoft closely to bring you the latest news about Windows, Office, Azure, Skype, HoloLens and all the rest of their products.

Recent News