Python is one of the top preferred languages for creating AI applications. The large number of frameworks and libraries available for machine learning, its flexibility and the big community are only part of the reasons why python is so popular among data scientists.
At the same time, more and more of these python-based AI applications are running in production and need to support large-scale data.
Building python code that handles large scale while still keeping the code simple and concise is a complicated task.
We will describe the different challenges that we tackle with respect to scale and give concrete examples from Salesforce Einstein – a python-based AI solution dealing with massive amounts of data.