Jenkins and Machine Learning Plugins for Data Science

Machine learning (ML) has established itself as a key data science (DS) technology in finance, retail, marketing, science, and many other fields. Machine learning algorithms learn by analyzing features of training data sets that can then be applied to make predictions, estimations, and classifications in new test cases. The power of machine learning comes at the cost of complex development and production environment to support the data, algorithms, and models generated from machine learning [1, see figure below]

It is envisioned that Jenkins and its plugin ecosystem can play a key role in supporting, facilitating and accelerating the integration of ML workflow components and orchestrating their reproducible execution. Currently, Python and interactive computational notebooks (such as Jupyter and Zeppelin) are the dominant software tools for teaching, composing and executing machine learning workflows. Interactive notebooks have proven valuable data exploration and teaching tools as they adopt a ‘literal programming’ paradigm where code fragments, results, instructions and documentation are integrated into a single UI, resembling a familiar paper ‘notebook’. However, interactive notebooks present significant limitations to the adoption of good software engineering principles (test-driven development, code versioning, reproducibility, etc) as well as scaling-up to data sizes typically used in DS production environments [2]. We propose the development of a Jenkins plugin that will integrate machine learning tasks and algorithms to build ML data systems that are easier to develop, test, deploy and maintain. The hidden power of interactive computational notebooks comes from back-end ‘polyglot’ interpreters (a.k.a kernels) that interpret the code fragments, and return the results of the computation is a useful display (graph, table or other). We propose to build a Jenkins plugin that will allow the integration of existing polyglot notebook kernels to support notebook - like computations as Jenkins build tasks. The goal is to allow data scientists to define production-grade ML workflows as Jenkins builds, integrating typical Jenkins build tasks with notebook code fragments as build steps. The plugin will support a new build task, to execute code via an appropriate language ‘kernel’ as currently supported by existing notebooks such as Zeppelin and Jupyter.

The project involves interaction between a Jenkins node and an IPython Kernel. Jenkins must be able to connect to a running IPython Kernel - or start one if necessary - to be used to evaluate code used in a Jenkins plugin. Apache Zeppelin will be used for this interaction, as they provide existing Java code to interact with an IPython Kernel [3]. In Jenkins, we will use the simple Builder plugin from the Jenkins tutorial [4], modifying its original code to call the IPython instead of printing Hello World.

Here’s a step-by-step of what the student will be expected to achieve.

Extra steps could involve investigate and document for future developers (which is a good practice in Software Engineering) what would be required to use other Kernels (e.g. Perl, R, Groovy, etc), where the code would have to be modified, and perhaps also see if we could use Apache Zeppelin Interpreters [5] from Jenkins.

Other tools that researchers use for experiment tracking (should not be seen as a direct alternative! Some can be used together with Jenkins!) * https://colab.research.google.com/ * https://neptune.ml/ ( ​$99/user/mo) * https://www.comet.ml/ ( ​$39/user/mo) * https://polyaxon.com/ ​ ​ * https://github.com/ModelChimp/modelchimp ​ * https://github.com/datmo/datmo * https://github.com/IDSIA/sacred + https://github.com/vivekratnavel/omniboard / https://github.com/chovanecm/sacredboard * Airflow * studio.ml * https://medium.com/polyaxon * Version Control System for Machine Learning Projects https://dvc.org/

We currently do not have any issues but can readily answer questions via the gitter channel here