Modeling is about capturing patterns in a moving and complex world. Working with live data, one must monitor the performance of the model over time and update it accordingly. Dataiku provides many capabilities to handle the whole lifecycle of a model in production.
When the design of a model is finished in a Visual Analysis, one has to deploy the model in the flow in order to score new data.
Deploying a model creates a “saved” model in the flow together with its lineage: it is the output of a train recipe which takes as input the original training data that was used while designing the model.
The “saved” model contains the model that was built within the Visual Analysis. Suppose that the data is changing over time. In order to capture the latest trends, one can retrain the “saved model” from the Actions menu on the right. A new version of the model will be created in the “saved model”:
The new version of the model is activated, meaning it will be used when data is being scored with that “saved model”. Note that the old version of the model is kept so that you can rollback to this version if the new one is not satisfactory.
The train recipe does a lot more than just training a new model. When a new version is trained, Dataiku DSS automatically computes its associated performance metrics. The train recipe splits its input dataset into two datasets used for training and validation. By default, the model trained this way is used. If you want to use a model trained on the full input dataset, you can change the setting of the train recipe:
During the design and the training stage of a model, data scientists are validating its quality through metrics measuring the difference between its predictions and the actual values observed in the past.
When put into production, the model starts scoring data it has never seen. One can imagine a situation where because the train & validation of the design and train stages has not been set properly, the model in production scores badly. It is thus important to create a validation feedback loop to verify that the newly scored data achieves the original goal of the model.
In order to do so, one creates a new validation set composed of newly labeled data. Its rows contain both the score given in production by the model and the final observed values. One then uses an evaluation recipe to compute the true performance of the “saved model” against this new validation dataset:
By frequently updating feedback loop validation data, one can monitor the model performance drift in time.
Note that you can use Scenarios to: