Big Data Architektur

Any Dataiku DSS tool, whether it is visual data manipulation recipes, a code recipe, guided machine learning or data visualizations, can be run using an in-cluster engine. Dataiku DSS leverages various technologies (Hive, Impala, Spark, MLlib, H2O…) to achieve this.

Machine learning algorithms can be run distributed both for training and for scoring using these engines:

Spark MLlib

H2O

Vertica Advanced Analytics

• SQL visual recipes can use Hive, Impala, and Spark engines.
• Data visualization can use Impala.

It’s all magic: prepare your data and create machine learning models as you would do for in-memory processing. Dataiku will take care of the plumbing to make it happen.

Create recipes using your favorite language Hive, Impala, Pig, SparkR, PySpark, SparkSQL, or Spark Scala. You can also code in a notebook.

• Reduce Spark engine overhead and unecessary intermediary datasets write thanks to the Spark pipelines
• Become a Spark master by learning the Spark tips and troubleshooting

Optimize the speed of your computation with partitions on HDFS.

The host running DSS should have client access to the cluster. Learn how to set up the Hadoop integration.

To set up your Spark environment and enable its integration in DSS, please refer to this page of our documentation.