Deep Learning on Qubole Using BigDL for Apache Spark – Part 2

In Part 1 you learned how to get started with installing distributed deep learning library BigDL on Qubole. In this Part 2 of a two-part series, you will learn how to write a deep learning application on Qubole that uses BigDL to identify handwritten digits (0 to 9). In this application, you will: Train and validate LeNet-5 (Convolutional Neural Networks) model using MNIST database and then save the model. Load the saved model and identify (“recognize”) digits in a given set of arbitrary images with >= 98% accuracy. To write the application, you will use Qubole Notebooks which are great for developing applications in Scala, Python, R, running ETL jobs in Spark, as well as visualizing results of SQL in a single, collaborative environment. Qubole Notebooks give data scientists and data analysts an easy way to interact with data stored in Cloud data stores such as Amazon S3, Microsoft Azure Blob Store, Oracle BMC Object Storage, etc. Steps #1 Start Spark cluster that you configured in Part 1. #2 Switch over to Notebooks interface #3 Create new Notebook by entering respective values and select Spark cluster you configured in Part 1. #4 Set spark.executor.instances and spark.qubole.max.executors to 2. (Note: At the time […]