{"id":3057,"date":"2018-08-15T10:22:00","date_gmt":"2018-08-15T08:22:00","guid":{"rendered":"https:\/\/test.viaboxx.de\/2018\/08\/15\/configuring-paperspace-ml-in-a-box-machine-to-run-a-deeplearning-showcase\/"},"modified":"2021-08-03T11:49:45","modified_gmt":"2021-08-03T11:49:45","slug":"configuring-paperspace-ml-in-a-box-machine-to-run-a-deeplearning-showcase","status":"publish","type":"post","link":"https:\/\/www.viaboxx.de\/en\/blog\/configuring-paperspace-ml-in-a-box-machine-to-run-a-deeplearning-showcase\/","title":{"rendered":"Configuring Paperspace ML-in-a-Box Machine to run a deeplearning-showcase"},"content":{"rendered":"\n

Introduction<\/strong><\/p>\n\n\n\n

Paperspace<\/a> is a cloud provider that specialises in GPU enabled machines. It has a set of publicly available images including the ML-in-a-Box<\/em> machines. These machines can be provisioned on multiple types of hardware, for example P4000, P5000, and P6000 machines.
\nThe ML-in-a-Box image comes preconfigured with Keras, Tensorflow, Tensorboard, Cuda, and other standard machine learning frameworks. Unfortunately, that does not mean it is ready to plug and play any deeplearning application.
\nIn fact, while trying to run our own “deeplearning-showcase” project we ran into many problems that needed some “not so easy” solutions sometimes. In this blog post we will provide a walkthrough of how we provisioned the ML-in-a-Box machine and got our “deeplearning-showcase” up and running.<\/p>\n\n\n\n

Setting up the Paperspace account<\/strong><\/p>\n\n\n\n

The first step is to sign up to Paperspace and setting up a payment method. Head to the sign-up page<\/a> on Paperspace and enter your details, then head to the payment methods page<\/a>  and click the “Add Card” button. After filling up the details, you can add a PromoCode if you have one. Some PromoCodes are easily found online and will add 10$ to your account credit.<\/p>\n\n\n\n

Request access to ML-in-a-Box Machines<\/strong><\/p>\n\n\n\n

The second step is to request access to the ML-in-a-Box images and the GPU enabled machines. Make sure to do that as early as possible as it takes some time to approve your request. Head to the console page<\/a>  and click the green “NEW MACHINE” button, choose the server location (e.g. CA1, NY2, AMS1) then navigate to “Public Templates”. A pop-up window will appear with the title “Unavailable” asking you for the reason you want to use this machine, close this window for now and proceed to choose the ML-in-a-Box Ubuntu 16.04 Templates, then choose one of the hardware specs available (P4000, P5000, P6000).<\/p>\n\n\n\n

Once you did that, the same pop-up window from before will appear, read carefully and fill in the reason you want to have access for these machines and image Templates.<\/p>\n\n\n\n

Starting the ML-in-a-Box machine<\/strong><\/p>\n\n\n\n

Some time after requesting access (could be up to 24 hours) you will receive an email stating that you now have access to the requested service. Proceed again to the console page and follow the following steps to provision your machine :<\/p>\n\n\n\n

  1. Go to the console page again and click the “NEW MACHINE” button.<\/li>
  2. Choose server location, OS (Public Templates > ML-in-a-Box Ubuntu 16.04), and the hardware wanted (P5000 in our case).<\/li>
  3. Choose the cost calculation method ( hourly vs monthly).<\/li>
  4. Follow the Paperspace guide and choose all your other preferences, then click to Provision the new machine.<\/li>
  5. Navigate back to the console page again and wait for your machine to start, then hover over the machine info box and click “launch”.<\/li>
  6. Now you are in your machine.<\/li><\/ol>\n\n\n\n

    Problems & Error Codes<\/strong><\/p>\n\n\n\n

    After starting our Ml-in-a-Box instance we tried to run our “deeplearning-showcase” using:<\/p>\n\n\n\n

    python3 train_DenseNet121.py<\/pre>\n\n\n\n
    But we faced a number of issues described below. All the issues revolved around the compatibility between Tensorflow on one side, and CUDA & the machines CPU on the other.<\/p>\n\n\n\n
    After solving all these problems we were able to run our showcase with 96% GPU utilisation and 97% GPU Memory utilisation. The details about the deeplearning showcase will be available in another blog post.<\/p>\n\n\n\n
    Problem 1 : Unable  to execute Tensorflow Callbacks<\/strong><\/p>\n\n\n\n
    Problem Description:<\/strong><\/p>\n\n\n\n
    While Tensorflow is trying to execute the callbacks, it errors out on a missing attribute in module ‘pandas.core.computation’. the problem is having an older version of Dask.
    \nsee     this Github Issue<\/a> for more details.<\/p>\n\n\n\n
    Stacktrace:<\/strong><\/p>\n\n\n\n
     File “train_DenseNet121.py”, line 50, in <module><\/div>
        main()<\/div>
      File “train_DenseNet121.py”, line 30, in main<\/div>
        validation=dnn.valid_batches)<\/div>
      File “\/home\/paperspace\/Desktop\/deeplearning-showcase\/model.py”, line 116, in train<\/div>
        write_graph=False)<\/div>
      File “\/home\/paperspace\/Desktop\/deeplearning-showcase\/model.py”, line 305, in __init__<\/div>
        super(TrainValTensorBoard, self).__init__(training_log_dir, **kwargs)<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/keras\/callbacks.py”, line 709, in __init__<\/div>
        from tensorflow.contrib.tensorboard.plugins import projector<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/__init__.py”, line 22, in <module><\/div>
        from tensorflow.contrib import bayesflow<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/bayesflow\/__init__.py”, line 28, in <module><\/div>
        from tensorflow.contrib.bayesflow.python.ops import layers<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/bayesflow\/python\/ops\/layers.py”, line 26, in <module><\/div>
        from tensorflow.contrib.bayesflow.python.ops.layers_dense_variational_impl import *<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/bayesflow\/python\/ops\/layers_dense_variational_impl.py”, line 30, in <module><\/div>
        from tensorflow.contrib.distributions.python.ops import deterministic as deterministic_lib<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/distributions\/__init__.py”, line 37, in <module><\/div>
        from tensorflow.contrib.distributions.python.ops.estimator import *<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/distributions\/python\/ops\/estimator.py”, line 21, in <module><\/div>
        from tensorflow.contrib.learn.python.learn.estimators.head import _compute_weighted_loss<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/learn\/__init__.py”, line 92, in <module><\/div>
        from tensorflow.contrib.learn.python.learn import *<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/learn\/python\/__init__.py”, line 23, in <module><\/div>
        from tensorflow.contrib.learn.python.learn import *<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/learn\/python\/learn\/__init__.py”, line 25, in <module><\/div>
        from tensorflow.contrib.learn.python.learn import estimators<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/learn\/python\/learn\/estimators\/__init__.py”, line 297, in <module><\/div>
        from tensorflow.contrib.learn.python.learn.estimators.dnn import DNNClassifier<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/learn\/python\/learn\/estimators\/dnn.py”, line 30, in <module><\/div>
        from tensorflow.contrib.learn.python.learn.estimators import dnn_linear_combined<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/learn\/python\/learn\/estimators\/dnn_linear_combined.py”, line 31, in <module><\/div>
        from tensorflow.contrib.learn.python.learn.estimators import estimator<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/learn\/python\/learn\/estimators\/estimator.py”, line 48, in <module><\/div>
        from tensorflow.contrib.learn.python.learn.learn_io import data_feeder<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/learn\/python\/learn\/learn_io\/__init__.py”, line 21, in <module><\/div>
        from tensorflow.contrib.learn.python.learn.learn_io.dask_io import extract_dask_data<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/tensorflow\/contrib\/learn\/python\/learn\/learn_io\/dask_io.py”, line 26, in <module><\/div>
        import dask.dataframe as dd<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/dask\/dataframe\/__init__.py”, line 3, in <module><\/div>
        from .core import (DataFrame, Series, Index, _Frame, map_partitions,<\/div>
      File “\/home\/paperspace\/anaconda3\/lib\/python3.6\/site-packages\/dask\/dataframe\/core.py”, line 40, in <module><\/div>
        pd.core.computation.expressions.set_use_numexpr(False)<\/div>
    AttributeError: module ‘pandas.core.computation’ has no attribute ‘expressions’<\/div><\/blockquote>\n\n\n\n
    Solution:<\/strong><\/p>\n\n\n\n
    Upgrade to latest version of Dask:<\/p>\n\n\n\n
     pip install dask --upgrade <\/span><\/code><\/pre>\n\n\n\n
    Note:<\/strong> At the time of writing this post the latest version of Dask is 0.18.2. After running the above command you should be able to see the following text as a conformation of Dask installation:<\/p>\n\n\n\n
    Successfully installed dask-0.18.2<\/p><\/blockquote>\n\n\n\n
    Problem 2: Tensorflow unable to run AVX instructions on CPU<\/strong><\/p>\n\n\n\n
    Problem Description:<\/strong><\/p>\n\n\n\n
    The Machine has an older CPU that does not support AVX instructions. Following is a quote from official Tensorflow release notes:<\/a><\/p>\n\n\n\n
    Breaking Changes
    \nPrebuilt binaries will use AVX instructions. This may break TF on older CPUs.<\/p><\/blockquote>\n\n\n\n
    For more details please take a look at this Stackoverflow Post<\/a> and this GitHub issue<\/a>.<\/p>\n\n\n\n
    Stacktrace:<\/strong><\/p>\n\n\n\n
    running the training using “python3 train_DenseNet121.py” will print out the following line:<\/p>\n\n\n\n
    Illegal instruction (core dumped)<\/p><\/blockquote>\n\n\n\n
    It could be that no more details or stacktrace is visible.<\/p>\n\n\n\n
    Solution:<\/strong><\/p>\n\n\n\n
    revert TensorFlow and the TensorFlow GPU from versions 1.8 to versions 1.5<\/p>\n\n\n\n
    pip3 uninstall tensorflow <\/span>\npip3 uninstall tensorflow-gpu <\/span>\npip3 install tensorflow==1.5 <\/span>\npip3 install tensorflow-gpu==1.5 <\/span><\/code><\/pre>\n\n\n\n
    To test the installation run the following commands to print out the current tensorflow and tensorflow-gpu versions<\/p>\n\n\n\n
    python3 <\/span>-<\/span>c <\/span>'import tensorflow as tf; print(tf.__version__)'\npython3 <\/span>-<\/span>c <\/span>'import tensorflow-gpu as tf-gpu; print(tf-gpu.__version__)' <\/span><\/code><\/code><\/pre>\n\n\n\n
    Problem 3: Unable to run Tensorflow 1.8 with Cuda 9.1<\/strong><\/p>\n\n\n\n
    Problem Description:<\/strong><\/p>\n\n\n\n
    TensorFlow and TesnorFlow GPU versions 1.5  and later are not compatible with Cuda 9.1.
    \nlooking at     TensorFlow Release Notes<\/a> again shows that the binaries are built against Cuda 9.0<\/p>\n\n\n\n
    Breaking Changes
     Prebuilt binaries are now built against CUDA 9.0 and cuDNN 7.<\/p><\/blockquote>\n\n\n\n
    Stacktrace:<\/strong><\/p>\n\n\n\n
    ImportError: Could not find ‘cudart64_90.dll’. TensorFlow requires that this DLL be installed in a directory that is named in your %PATH% environment variable. Download and install CUDA 9.0 from this URL: https:\/\/developer.nvidia.com\/cuda-toolkit<\/p><\/blockquote>\n\n\n\n
    Solution:<\/strong><\/p>\n\n\n\n
    revert back to Cuda 9.0 following this guide.<\/a><\/p>\n\n\n\n