Operationalize models from AML workbench into Azure - Part 1

Hi All,

One of the primary tasks for any machine learning practitioner or data scientist is to operationalize machine learning models into a serving environment. This first blog post would be showing how to operationalize models using Azure Machine Learning (AML) workbench in a step to deploy models as realtime webservices into Azure.

First, Once you have your model ready to be operationlized, you need to create and set the right execution environment for your model to be deployed to in Azure.

A) Creating a machine learning model management service using CLI in Azure

First, You need to know that to provision a ml model management service in azure that acts as a serving environment for your models. You can create a new one using azure portal or by executing this command in CLI from your computer:

az ml account modelmanagement create --location westcentralus -n mymodelmanagement -g myresourcegroup

-n: where you can specify the name of the service when it is provisioned in azure.
-g: the resource group name that you would like this account model management services assigned to.

Checkpoint #1: You need to make sure to provision a ML model management account is created in azure before going forward with the rest of this article.

B) Configure the execution environment using Azure Machine Learning Workbench using CLI

Once you have a model management account, you would not be able deploy models from your workbench tool into azure using docker container images unless you provision an environment. In the below steps, we will walk through how to accomplish this.


Follow the below steps:

1) From AML workbench, open the command line window from File --> Open Command prompt.
2) You need to login to your azure account, type the following command:

     az login

3) Follow the screen instructions by logging into your azure subscription.
4) Make sure that you are using the right subscription, The below first command list all subscriptions you have, the second line set the intended subscription you would like to use.

      az account list -o table
      az account set -s <subscriptionId>

5) Verify that your current account is set correctly by exeucting the following command:

       az account show

6) To create an environment which creates set of cloud assets containing blob storage, container registry and other assets in a resource group to host docker images for your models into the cloud:

      az ml env setup -n mymldevenv --location westcentralus

Please note that we created an environment on the same location of the model management services in west central us.

7) Set the model management account

     az ml account modelmanagment set -n mymldevenv -g mymldevenvrg

 Please note that when we created an environment in step #6, it has created a resource group for this environment with the same name with rg suffix, which we used in this step to set our model management service we created in step A with this environment.

Checkpoint #2: An AML workbench environment and a model management service were provisioned before moving forward to the rest of this article.

8) Now, we are going to set the correct execution environment in CLI to connect to the provisioned environment and therefore the model management service in azure.

    a) To see available environments, run:
   
         az ml env list

    b) In my case, i do have few of these for different project. To set an active environment, run:

         az ml env set -n mymodelmanagement  -g myresourcegroup

     c) Once you run this command, you should see this confirmation message in the console window

         Compute set to mymldevenv .

9) Every time you re-open AML workbench, you need to make sure to set the right environment before deploying models as webservices into azure model management service.


In the second part, I will take the next step of publishing models into real-time web services in azure model management service in azure.


Enjoy ML :-)

- Mostafa

How to convert epoch time to datetime in Pandas

Hi,

While i am working with iot data to transform UNIX epoch time in seconds. I would like to convert epoch time in seconds into human readable date time and not a reference date which is based upon 1970.

I have my data in pandas dataframe, below screen shot shows "createdTime" column in epoch time in seconds:

Here is the code segment that convert UNIX epoch time into date time:

convert = lambda x: datetime.datetime.fromtimestamp(x / 1e3)
ds['ts'] = ds['createdTime'].apply(convert)
ds.head()


This code generates the expected output, below screen shot shows the output:

Hope this helps!

Enable Jupyter notebook in Anaconda Navigator

Hi,

After i installed the latest conda runtime (anaconda 3 x64 distro) that uses Anaconda 3 on Windows 64 bit.

When i try to click on a target environment, i see that "Open with IPython" or "Open with Jupyter Notebook"

The question is how to enable this? I found how to install Jupyter notebook package for conda environment where it would be accessible through the Navigator tool.

Follow below steps:

1) Select any of the available environment, Click on "Open terminal" window.
2) Type the following command:

conda install nb_conda

3) This will install notebook packages for Jupyter and once it is completed, The Jupyter notebook will be available for all environments.

4) To testify this work properly, Click on Jupyter link and this will open up the notebook.

5) Write some code to make sure this works with no issues.

import pandas as pd
s = pd.Series([1508258340299])
pd.to_datetime(s)

The code executed with no issues:

Hope this helps!

Error downloading files from secure sites in .NET apps 4.6.2

Hi All,

I was working on upgrading a .Net application runtime from version 4.5 to latest one 4.6.2. After i did that, the application threw an error in the step of downloading a zip file from a secure site.

The app throws thew following error:

{"The request was aborted: Could not create SSL/TLS secure channel."} when trying to download file

The code snippet that was throwing the error in the DownloadFile method in .NET WebClient class:

C# code:

 using (WebClient wc = new WebClient())

            {
                wc.DownloadFile(ssl_url, downloadedFilePath);
            }


The destination is a secure site uses SSL, after searching and trying different options, I found out the solution by enabling the TLS 1.1 and TLS 2.2 before calling download file method.

Here is the modified code snippet:


 System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls11| SecurityProtocolType.Tls12;
            using (WebClient wc = new WebClient())
            {
                wc.DownloadFile(nhtsa_url, downloadedFilePath);
            }


Hope this helps!

Load datasets from azure blob storage into Pandas dataframe

Hi,

In this post, I am sharing how to work and load data sets that are stored in Azure blob storage into Pandas data frame.

I have the full code posted in Azure notebooks. This code snippet is useful to use in any Jupyter notebook while working on your data pipeline while developing Machine Learning models.

I have exported a data set into a csv file and stored it into an Azure blob storage so i can use it into my notebooks.

Python code snippet:

import pandas as pd
import time
# import azure sdk packages
from azure.storage.blob import BlobService

def readBlobIntoDF(storageAccountName, storageAccountKey, containerName, blobName, localFileName):    
    # get an instance of blob service 
    blob_service = BlobService(account_name=storageAccountName, account_key= storageAccountKey)
    # save file content into local file name
    blob_service.get_blob_to_path(CONTAINERNAME,blobName,localFileName)
    # load local csv file into a dataframe    
    dataframe_blobdata = pd.read_csv(localFileName, header=0)
    
    return dataframe_blobdata

STORAGEACCOUNTNAME= 'STORAGE_ACCOUNT_NAME'
STORAGEACCOUNTKEY= 'STORAGE_KEY'    
CONTAINERNAME= 'CONTAINER_NAME'
BLOBNAME= 'BLOB_NAME.csv'
LOCALFILENAME = 'FILE_NAME-csv-local'

# load blob file into pandas dataframe
tmp = readBlobIntoDF(STORAGEACCOUNTNAME,STORAGEACCOUNTKEY,CONTAINERNAME,BLOBNAME, LOCALFILENAME)
tmp.head()

The full code snippet is posted in Azure Notebook here.

Enjoy!

Setup Remote Desktop for Raspberry Pi with no need for an external display

Hi,

If you are thinking about how to setup remote desktop to raspberry pi, this article is for you. I will show a walk-through to install required packages so that you are able to remote desktop from your windows machine or any other remote machine.

Steps to configure remote desktop on raspberry pi:

1) Connect to your Raspberry Pi using Putty.
2) Open a terminal window.
3) We are going to install XRDP package to configure RDP on the Pi. Before installing xrdp, we must first install the tightvncserver package.  The tightvncserver installation will also remove the RealVNC server software that ships with newer versions of Raspbian OS since tightvncserver (xrdp) will not work if RealVNC is installed. 

$ sudo apt install -y tightvncserver
$ sudo apt install -y xrdp

3) Now, Just install Samba package that provides a GUI when accessing a Pi using RDP.

$ sudo apt install -y samba

4) Open up the remote desktop tool in windows or your host OS and set the name or IP of your Pi and hit connect.

With that, we can connect to any remote Pi or Linux based IoT device from your computer; therefore no need to connect an IoT device to an external screen.


[Update 03/16/2018]

If you have a newer version of Raspbia Jessie or above, You can remote desktop without the need to install XRDP since Raspbian now has built in "VNC Server" which we can use to remote desktop to any Raspberry PI. To enable remote desktop, follow these steps:

1) From the terminal window, execute the following:

sudo rasp-config

2) This will open up a graphical interface, select "Interfacing Options".
3) Then select "VNC Server" to enable VNC server.
4) This will install all VNC components on the PI, after about 30 seconds, you will be prompted that the VNC server is enable.
5) Hit enter to hit Ok button.
6) Scroll down to Finish and hit enter.
7) Reboot the PI.
8) Download VNC Client on your machine (Windows, Mac or Linux) from here.
9) Open VNC client viewer, type the hostname or the ip of your pi.
10) You are in!

Enjoy!

Linear Regression Algorthims in Scikit-Learn

Hi,

While i am working on different regression algorithms in scikit-learn library. I would like to share some important tips to differentiate between major linear regression algorithms in Machine Learning space.

Below is a comparison table to compare among four linear regression algorithms:

The general idea of Gradient Descent (GD) is to tweak parameters iteratively in order to minimize a cost function.

Batch and Stochastic Gradient Descent: at each step, both algorithms compute the gradients based on the full training dataset (as in Batch GD) or based on just one instance (as in Stochastic GD).

While in Mini-Batch Gradient Descent algorithm: computes the gradients based on small random sets of instances called mini batches.


There are more linear regression algorithms in sklearn that is not covered in this blog post, you can find it here:  http://scikit-learn.org/stable/modules/sgd.html#regression


Hope this helps!

What is the difference between estimators vs transformers vs predictors in sklearn?

Hi All,

While working in Machine Learning projects using scikit-learn library, I would like to highlight important and fundamental concepts that every ML ninja needs to be aware of. In this post i am highlighting few concepts to differentiate estimators vs transformers vs predictors in building machine learning solutions using sklearn.


1) Estimators: Any objects that can estimate some parameters based on a dataset is called an estimator. The estimation itself is performed by calling fit() method.
This method takes one parameter (or two in case of supervised learning algorithms). Any other parameter needed to guide the estimation process is called hyperparameter and must be set as in instance variable.

For example: i would like to estimate a mean, median or most frequent value of a column in my dataset.


This is a cheat sheet of sklearn estimators. you can find the up to date version here.

2) Transformers: Transform a dataset. It transforms a dataset by calling transform() method and it returns a transformed dataset. some estimators can also transform a dataset.

For example: Imputer class in sklearn is an estimator and a transformer. You can call fit_transform() method that estimate and transform a dataset.

Python code: 

from sklearn.preprocessing inport Imputer

imputer = Imputer(strategy="mean") #estimate mean value for dataset columns

imputer.fit(mydataset)    # Imputer as an estimator

imputer.fit_transform(mydataset)   # Imputer as a transformer and estimator (Combined two steps)




3) Predictors: making predictions for  given a dataset. A predictor class has predict() method that takes a new instances of a dataset and returns a dataset with corresponding predictions. Also, it contains score() method that measures the quality of the predictions for a giving test dataset.

For example: LinearRegression, SVM, Decision Tree,..etc are predictors.


You can combine building blocks of estimators, transformers and predictors as a pipeline in sklearn. This allows developers to use multiple estimators from a sequence of transformers followed by a final estimator or predictor. This concept is called composition in Machine Learning.


Hope this helps

How to configure X2GO Client on Data Science Virtual Machine

Hi,

While i was trying to connect to a newly provisioned data science virtual machine in Azure. I have received few challenges on successfully start a session in X2GO client app.

The Data Science Virtual Machine (DSVM) VM image makes it easy to get started doing data science in minutes, without having to install and configure each of the tools individually.
This virtual machine contains: Cent OS, Microsoft R Developer edition, Anaconda python distribution, Standalone spark, CNTK, Rattle, XGBoost, in addition to other tools. Check out this article for the full details of this VM.

X2GO client provides a client tool for windows users to RDP to linux VMs, you can install this tool from here.

After you install this tool and try to connect to the DSVM VM, you will get this error:

unable to start startkde

To solve this problem, follow these steps:

1) Connect to the VM using any client linux tool to such as Putty.
2) After you login to the VM, execute the following command:

sudo yum install @kde 

This will install and upgrade existing packages on the VM. The VM will prompt you to accept installing all required packages and upgrades.

3) This command will take few seconds to complete. Below screenshot upon completion step is finished.

4) Return back to X2GO client and login using your username and password.

5) You will be able to successfully RDP to DSVM machine.

Hope this helps!

How to install Keras on Windows 10 with 64 bit

Hi,

I was trying to install Keras library on Windows 10 with 64 bit build machine. Since i use Anaconda to manage python packages on my machine, The first thing i tried was to install the package from the Anaconda command line by executing the following command:

conda install keras

I got the following error:

PackageNotFoundError: Package missing in current win-64 bit channels:
 - keras


To fix this issue, Follow these steps:

1) Check the latest Keras package from Anaconda website by visiting this link:
https://anaconda.org/search?q=keras

2) Select Keras library from the list, then copy the displayed command from the website:

conda install -c conda-forge keras=2.0.2

3) Run this command in the Anaconda command prompt window.

4) Keras library is installed and you will be able to start deep learning with Keras!

Enjoy!

How to install and run Jupyter from your local computer for python development

Hi,

If you are planning to program in Python from your local computer, the best development environment to code, instruct and visualize data is using Jupyter notebook.

I really like working with Jupyter notebook (aka IPython Notebook) for coding in Python, R programs.

As a lot of us download and look at ipynb files to use it in our applications. Instead of copy and paste code into Python console window, Jupyter notebook provides more interactive way to write code in Python and tons of other languages.

If you got a punch of ipynb files and would like to install and start working with Jupyter, follow these below steps:

1) Open command prompt window, write below command:

pip install jupyter notebook

2) After this installation is complete, navigate to the folder where you have set of ipynb files.

3) Run Jupyter notebook by executing the following command in the ipynb files folder:

jupyter notebook

4) A new browser window will open where it has jupyter files to start viewing or creating new ipynb files. Jupyter usually run on port 8888. The url for jupyter notebook looks like: http://localhost:8888/


Enjoy!

How to set storage account connection string in Azure Functions

Hi,

I was developing an Azure Function App that connects to an Azure blob storage. After setting up the binding for my blob storage account. I got the following error message when running my function app:

The error message in the screen shot above suggests three options to fix this. I will walk through how to implement the first option as one of the available solutions. The first solution is to set the connection string name in the appsettings.json file so it will look like this.

appsettings.json
{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "",
    "AzureWebJobsDashboard": "",
    "AzureWebJobsmofunctions_STORAGE": "DefaultEndpointsProtocol=https;AccountName=mofunctions;AccountKey=KEY"
  }
}

function.json (Just a section where i define my blob binding info)

 {
      "type": "blob",
      "name": "iBlob",
      "path": "mydata/file1.csv",
      "connection": "mofunctions_STORAGE",
      "direction": "in"
    }

NOTE: 
You will notice that the connection name value in function.json is a suffix for AzureWebJobs key in the appsettings.json file.


Once you set this, Press F5 and you will be able to connect and read blob contents from Azure storage accounts.



Enjoy!

Mashing RDDs in Apache Spark from RDBMs perspective

Hi,

Happy new year! This is my first post in 2017!. 2016 was amazing year for me. lots of work, projects and achievements. Looking forward to 2017.

I am writing this blog post to cover the standard techniques to work with Resilient Distributed Datasets (RDDs) to join data in Apache Spark.


I would like to share some insights when working with RDDs in Spark. That's related to how to work with multiple RDDs as we do when working with relational database management systems.

Apache Spark support joins in RDDs, where you can implement all kinds of joins that we are aware of in RDBMS. Below i will list how would you implement this on this platform.

Apache Spark Join Transformations Operations:

1) join: This is equivalent to inner join in RDBMs. It returns a new pair RDD with the elements containing all possible pairs of values from the first and second RDDs that has the same keys. For the keys that exist in only one of the the two RDDs. the resulting RDD will have no elements.

2) leftOuterJoin: This is equivalent to left outer join in RDBMs. The resulting RDD will also contain the elements for those keys that don't exist in the second RDD.

3) rightOuterJoin: This is equivalent to right outer join in RDBMs. The resulting RDD will also contain the elements for those keys that don't exist in the first RDD.

4) fullOuterJoin: This is equivalent to cross join in RDBMs. The resulting RDD will also contain the elements for both keys that exist in either RDDs.

In case of the RDDs contain duplicate keys, these keys will be joined multiple times.


Hope this helps!

Introducing Power BI Embedded talk at Cloud Summit

Hi All,

Earlier today, I have presented "Introducing Power BI Embedded" top that covers platform capabilities and tools in Cloud Summit event at Microsoft Chevy Chase office.

The session covered Power BI Platform capabilities, tools and Power BI Embedded as PaaS option in Microsoft Cloud platform.

I have got a lot of questions about Power BI data set scheduling, working with data capabilities including direct queries vs. import options while authoring reports in Power BI desktop. I also covered the need for Power BI Gateway for hybrid scenarios.

Introducing Power BI Embedded from Mostafa Elzoghbi

Enjoy.

Fixing powerbi.d.ts missing modules errors in Visual Studio 2015

Hi,

While i was working on Visual Studio 2015; I have got this errors due to missing modules in powerbi.d.ts file. I have an ASP.NET MVC project that uses Power BI Embedded and i would like to get this application up and running but i am getting these errors while building my app.

These errors are due to missing TypeScript tools for Visual Studio 2015. Once you install them, you will be able to run your app and all these errors disappear.


To fix this problem, follow these steps:

• Open Tools | Extensions and Updates.
• Select Online in the tree on the left.
• Search for TypeScript using the search box in the upper right.
• Select the most current available TypeScript version.
• Download and install the package.
• Build your project!

Hope this helps.

Extending Product Outreach with Outlook Connectors

Hi All,

I presented last Saturday at SharePoint Detroit a talk with title "Extending Product Outreach with Outlook Connectors"; Since i covered how to utilize office 365 groups to extend product outreach using outlook group connectors with demos.

Session Description:

Office 365 Connectors is a brand new experience that delivers relevant interactive content and updates from popular apps and services to Office 365 Groups. We are now bringing this experience to you, our Office 365 customers. Whether you are tracking a Twitter feed, managing a project with Trello or watching the latest news headlines with Bing—Office 365 Connectors surfaces all the information you care about in the Office 365 Groups shared inbox, so you can easily collaborate with others and interact with the updates as they happen. Session will cover how to build your office 365 connectors and how to work with Microsoft to help you build your company one.

Extending Product Outreach with Outlook Connectors from Mostafa Elzoghbi

Event Url: http://www.spsevents.org/city/Detroit/Detroit2016/home

Presentation demos are in GitHub:  https://github.com/melzoghbi/OutlookConnectorSDK

Enjoy!

Build Intelligent Microservices Solutions using Azure

Hi All,

I had the pleasure last night to present at one of our local user groups to talk about building intelligent microservices in Azure.

The session covers in detail how to build intelligent microservices solutions using Cloud Services including web and worker roles, Azure App Service features in Azure & Service Fabric. The session was a demo driven and i demonstrated how to design and provision complete end-to-end solutions using cloud services using web roles, worker roles and service bus in Azure.
I also covered Azure App Service capabilities that help developers to scale and monitor production applications; in addition to setup continuous deployment.

Session objectives and takeaways:

1. Benefits of creating micro services in the cloud
2. End-To-End Use case for building cloud service with web & worker roles with service bus integration
3. Azure App Service intelligent features including troubleshooting, CI, back up, routing, scheduling & other features
4. Azure Service Fabric microservices platform

The presentation is posted below.

Build intelligent solutions using Azure from Mostafa Elzoghbi

Case manager demo source code:
https://github.com/melzoghbi/AzureCloudServices

Enjoy!

Building Big Data Solutions in Azure Data Platform @ Data Science MD

Hi All,

Yesterday i was at Johns Hopkins University in Laurel, MD presenting how to build big data solutions in Azure. The presentation was focused on the underling technologies and tools that are needed to build end to end big data solutions in the cloud. I presented the capabilities that Azure offers out of the box in addition to cluster types and tiers that are available for ISVs and developers.

The session covers the following:

1) What HDInsight cluster offers in hadoop ecosystem technology stack.
2) HDInsight cluster tiers and types.
3) HDInsight developer tools in Visual Studio 2015, HDInsight developer tools.
4) Working with HBase databases and Hive View, deploying Hive apps from Visual Studio.
5) Building, Debugging and Deploying Storm Apps into Storm clusters.
6) Working with Spark clusters using Jupyter, PySpark.


Session Title: Building Big Data Solutions in Azure Data Platform

Session Details:

The session covers how to get started to build big data solutions in Azure. Azure provides different Hadoop clusters for Hadoop ecosystem. The session covers the basic understanding of HDInsight clusters including: Apache Hadoop HDFS, HBase, Storm and Spark. The session covers how to integrate with HDInsight in .NET using different Hadoop integration frameworks and libraries. The session is a jump start for engineers and DBAs with RDBMS experience who are looking for a jump start working and developing Hadoop solutions. The session is a demo driven and will cover the basics of Hadoop open source products.

Big data solutions in Azure from Mostafa Elzoghbi

meetup link: https://www.meetup.com/Data-Science-MD/events/232936519/

Thanks and Enjoy working with data!

Study notes for exam 70-475: Designing and Implementing Big Data Analytics Solutions

Hi All,

Today I passed the "Designing and Implementing Big Data Analytics Solutions" Microsoft exam.

I have been preparing for this exam (70-475) for a couple of months and I have been using Hadoop ecosystem tools and platforms for awhile.

I wanted to master building big data analytics solutions using HDInsight clusters using Hadoop ecosystem which contains: Storm, Spark, HBase, Hive and HDFS. I worked to cover any gap in understanding I had in Azure Data Lake, ML, Python & R programming and Azure Machine Learning.

This exam covers the following primarily four main technologies (from most covered to least):

1) Hadoop ecosystem: Working with HDFS, HBase, Hive, Storm, Spark and understanding Lambda Architecture. If you want to know more about Lambda Architecture, read my blog post explaining it here.

2) Azure Machine Learning: building/training models, predictive models, classification vs regression vs clustering, recommender algorithms. building custom models, Executing code in R and Python. Ingesting data from Azure Event Hub & transformation in Stream analytics.

3) Azure Data Lake: building pipeline, activities, linked services, move, transform and analyze data, working with storage options in Azure (blob vs block) & tools to transform data.

4) SQL Server and Azure SQL: Security in transit and at rest, SQL Data Warehouse. Working with R in Sql Server 2016/Azure SQL.


My study notes while preparing to pass this test:

1) To protect data at rest as well as querying in Azure SQL Database: Use "Always Encrypted" to make sure data in transit is encrypted. Use "Transparent Data Encryption" to make sure that data at rest is encrypted. Read more about TDE here. Read more about Always Encrypted feature here.

2) When running an Azure ML experiment and you are getting "Out of memory error" here is how to fix it:
   a) Increase the memory settings for the map and reduce operations in the import module.
   b) Use Hive query to limit the amount of data being processed in the import module.

3) The easiest way to manage Hadoop clusters in Azure is to assign every HDInsight cluster to a resource group and to apply tags to all related resources.

4) In Hadoop, When the data is row-based, self-describing with schema and provide compact binary data serialization: it is recommended to use Avro.

5) Which Hadoop cluster type for query and analysis batch jobs:
     a) Spark: A cluster for In-memory processing, interactive queries, and micro-batch stream processing.
     b) Storm: A real-time event processing.
     c) HBase: NoSQL data storage for big data systems.

6) Importing data using Pyhon in Azure ML tips:
    a) Missing values are converted into NA for processing. NA will be converted back to missing values when converted back to datasets.
    b) Azure Dataset are converted to data frames in Pandas. Pandas module is used to work with data in Python.
    c) Number names columns are not ignored. str() function is applied to those.
    d) Duplicate column names are not ignored. The duplicate column names are modified to make sure they have unique names.

7) The only platform that supports ACID transaction in Hadoop file storage options is Apache Orc.

8) You have three utilities you can use to move data from local storage to managed cluster blob storage. These tools are: Azure CLI, PowerShell & AzCopy.

9) How to improve Hive queries using static vs dynamic partitioning, read more here.

10) Understand when to use Filter based Feature Selection in Azure ML.

11) AzureML requires Python to store visualizations as PNG Files. To configure MatPlotLib in AzureML, you should configure it to use AGG backend for rendering and you should save charts as PNG files.

12) To detect potential SQL injection attempts on Azure SQL database in ADL cluster: Enable Threat Detection.

13) To create synthetic samples of dataset for classes that are under represented: use SMOTE module in AzureML.

14) D14 V2 Virtual Machines in Azure supports 100GB in memory processing.

15) You can add multiple contributors to AzureML workspace as users.

16) Understand the minimum requirements for each cluster type in HDInsight;
       a) At least 1 data node for Hadoop cluster type.
       b) At least 1 region server for HBase cluster type.
       c) Two Nimbus nodes for Storm cluster type.
       d) At least 1 worker role for Spark cluster type.

17) If you want to store a file with a file size is greater than 1 TB, you need to use Azure Data Lake Store.

18) In Azure Data Factory (ADF), you can train, score and publish experiments to AzureML using:
      a) AzureML Batch execution: to train and score.
      b) AzureML Update resource activity: to update AzureML web services.

19) In Azure Data Factory (ADF), A pipeline is used to configure several activities, including the sequence and timing activities in a pipeline can be managed as a unit.

20) Working with R models in SQL Server 2016/AzureSQL: read more here.

21) Apache Spark in HDInsight can read files from Azure blob storage (WASB) but not SQL Server.

22) Always Encrypted protects data in transit and at rest will be encrypted. Also this feature allows you to store encryption keys on premise.

23) Transparent Data encryption (TDE) : secure data at rest, it will not protect data in transit and the keys are stored in the cloud.


24) Distcp is a Hadoop tool to copy data to and from HDInsight clusters storage blob into Azure Data lake store.

25) Adlcopy: is a command line utility to copy data from azure blob storage into azure data lake storage account.

26) AzCopy: A tool to copy data from and to Azure blob storage.

27) While working with large binary files and you would like to optimize the speed of AzureML experiment, you can do the following:
      a) Developers should write data as block blob.
      b) The blob format should be in CSV or TSV.
      c) You should NOT turn off the cached results option.
      d) You can NOT filter data using SQL but R language.


28) SQL DB contributor role allows monitoring and auditing of SQL databases without granting permissions to modify security or audit policies.

29) To process data in HDInsight clusters in Azure Data Factory (ADF):
      a) Add a new item to the pipeline in the solution explorer.
      b) Select Hive Transformation.
      c) Construct JSON to process the cluster data in an activity.

30) Understanding Tumbling vs Hopping vs Sliding Windows in Azure Stream Analytics. (link)

Hope this helps you get ready to pass the test, and good luck everyone!
Let's get all certified ya'll data wranglers :-)


-- ME


References:
1) Microsot Exam 70-475 details, skills measured and more:
https://www.microsoft.com/en-us/learning/exam-70-475.aspx

GIT 101 in Visual Studio Team Services (VSTS)

Hi All,

I have been working with multiple developers on sharing project code using git. I found out that git is new to a lot of developers who have been using Visual Studio Team Foundation Server (TFS), Visual Studio Online (VSO aka VSTS now), or any other centralized source control system.

What is the difference between TFS/VSO/VSTS versus Git?

If you have been using TFS, VSTS or VSO, those all fall under Team foundation Version Control (TFVC) which is a centralized source control system.

While git, is a distributed source control system (DVCS). which means: you have a local and remote code repositories. you can commit your code to you local repo but not remote repo (unless you want to). Also, you can share your code to the remote repo so other team members can get these changes.
This is a fundamental concept to understand when working with git. git is distributed, contains local and remote repos & works offline and it is a great way to enable collaborations among developers.

**Popular git platforms: GitHub, VSTS, Bitbucket, GitLab, RhodeCode and others.

This article is focusing on managing multiple developers code working in a team & what is the git best practices around that, This also applies to any other git platform. For the sake of simplicity, This article will be focusing on using Git in VSTS.

Basic terminology and keywords to know when working with Git:

1) A branch: In Git, every developer should have his own branch. you write code and commit your changes into your local branch. To sync with other developers, get latest from the master branch and merge it into yours so you make sure everything is compiling & working before creating a new pull request to the master branch (by merging back your code into master).

2) Fetch: It download changes to your local branch from the remote branch. Fetch downloads these commits and adds them to the local repo without updating your local branch. To update your local branch either by executing merge or pull requests to your local repo to be up to date with its remote.

3) Pull: Get updates from your remote branch into your local branch. basically keeps your branch up to date with its remote one. Pull does a fetch and then a merge into your local branch.
So just use Pull to get your local branch updates from its remote one.

3) Pull vs Fetch: git pull does a git fetch. so if you used git pull this means that your have executed git fetch.  you can execute fetch if you want to get the updates but do not want to merge them into yor local branch yet.

5) Push: sends committed changes to remote branch so it is shared with others.


Basic rules to work with git in Visual Studio that everyone should be aware of before start coding:
This section i cover all needed actions to work with using git in Visual Studio Team Explorer window.

1) You need to click on Sync in team explorer to refresh the current branch from the remote branch. followed by pull to get those changes merged into the current local branch. sync just show status but to actually merge those changes you need to click on pull link.

2)  You need to click on Changes in team explorer every time you want to check in or get latest updates of the current branch.

3) You need to click on Branches in team explorer every time you want to manage branches in Visual Studio.

4) You need to click on Pull Requests in team explorer every time you want to manage pull requests in Visual Studio.


A) Setup a project for your team using Git in VSTS:

1) Visit http://visualstudio.com.
2) Login to your account.
3) Click on New button to create a new git project.



4) Once you hit create project button, the project will be created in few seconds and then we will use Visual Studio to do some necessary steps.

5) To start using Visual Studio with the created project, Click on Code tab for the newly created project "MicrosoftRocks".

6) Click on Clone in Visual Studio button.
7) This will open up VS and then open up Team Explorer window.
8) You need to click on "Clone this repository" this will allow you to create a local repo of the remote repo we have just created in VSTS.

9) Select a local repo folder and click on Clone.

10) Now, VS shows a message that we can create a new project or solution.

11) You can go ahead and create any project in VS, the only thing to notice to uncheck create a new Git repository checkbox when creating any new project since we have created already our local repo.

12) First things first, you need to exclude bin and debug folders from getting checked in Git. So, click on Settings in Team explorer -- > Click on Repository Settings link under Git --> Click on Add link to add .gitignore file.

13) To edit .gitignore file, click on edit link. Then, add the following at the bottom of the file:

# exclude bin and debug folders
bin
debug


14) Build the project and then we will do our first check in to the master branch.

15) Click on Home icon in team explorer to go back to the home page to manage source control options.

16)  Click on Settings, Type a check in message and then click on Commit Staged.

17) Commit staged action has check in all our changes to our local repo, these changes have not been share to the remote, so we need to click on sync to share it with others.
You will notice, that VS shows you a sync link afterwards so you can sync changes immediately or your can click on Sync from team explorer and then click on Push.

18) Now, the project is ready in the master branch for everyone with git ignore file before everyone will create his own branch and start developing.


B) Create your own branch in Visual Studio:

1) Every developer in a team, should create his own branch and get the latest from master to start developing in our project.

2) From Visual Studio, Click on master branch from the bottom bar and click on new branch.

3) Enter your branch name "dev1" and from which branch your want to create yours "master" and then click on Create Branch button. This step will create your own branch and get latest from master and switch to your branch to start coding in it.

4) You will notice, the name of the current branch has changed from master to dev1 in Visual Studio. now you can start working in your branch.

5) Once you are done coding a feature or at a good point to check in some code, Follow these steps to check in your changes:

• Click on Changes in team explorer, write a message and then click on Commit All button.
• You can also click on sync to push these changes to the remote branch in VSTS online.
• Remember, these changes are still in your branch no one else has seen it until you submit it to the master branch.

6) Publish your branch: It is important to publish your branch to VSTS, Follow these steps:

• From team explorer, click on branches.
• Right click on your branch.
• Click on Publish Branch.

C) How to submit your code to the master branch:

1) First, you need to make sure that your local master branch is up to date. to do that, switch to master branch and click on sync and then click on pull in Team explorer window.

2) Second, Switch back to your branch "dev1" and then click on branches in team explorer.

3) Click on Merge link.

4) Select to merge from "master" into "dev1" and then click on Merge. This step will merge all master changes into your branch so your branch will get other people work and fix any conflict (if any) before submitting all changes to master using Pull Request (PR) action.

5) Now, we need to submit all these changes after making sure there are no conflicts to the master branch. Click on Pull Requests in Team explorer.

6) Click on New Pull Request link.

7) This will open Visual Studio online webpage to submit new pull request.
8) Click on New Pull Request button.

9) Submitted Pull Requests (PRs) will be either approved or rejected by the repository admins. unless you are an admin, you will be able to approve/reject and complete submitted PRs in any project and therefore these changes are committed/merged to the master branch.


10) Click on Complete button to complete the pull request. Visual Studio will prompt a popup window if you want to add any notes and then click on Complete merge button. This is the last step to merge your changes into master after your PR has been approved.

11) Repeat the same steps every time you want to merge your changes into master using PRs.



Hope this article has shown in detailed walk-though how to work in a team using Git in Visual Studio Team Services and manage your code checkins/checkouts/merge/branching and PRs in Git.

Enjoy!

-- ME