Learn R for Data Science Applications and Machine Learning

Hi All,

I am writing this blog post to highlight cool resources to learn programming in R. R language is a widely used programming language for data scientists and engineers who build programmable components in big data solutions.

Here are the resources to get started:

1) Quick-R: the easiest and fastest way to learn R
http://www.statmethods.net/ 

2) Official CRAN (Comprehensive R Archive Network) resource:
https://cran.r-project.org/manuals.html 

3) Introduction to R Programming: Free Course (On-Demand)
https://www.edx.org/course/introduction-r-programming-microsoft-dat204x-0 


Hope this helps.

Apache Spark for developers - part 1

In this blog post i am sharing basic understanding concepts of Apache Spark for developers. My target is to educate developers/engineers with no big data experience on what is Apache Spark and the importance of this platform in working with Hadoop big data solutions.

The target developers should have minimum experience in building  business applications or products (desktop, web or mobile) using any OOB language such as: C++. Java or C# developers.

What is Apache Spark? 
Apache Spark is a distributed computation framework for big data. It is an open source platform for processing large amount of data in Hadoop ecosystem solutions.

Because it is a distributed platform, there are important concepts to solidify such as:

1) Any spark application contains a driver program which is the main entry point for the application that executes the main function and executes various parallel operations in a cluster.

2) Spark provides a resilient distributed dataset (RDD) which is a collection of data elements that are partitioned across different nodes in a cluster that can be operated on in parallel.

3) We can persist RDD in memory to allow it to be reused efficiently across parallel operations.

4) RDDs can automatically recover from nodes failures.


Tip: To start working with RDDs in Spark, RDD starts with a file in HDFS or any Hadoop supported file system.

5) Spark supports shared variables in parallel operations. There are two types of shared variables in Spark, the first is broadcast variables and the second is accumulators.


You can write Spark applications in Scala, Python and Java programming languages.

To connect with Spark, you need to have a Spark context object which requires a Spark configuration object. The Spark configuration object contains information about your application.


Spark contains Scala and Python shells where you can write and execute your code against Spark cluster nodes.

More to come in the following posts...

- ME

Data Science and Machine Learning Training Course for Free

Hi All,

To get started to learn data science and Machine Learning principals i would strongly recommend to try this free course from Microsoft Virtual Academy (MVA) course:

Some key points from the course:

• Data science is about using data to make decisions that drive actions.

• Data science involves:

1. Finding data

1. Acquiring data

1. Cleaning and transforming data

1. Understanding relationships in data

1. Delivering value from data

• Predictive analytics is about using past data to predict future values.

• Prescriptive analytics is about using those predictions to drive decisions.

Learn more in this course:

https://mva.microsoft.com/en-US/training-courses/data-science-and-machine-learning-essentials-14100?l=UyhoTxWdB_3505050723

Enjoy! 

Get Started with Apache Spark Resources in HDInsight

Hi All,

I am writing this blog post to share some important Apache Spark framework for starters topics and foundation understanding points..

Apache Spark is an open-source parallel processing framework that supports in-memory processing to boost the performance of big-data analytic applications. 

Spark processing engine is built for speed, ease of use, and sophisticated analytics. Spark's in-memory computation capabilities make it a good choice for iterative algorithms in machine learning and graph computations. 

You can write applications in Python, Scala and R in Spark clusters. HDInsight contains out of the box notebooks (tools/Dev IDEs) that allows data scientist to write programs in Spark using:

1) Python language using Jupyter , It also supports R, Julia, and Scala

2) Scala language using Zepplein 

Most important libraries for Apache Spark:

1) Spark Sql: A module for structured data manipulation using SQL or DataFrame API.

http://spark.apache.org/sql/

2) Spark Streaming: A module for building stream processing apps the same way your write batch jobs. It supports Java, Scala and Python.

http://spark.apache.org/streaming/

3) MLLib: A library for machine learning

http://spark.apache.org/mllib/

Some useful links and resources for Apache Spark:

1) Apache Spark homepage:

http://spark.apache.org/ 

2) Learning Python:

https://www.python.org/doc/

3) Learning Scala:

http://www.scala-lang.org/documentation/

HDInsight Apache Spark provides tons of tools out of the box, check out this link to see why would use HDInsight Apache Spark in Azure:

https://azure.microsoft.com/en-us/documentation/articles/hdinsight-apache-spark-overview/

Hope this helps! Enjoy crushing streams of data...

The best way to test website compatibility issues

Hi All,

I had a chance to attend one of the New York city JavaScript events (JS Open NYC) that was hosted at Microsoft NYC office. I had the opportunity to talk and chat with dozens of open source front end developers about web compatibility and interoperability in modern web design and development.

While i was talking with the event attendees, I introduced new scanning tools that every PM, BA or front end engineer could use to test website compatibility issues.

Microsoft developed tools to scan and test your website for free, below is the website to check all the available tools.

Microsoft Edge Dev Center homepage:  
https://dev.windows.com/en-us/microsoft-edge/

In this website, you have four tools to use. I will go through each one of them in this blog post.

1) Quick Scan tool: (my favorite one for technical analysis)
The best tool to do quick scan to your website, it points out all out of date libraries, layouts and things to change in your website to be compatible with most of modern browsers.

Url: https://dev.windows.com/en-us/microsoft-edge/tools/staticscan/ 

This tool is open source, here is the source code repo in GitHub:
https://github.com/MicrosoftEdge/static-code-scan


2) Browser Screenshots tool: (my favorite one for UX)
This tool shows you how your website looks like in all browser! very handy and useful.

Url: https://dev.windows.com/en-us/microsoft-edge/tools/screenshots/ 

3) Virtual Machine (VM):
You can download various VMs that have all IE version with different OS options including Linux and Mac. This is a useful tool for internal and intranet sites.

Url: https://dev.windows.com/en-us/microsoft-edge/tools/vms/

4) Remote App:
A way to test your site using Remote App session on any OS you use.

Url: https://dev.windows.com/en-us/microsoft-edge/tools/remote/


I also had a discussion about Chakra (The Core Engine for MS Edge). Since Microsoft announce that the core Edge engine is open source (ChakraCore) and it will be available in GitHub on Jan 2016. Check out the official announcement from Microsoft Edge Dev Team:

https://blogs.windows.com/msedgedev/2015/12/05/open-source-chakra-core/


Hope this helps!

Get Started with Apache Storm Resources in HDInsight

Hi All,

I'd like to share some useful resource to get started with Apache Storm in this blog post.
Apache storm is a distributed real-time computational system that allows engineers to process streams of data at scale.

Apache storm is one of the major hadoop ecosystem components, where engineers use it to process the sources of data into hadoop ecosystem.

Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate.

Imagine you need to process endless source of data (such as: Facebook news feed or Twitter feed) and you are going to process this large volume of  information and then store in Hadoop. In this case, you want to build a storm application specifying the topology by defining the sources of information (Spouts)  and how to process this chunk of data (Bolts).

Every Storm application contains a topology, Set of spouts and bolts in addition to a specification file for the topology.

I compiled some useful resources to get started and work with Apache storm:

1) Apache Storm main website:
http://storm.apache.org/index.html 

2) HDInsight Hadoop documentation in Azure:
https://azure.microsoft.com/en-us/documentation/services/hdinsight/ 

3) SCP.NET, Get started with building .NET apps in C# in Storm:
https://github.com/hdinsight/hdinsight-storm-examples/blob/master/SCPNet-GettingStarted.md

4) Power of Storm with examples:
https://github.com/hdinsight/hdinsight-storm-examples/blob/master/README.md

5) EDX Free online course (Implementing Real-Time Analytics with Hadoop in Azure HDInsight) :
https://www.edx.org/course/implementing-real-time-analytics-hadoop-microsoft-dat202-2x

.
Hope this helps.

Working with HBase in Azure

I have published a video on how to work with HBase tables in HDInsight HBase cluster. The video is a walk-through on the basics of CRUD operations in HBase.

The video covers the following topics:
1) How to connect to hbase shell tool.
2) How to create tables in HBase.
3) How to select, insert, update, records in HBase.
4) Understanding create, put, delete, deleteall commands in HBase.

The video is giving a basic "Order" table structure as an example and execute all the above operations to it.



Video Link:

https://channel9.msdn.com/Blogs/MostafaElzoghbi/Working-with-HBase-in-Azure


Enjoy!

HBase introduction in Azure

Hi,

I have published a new channel9 video about HBase Introduction in Azure. 

This video covers an introduction to HBase in Azure. It covers what is HDInsight clusters, What are the available cluster types. What Microsoft Azure offers as Hadoop ecosystem components. The video focuses on HDInsight HBase cluster type and the need for HBase in Hadoop ecosystem to store NoSQL data and the available tools (such as: hbase shell) and commands to use to manipulate data within HBase tables.

The video covers the column families concept for engineers who come from RDBMS background.

This video helps any engineer with no Hadoop experience to understand what is the role of HBase in Hadoop and big data applications.

Channel 9 video url:
https://channel9.msdn.com/Blogs/MostafaElzoghbi/HBase-Introduction-in-Azure

Enjoy!

How to migrate console applications to the cloud using Azure WebJobs

Hi All,

This post is to answer a question i frequently receive from developers and architect on what is the best way to take legacy console applications (exe) to the cloud.

I have creates a video that answers the address the following issues & concerns:

1) The challenges of scaling console applications.
2) The available options to migrate console applications to the cloud.
3) How Azure webjobs is a great option for migration light weight backend jobs to the cloud.
4) Scaling up or down web jobs in Azure.
5) Scheduling options for Azure web jobs.

Enjoy watching the video and keep the feedback coming.



Channel 9 Video Url:

https://channel9.msdn.com/Blogs/MostafaElzoghbi/How-to-migrate-Console-Apps-as-a-cloud-service

thread 1: signal sigabrt error when running ios app

Hi,

While i was developing an iOS app using Xcode, After i designed my page by adding few controls to my view. I was getting the following error when i run the app in the simulator:

thread 1: signal sigabrt error when running iOS app


My app didn't have code at all, I just added controls and linked these controls in the ViewController.swift file.

This is how i fixed it:

1) Open your storyboard file.
2) For each control in your view, Open "Connection Inspector" menu and delete all related connections.

3) Clean the project, From the the product menu --> click on Clean.
4) Run your app.


Hope this helps.

How to create Outlook apps in Visual Studio 2015

Hi All,

I have been asked on what is needed to create outlook extensions, add-ins or apps. The new standard name from Microsoft  for these extensions is office apps. So in this blog post i will use apps which refer to add-ins or extensions.


With the new app model, All office apps are HTML5 & web based projects. When you create a project in Visual Studio 2015, VS creates two projects in the solution.

The first project is a web app project where you can create all your HTML pages with all required assets (CSS, JS...etc). The second project is an office wrapper project for your web app.  This project is being used to register your app in office client apps or office 365.

The office wrapper project contains information such as: application name, version, company information, end point urls and any required permissions that you need to configure when a user install the Outolook add-in.

Below screen shots shows how to create an Outlook Add-In to show in outlook email messages.

1) Open Visual Studio 2015.
2) From the file menu, click on New project.
3) Under Office/SharePoint, Select Apps from the right pane.

4) Click OK.
5) Visual Studio will prompt which project type you want for your outlook app.

6) Select Mail, Then click Next.
7) Choose where you want the app to appear.

8) Select Read and Compose Email Message, Un-check the rest.
9) Click on Finish.

Below shows how the two projects that have been created in VS 2015.

Enjoy!

How to execute on your idea

Hi All,

I'd like to share a presentation on how to start executing your idea. This presentation i prepare for startup office hours that i hold on monthly basis.

I have put together this content to answer common questions, challenges and things to expect when you start executing on your startup idea.

This presentation is very useful for entrepreneurs and first time founders on how to get started on executing an idea.

Start executing your idea from Mostafa Elzoghbi

How to open Cordova projects using Xcode

Hi All,

I have created a Cordova Project using CLI commands, and then i was trying to figure out how to open a Cordova project using Xcode to build and run my Cordova projects in the available emulators.

Follow the following steps:

1) Open Xcode.
2) From File menu -> Click on Open.
3) Navigate to the Cordova App folder.
4) Navigate inside platforms folder, and then select ios folder.

5) The project will be loaded inside Xcode.
6) Click on Run and the Cordova app will run in an iPhone emulator.

You will notice that Xcode created (*.xcodeproj) file that you can use to open cordova projects in Xcode.


Hope this helps.

npm ERR! while installing Ionic on OSX

Hi All,

While I was trying to install Ionic framework to start develop cross mobile apps using Ionic, I was getting errors while running the following command:

$ npm install -g cordova ionic

I have El Capitan version 10.11.1 with Node.js version 5.0.0 and npm version 3.3.6.

I tried to uninstall cordova first, then re-install cordova. After that, I tried to install ionic by itself but i was getting the following errors in the terminal window:

npm ERR! Darwin 15.0.0
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "ionic"
npm ERR! node v5.0.0
npm ERR! npm  v3.3.6
npm ERR! code ECONNRESET
npm ERR! errno ECONNRESET
npm ERR! syscall read

I tried to read more if i am missing any requirements for Cordova CLI in OSX. I found out that XCode contains all OSX SDK tools and dependencies for Cordova apps development in OSX.

Cordova CLI Requirements url for OSX: 
http://cordova.apache.org/docs/en/5.1.1/guide/platforms/ios/index.html


So i have decided to install XCode first, and the i re-ran the installation command for Ionic and i was able to create Ionic apps on my OSX dev machine.

So installing XCode on my machine did the trick!

Hope this helps.

Thanks a lot.

How to enable Ionic Cordova Project template to run Windows 10 mobile apps

Hi,

If you are using Ionic Framework project template in Visual Studio 2015, You will not be able to develop and build windows 10 apps in Visual Studio 2015 until you do some changes in your project to support that.

I have posted and answered this issue here:

https://social.msdn.microsoft.com/Forums/en-US/b6f638f9-27ee-4918-ae74-dfe4815a070d/how-to-enable-windows-10-target-in-ionic-cordova-project-templates?forum=ToolsForApacheCordova

Hope this helps.

Ionic framework must to know tips

Hi All,

I am writing this blog post to cover essential Ionic framework commands that every developer should be aware of specially if you are new to Ionic Development using Cordova platform.

Here is my tips that must to know when working with Ionic framework:

1) Ionic is a front-end SDK that is used to create incredible mobile apps based on Cordova framework.

2) To start using Ionic to create mobile apps, Ionic uses npm which uses node.js. So you need to install Node.js to start using npm commands.

3) To install Node.js, here is the installation link: https://nodejs.org/en/

4) Creating Ionic mobile apps project is a folder based, So before executing the following command you need to make sure that you are in the right directory path structure where Ionic framework creates a folder for the newly created application "app1" :

     ionic start app1

The above command creates a folder named "app1" under the current directory path.

5) All Ionic Framework commands starts with ionic either to create, build or run Ionic mobile apps.

6) You can open Ionic projects using Visual Studio Code by executing the following steps:
     a) From File menu, Click on Open Folder.
     b) Select your Ionic project folder and click on Ok.
You will be able to use Visual Studio as your Development Environment.

7) If you want to run your Cordova project into the browser, Follow these steps:
    a) Open the terminal window, navigate to the project folder.
    b) Type the following:
         ionic serve
    c) A new browser will run for your Ionic Cordova project.

8) How to add Git integration to an existing project in VS Code, check out this reference:
https://help.github.com/articles/adding-an-existing-project-to-github-using-the-command-line/

Update [11/2/2015]: Updates for Git integration and how to run Cordova apps in the browser.

Hope this helps.

References:
1) Ionic Framework website: http://ionicframework.com/

DC Startups join Microsoft Ventures - Join our Equity free program $25k & $500k Azure Credits

Hi All,

I am calling all DC startups who are looking for getting mentorship, grow, get funded and get a great working environment and to join Microsoft Accelerator program.

Microsoft Ventures is a global initiative empowering entrepreneurs around the world on their journey to build great companies. We’re focused on building partnerships with venture funds, angel investors, incubators, accelerators, corporate executives, and at the core startups, as well as run its own accelerators around the world. Our goal and objective is to help scale and grow startups by giving them a great working environment, access to mentors and service providers, free technological tools and connecting them to Microsoft’s customers and business partners. 

The next batch of the program opens on Feb 2016 and it’s going to be themed around Machine Learning and Data Science. We will be targeting startups that are a more mature than previous batches – such that raised $1M-$1.5M, with an existing product and maybe even initial traction.

Having said that, startups that choose to use Azure will enjoy the help of the ML product group in architecting their solutions to the right size and using the cutting edge services Azure ML offers. 

• Access to the Microsoft distribution channels – Microsoft Ventures is committed to help startups who have relevant products to our customers and partners get connected to them through our massive distribution channels. In addition, startups will receive assistance in getting set up on the Azure Marketplace allowing the Microsoft sales reps to sell their solutions with higher incentives. 

• Free of Microsoft Azure – startups in the program will be automatically enrolled to BizSpark Plus program and be credited for $500K of cloud usage (storage and compute) for 3 years, to be used in their own pace. This allows the startup to focus on the crucial business aspects of their company.

Meet us in person, learn more about the program and benefits. Besides these locations we are evaluating a few more venues so please check out for more details at microsoftventures.com/seattle

Meet us in person on November 4th, 2015 at 1776:

aka.ms/domoredata-dc

Let me know if you have any questions or would like to get an invite to attend.

Microsoft Azure Tour in Philly - Microsoft Azure Developer Immersion labs

Hi All,

Today we are holding first Azure tour event in Philly this year. Since we have Microsoft Azure Developer Immersion lab during the event and we have limited seating, you will be able to do all labs through the link below.

If you are not able to make it to the event, you will be able to do all labs from the below url:

http://rgroup.us


The link contains all labs, source code before and after the labs.

Enjoy!

Philly Code Camp Event Sessions

Hi All,

Today i am pleased to join the speakers team of Philly DotNet Code Camp (PhillyDotNet.Org) which we have it at Microsoft Technology Center at Malvern, PA on 9-10th Oct 2015.

I am delivering two sessions and below is the details of each session:

1) Developing Cross-Platform Mobile Apps using Apache Cordova

This session will cover how to build cross platform mobile applications using Apache Cordova. Cordova allows web developers to use their existing web development skills to build cross platform mobile apps. Write a single code base in HTML/JS/CSS and deploy your app to iOS, Android and Windows Phone. The session will cover how to use JS libraries such as Ionic, Angular and Backbone to build Apps in Visual Studio 2015 with latest TACO official release v1.0.0 on 10/7/2015.

The session will cover insights on understanding the project structure and best practices in building Cordova apps and it is demo driven!

Developing cross platform mobile apps using Apache Cordova from Mostafa Elzoghbi

Session url: https://phillydotnet.org/sessions/building-mobile-cross-platform-apps-using-cordova/



2) Get you site Edge Ready

Microsoft Edge – what are the technical details of Microsoft’s new browser, what’s new in Edge, and what’s the best way to develop for it? This session will go over the technical details of Microsoft Edge. We’ll discuss the value of the browser and how you can make sure your web sites are ready for Edge!

Get your site microsoft edge ready from Mostafa Elzoghbi

Session url: https://phillydotnet.org/sessions/get-your-site-microsoft-edge-ready/

Hope you enjoy my sessions, and have fun!

Secure sensitive data with SQL Server on Azure

Hi All,

SQL Server 2016 has released a new feature that is called "Dynamic Data Masking" which allows to mask sensitive information to be masked on the fly without a change to a line of code in your application. The video covers the DDM technology, the need for it and how to implement it on SQL DB in Azure. This feature can be implemented either using SQL PaaS (Azure SQL DBs) or IaaS (SQL Server VMs) on Azure.

If you have for example Personal Identification Information (PII), Credit Card Numbers, or Social Security Numbers that you would like to secure from non-authorized view from viewing restricted information to comply with your security policies. DDM is the right feature for you with zero change line of code.

SQL Database Dynamic Data Masking (DDM) limits sensitive data exposure by masking it to non-privileged users. Dynamic data masking is in preview for Basic, Standard, and Premium service tiers in the V12 version of Azure SQL Database.

Dynamic data masking helps prevent unauthorized access to sensitive data by enabling customers to designate how much of the sensitive data to reveal with minimal impact on the application layer. It's a policy-based security feature that hides the sensitive data in the result set of a query over designated database fields, while the data in the database is not changed.

In this video, I cover how to configure DDM feature in SQL Database on Azure using the new Azure portal. How to manage existing masks and how to test out this feature using SQL Server Management Studio (SSMS).

The video covers how to manage multiple users scenario to allow either masked or unmasked data on a SQL database in Azure.

Enjoy the video below.

Channel 9 video link


Enjoy!