How to convert epoch time to datetime in Pandas

Hi,

While i am working with iot data to transform UNIX epoch time in seconds. I would like to convert epoch time in seconds into human readable date time and not a reference date which is based upon 1970.

I have my data in pandas dataframe, below screen shot shows "createdTime" column in epoch time in seconds:

Here is the code segment that convert UNIX epoch time into date time:

convert = lambda x: datetime.datetime.fromtimestamp(x / 1e3)
ds['ts'] = ds['createdTime'].apply(convert)
ds.head()


This code generates the expected output, below screen shot shows the output:

Hope this helps!

Enable Jupyter notebook in Anaconda Navigator

Hi,

After i installed the latest conda runtime (anaconda 3 x64 distro) that uses Anaconda 3 on Windows 64 bit.

When i try to click on a target environment, i see that "Open with IPython" or "Open with Jupyter Notebook"

The question is how to enable this? I found how to install Jupyter notebook package for conda environment where it would be accessible through the Navigator tool.

Follow below steps:

1) Select any of the available environment, Click on "Open terminal" window.
2) Type the following command:

conda install nb_conda

3) This will install notebook packages for Jupyter and once it is completed, The Jupyter notebook will be available for all environments.

4) To testify this work properly, Click on Jupyter link and this will open up the notebook.

5) Write some code to make sure this works with no issues.

import pandas as pd
s = pd.Series([1508258340299])
pd.to_datetime(s)

The code executed with no issues:

Hope this helps!

Load datasets from azure blob storage into Pandas dataframe

Hi,

In this post, I am sharing how to work and load data sets that are stored in Azure blob storage into Pandas data frame.

I have the full code posted in Azure notebooks. This code snippet is useful to use in any Jupyter notebook while working on your data pipeline while developing Machine Learning models.

I have exported a data set into a csv file and stored it into an Azure blob storage so i can use it into my notebooks.

Python code snippet:

import pandas as pd
import time
# import azure sdk packages
from azure.storage.blob import BlobService

def readBlobIntoDF(storageAccountName, storageAccountKey, containerName, blobName, localFileName):    
    # get an instance of blob service 
    blob_service = BlobService(account_name=storageAccountName, account_key= storageAccountKey)
    # save file content into local file name
    blob_service.get_blob_to_path(CONTAINERNAME,blobName,localFileName)
    # load local csv file into a dataframe    
    dataframe_blobdata = pd.read_csv(localFileName, header=0)
    
    return dataframe_blobdata

STORAGEACCOUNTNAME= 'STORAGE_ACCOUNT_NAME'
STORAGEACCOUNTKEY= 'STORAGE_KEY'    
CONTAINERNAME= 'CONTAINER_NAME'
BLOBNAME= 'BLOB_NAME.csv'
LOCALFILENAME = 'FILE_NAME-csv-local'

# load blob file into pandas dataframe
tmp = readBlobIntoDF(STORAGEACCOUNTNAME,STORAGEACCOUNTKEY,CONTAINERNAME,BLOBNAME, LOCALFILENAME)
tmp.head()

The full code snippet is posted in Azure Notebook here.

Enjoy!

What is the difference between estimators vs transformers vs predictors in sklearn?

Hi All,

While working in Machine Learning projects using scikit-learn library, I would like to highlight important and fundamental concepts that every ML ninja needs to be aware of. In this post i am highlighting few concepts to differentiate estimators vs transformers vs predictors in building machine learning solutions using sklearn.


1) Estimators: Any objects that can estimate some parameters based on a dataset is called an estimator. The estimation itself is performed by calling fit() method.
This method takes one parameter (or two in case of supervised learning algorithms). Any other parameter needed to guide the estimation process is called hyperparameter and must be set as in instance variable.

For example: i would like to estimate a mean, median or most frequent value of a column in my dataset.


This is a cheat sheet of sklearn estimators. you can find the up to date version here.

2) Transformers: Transform a dataset. It transforms a dataset by calling transform() method and it returns a transformed dataset. some estimators can also transform a dataset.

For example: Imputer class in sklearn is an estimator and a transformer. You can call fit_transform() method that estimate and transform a dataset.

Python code: 

from sklearn.preprocessing inport Imputer

imputer = Imputer(strategy="mean") #estimate mean value for dataset columns

imputer.fit(mydataset)    # Imputer as an estimator

imputer.fit_transform(mydataset)   # Imputer as a transformer and estimator (Combined two steps)




3) Predictors: making predictions for  given a dataset. A predictor class has predict() method that takes a new instances of a dataset and returns a dataset with corresponding predictions. Also, it contains score() method that measures the quality of the predictions for a giving test dataset.

For example: LinearRegression, SVM, Decision Tree,..etc are predictors.


You can combine building blocks of estimators, transformers and predictors as a pipeline in sklearn. This allows developers to use multiple estimators from a sequence of transformers followed by a final estimator or predictor. This concept is called composition in Machine Learning.


Hope this helps

How to install and run Jupyter from your local computer for python development

Hi,

If you are planning to program in Python from your local computer, the best development environment to code, instruct and visualize data is using Jupyter notebook.

I really like working with Jupyter notebook (aka IPython Notebook) for coding in Python, R programs.

As a lot of us download and look at ipynb files to use it in our applications. Instead of copy and paste code into Python console window, Jupyter notebook provides more interactive way to write code in Python and tons of other languages.

If you got a punch of ipynb files and would like to install and start working with Jupyter, follow these below steps:

1) Open command prompt window, write below command:

pip install jupyter notebook

2) After this installation is complete, navigate to the folder where you have set of ipynb files.

3) Run Jupyter notebook by executing the following command in the ipynb files folder:

jupyter notebook

4) A new browser window will open where it has jupyter files to start viewing or creating new ipynb files. Jupyter usually run on port 8888. The url for jupyter notebook looks like: http://localhost:8888/


Enjoy!

Building Big Data Solutions in Azure Data Platform @ Data Science MD

Hi All,

Yesterday i was at Johns Hopkins University in Laurel, MD presenting how to build big data solutions in Azure. The presentation was focused on the underling technologies and tools that are needed to build end to end big data solutions in the cloud. I presented the capabilities that Azure offers out of the box in addition to cluster types and tiers that are available for ISVs and developers.

The session covers the following:

1) What HDInsight cluster offers in hadoop ecosystem technology stack.
2) HDInsight cluster tiers and types.
3) HDInsight developer tools in Visual Studio 2015, HDInsight developer tools.
4) Working with HBase databases and Hive View, deploying Hive apps from Visual Studio.
5) Building, Debugging and Deploying Storm Apps into Storm clusters.
6) Working with Spark clusters using Jupyter, PySpark.


Session Title: Building Big Data Solutions in Azure Data Platform

Session Details:

The session covers how to get started to build big data solutions in Azure. Azure provides different Hadoop clusters for Hadoop ecosystem. The session covers the basic understanding of HDInsight clusters including: Apache Hadoop HDFS, HBase, Storm and Spark. The session covers how to integrate with HDInsight in .NET using different Hadoop integration frameworks and libraries. The session is a jump start for engineers and DBAs with RDBMS experience who are looking for a jump start working and developing Hadoop solutions. The session is a demo driven and will cover the basics of Hadoop open source products.

Big data solutions in Azure from Mostafa Elzoghbi

meetup link: https://www.meetup.com/Data-Science-MD/events/232936519/

Thanks and Enjoy working with data!

Python for .Net Developers

Hi All,

I have been working with Python recently and i would like to share a quick and easy tutorial to learn Python for developers with OOP experience such as: C#, JAVA or .Net experience.

This is a mini course, gets you up to speed to start developing in Python with no prior knowledge because it covers what every developers wants to know for Python programming language basics.

http://ai.berkeley.edu/tutorial.html#PythonBasics

Here is my takeaways of this tutorial:

1) Python uses indentation for code execution, Python doesn't use curly brackets for opening and closing functions, classes..etc as in C# or Java. So it is preferred to use tabs than spaces while coding.

2) You can use lists ([]), dictionaries ({key:value}), tuples (()), and sets (set([list])) for storing collections in code. Use an appropriate option as in your code case. Sets items has no duplicates. Tuples are immutable (Once it is created, you can't change it).

3) You can create main function as we do in Java or C# console apps as an entry point for your program.

# Main Function
if __name__ == '__main__':        
    // YOUR CODE HERE  

4) You can include other files in your python file by adding import statement: import myotherfile, note do not include .py in the name of your python file.

5) You can define classes and functions in Python. also, you can create instance and static variables for class members.

class Person:
    population = 0
    def __init__(self, myAge):
        self.age = myAge        --> Instance Variable
        Person.population += 1  --> Static Variable 

6) You can use Visual Studio Code as a development IDE to code in Python. Download & install VS Code for free and then install Python & Python VS Code extension. Here is the steps:

a) Download Visual Studio Code for free: https://code.visualstudio.com/Download
b) Install Python on your machine: https://www.python.org/downloads/
c) Open VS Code, Press F1 and then type install extension and hit enter and then type: python. VS Code provides intellisense & debugging capabilities for Python.

Enjoy!


More useful links:

1) Python Tutorial:
https://docs.python.org/2/tutorial/