Adding certs to java keytool

If you have troubles connecting to https site, the issue might be with certs. In order to test that, use this handy SSLPoke (https://gist.github.com/krinkere/8a4b526cf37a66261a7f560d81078cdb)

java SSLPoke server 443

you should get something like when connection is unsucessful
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

In order to install cert

openssl s_client -connect server:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/server.crt

The cert was saved into /tmp/server.crt. Now let's add it to the keystore of Java

/jre/bin/keytool -import -alias server -keystore /jre/lib/security/cacerts -file server.crt

See the list of certs: /jre/bin/keytool -list -v -keystore /jre/lib/security/cacerts

positive test cert / keytool:

java SSLPoke server 443

you should get this:

Successfully connected

How to use confusion matrix to calculate accuracy, precision, recall, and f1 score

Accuracy = (TP + TN) / (TP + TN + FP + FN)

Precision = TP / (TP + FP)

Recall = TP / (TP + FN)

F1 Score = 2 * Precision * Recall / (Precision + Recall)

TP = # True Positives, TN = # True Negatives, FP = # False Positives, FN = # False Negatives

An Intuitive Understanding of Word Embeddings: From Count Vectors to Word2Vec

Introduction

Before we start, have a look at the below examples.

• You open Google and search for a news article on the ongoing Champions trophy and get hundreds of search results in return about it.
• Nate silver analysed millions of tweets and correctly predicted the results of 49 out of 50 states in 2008 U.S Presidential Elections.
• You type a sentence in google translate in English and get an Equivalent Chinese conversion.

So what do the above examples have in common?

You possible guessed it right – TEXT processing. All the above three scenarios deal with humongous amount of text to perform different range of tasks like clustering in the google search example, classification in the second and Machine Translation in the third.

Humans can deal with text format quite intuitively but provided we have millions of documents being generated in a single day, we cannot have humans performing the above the three tasks. It is neither scalable nor effective.

So, how do we make computers of today perform clustering, classification etc on a text data since we know that they are generally inefficient at handling and processing strings or texts for any fruitful outputs?

Sure, a computer can match two strings and tell you whether they are same or not. But how do we make computers tell you about football or Ronaldo when you search for Messi? How do you make a computer understand that “Apple” in “Apple is a tasty fruit” is a fruit that can be eaten and not a company?

The answer to the above questions lie in creating a representation for words that capture their meanings, semantic relationships and the different types of contexts they are used in.

And all of these are implemented by using Word Embeddings or numerical representations of texts so that computers may handle them.

Below, we will see formally what are Word Embeddings and their different types and how we can actually implement them to perform the tasks like returning efficient Google search results.

What are Word Embeddings?

In very simplistic terms, Word Embeddings are the texts converted into numbers and there may be different numerical representations of the same text. But before we dive into the details of Word Embeddings, the following question should be asked – Why do we need Word Embeddings?

As it turns out, many Machine Learning algorithms and almost all Deep Learning Architectures are incapable of processing strings or plain text in their raw form. They require numbers as inputs to perform any sort of job, be it classification, regression etc. in broad terms. And with the huge amount of data that is present in the text format, it is imperative to extract knowledge out of it and build applications. Some real world applications of text applications are – sentiment analysis of reviews by Amazon etc., document or news classification or clustering by Google etc.

Let us now define Word Embeddings formally. A Word Embedding format generally tries to map a word using a dictionary to a vector. Let us break this sentence down into finer details to have a clear view.

Take a look at this example – sentence=” Word Embeddings are Word converted into numbers ”

A word in this sentence may be “Embeddings” or “numbers ” etc.

A dictionary may be the list of all unique words in the sentence. So, a dictionary may look like – [‘Word’,’Embeddings’,’are’,’Converted’,’into’,’numbers’]

A vector representation of a word may be a one-hot encoded vector where 1 stands for the position where the word exists and 0 everywhere else. The vector representation of “numbers” in this format according to the above dictionary is [0,0,0,0,0,1] and of converted is[0,0,0,1,0,0].

This is just a very simple method to represent a word in the vector form. Let us look at different types of Word Embeddings or Word Vectors and their advantages and disadvantages over the rest.

Different types of Word Embeddings

The different types of word embeddings can be broadly classified into two categories-

• Frequency based Embedding
• Prediction based Embedding

Let us try to understand each of these methods in detail.

1 Frequency based Embedding

There are generally three types of vectors that we encounter under this category.

• Count Vector
• TF-IDF Vector
• Co-Occurrence Vector

Let us look into each of these vectorization methods in detail.

1.1 Count Vector

Consider a Corpus C of D documents {d1,d2…..dD} and N unique tokens extracted out of the corpus C. The N tokens will form our dictionary and the size of the Count Vector matrix M will be given by D X N. Each row in the matrix M contains the frequency of tokens in document D(i).

Let us understand this using a simple example.

D1: He is a lazy boy. She is also lazy.

D2: Neeraj is a lazy person.

The dictionary created may be a list of unique tokens(words) in the corpus =[‘He’,’She’,’lazy’,’boy’,’Neeraj’,’person’]

Here, D=2, N=6

The count matrix M of size 2 X 6 will be represented as –

	He	She	lazy	boy	Neeraj	person
D1	1	1	2	1	0	0
D2	0	0	1	0	1	1

Now, a column can also be understood as word vector for the corresponding word in the matrix M. For example, the word vector for ‘lazy’ in the above matrix is [2,1] and so on.Here, the rows correspond to the documents in the corpus and the columns correspond to the tokens in the dictionary. The second row in the above matrix may be read as – D2 contains ‘lazy’: once, ‘Neeraj’: once and ‘person’ once.

Now there may be quite a few variations while preparing the above matrix M. The variations will be generally in-

• The way dictionary is prepared.
• Why? Because in real world applications we might have a corpus which contains millions of documents. And with millions of document, we can extract hundreds of millions of unique words. So basically, the matrix that will be prepared like above will be a very sparse one and inefficient for any computation. So an alternative to using every unique word as a dictionary element would be to pick say top 10,000 words based on frequency and then prepare a dictionary.
• The way count is taken for each word.
• We may either take the frequency (number of times a word has appeared in the document) or the presence(has the word appeared in the document?) to be the entry in the count matrix M. But generally, frequency method is preferred over the latter.

Below is a representational image of the matrix M for easy understanding.

1.2 TF-IDF vectorization

This is another method which is based on the frequency method but it is different to the count vectorization in the sense that it takes into account not just the occurrence of a word in a single document but in the entire corpus. So, what is the rationale behind this? Let us try to understand.

Common words like ‘is’, ‘the’, ‘a’ etc. tend to appear quite frequently in comparison to the words which are important to a document. For example, a document A on Lionel Messi is going to contain more occurences of the word “Messi” in comparison to other documents. But common words like “the” etc. are also going to be present in higher frequency in almost every document.

Ideally, what we would want is to down weight the common words occurring in almost all documents and give more importance to words that appear in a subset of documents.

TF-IDF works by penalising these common words by assigning them lower weights while giving importance to words like Messi in a particular document.

So, how exactly does TF-IDF work?

Consider the below sample table which gives the count of terms(tokens/words) in two documents.

Now, let us define a few terms related to TF-IDF.

TF = (Number of times term t appears in a document)/(Number of terms in the document)

So, TF(This,Document1) = 1/8

TF(This, Document2)=1/5

It denotes the contribution of the word to the document i.e words relevant to the document should be frequent. eg: A document about Messi should contain the word ‘Messi’ in large number.

IDF = log(N/n), where, N is the number of documents and n is the number of documents a term t has appeared in.

where N is the number of documents and n is the number of documents a term t has appeared in.

So, IDF(This) = log(2/2) = 0.

So, how do we explain the reasoning behind IDF? Ideally, if a word has appeared in all the document, then probably that word is not relevant to a particular document. But if it has appeared in a subset of documents then probably the word is of some relevance to the documents it is present in.

Let us compute IDF for the word ‘Messi’.

IDF(Messi) = log(2/1) = 0.301.

Now, let us compare the TF-IDF for a common word ‘This’ and a word ‘Messi’ which seems to be of relevance to Document 1.

TF-IDF(This,Document1) = (1/8) * (0) = 0

TF-IDF(This, Document2) = (1/5) * (0) = 0

TF-IDF(Messi, Document1) = (4/8)*0.301 = 0.15

As, you can see for Document1 , TF-IDF method heavily penalises the word ‘This’ but assigns greater weight to ‘Messi’. So, this may be understood as ‘Messi’ is an important word for Document1 from the context of the entire corpus.

1.3 Co-Occurrence Matrix with a fixed context window

The big idea – Similar words tend to occur together and will have similar context for example – Apple is a fruit. Mango is a fruit.

Apple and mango tend to have a similar context i.e fruit.

Before I dive into the details of how a co-occurrence matrix is constructed, there are two concepts that need to be clarified – Co-Occurrence and Context Window.

Co-occurrence – For a given corpus, the co-occurrence of a pair of words say w1 and w2 is the number of times they have appeared together in a Context Window.

Context Window – Context window is specified by a number and the direction. So what does a context window of 2 (around) means? Let us see an example below,

Quick

Brown

Fox

Jump

Over

The

Lazy

Dog

The green words are a 2 (around) context window for the word ‘Fox’ and for calculating the co-occurrence only these words will be counted. Let us see context window for the word ‘Over’.

Quick

Brown

Fox

Jump

Over

The

Lazy

Dog

Now, let us take an example corpus to calculate a co-occurrence matrix.

Corpus = He is not lazy. He is intelligent. He is smart.

	He	is	not	lazy	intelligent	smart
He	0	4	2	1	2	1
is	4	0	1	2	2	1
not	2	1	0	1	0	0
lazy	1	2	1	0	0	0
intelligent	2	2	0	0	0	0
smart	1	1	0	0	0	0

Let us understand this co-occurrence matrix by seeing two examples in the table above. Red and the blue box.

Red box- It is the number of times ‘He’ and ‘is’ have appeared in the context window 2 and it can be seen that the count turns out to be 4. The below table will help you visualise the count.

He	is	not	lazy	He	is	intelligent	He	is	smart
He	is	not	lazy	He	is	intelligent	He	is	smart
He	is	not	lazy	He	is	intelligent	He	is	smart
He	is	not	lazy	He	is	intelligent	He	is	smart

while the word ‘lazy’ has never appeared with ‘intelligent’ in the context window and therefore has been assigned 0 in the blue box.

Variations of Co-occurrence Matrix

Let’s say there are V unique words in the corpus. So Vocabulary size = V. The columns of the Co-occurrence matrix form the context words. The different variations of Co-Occurrence Matrix are-

• A co-occurrence matrix of size V X V. Now, for even a decent corpus V gets very large and difficult to handle. So generally, this architecture is never preferred in practice.
• A co-occurrence matrix of size V X N where N is a subset of V and can be obtained by removing irrelevant words like stopwords etc. for example. This is still very large and presents computational difficulties.

But, remember this co-occurrence matrix is not the word vector representation that is generally used. Instead, this Co-occurrence matrix is decomposed using techniques like PCA, SVD etc. into factors and combination of these factors forms the word vector representation.

Let me illustrate this more clearly. For example, you perform PCA on the above matrix of size VXV. You will obtain V principal components. You can choose k components out of these V components. So, the new matrix will be of the form V X k.

And, a single word, instead of being represented in V dimensions will be represented in k dimensions while still capturing almost the same semantic meaning. k is generally of the order of hundreds.

So, what PCA does at the back is decompose Co-Occurrence matrix into three matrices, U,S and V where U and V are both orthogonal matrices. What is of importance is that dot product of U and S gives the word vector representation and V gives the word context representation.

Advantages of Co-occurrence Matrix

• It preserves the semantic relationship between words. i.e man and woman tend to be closer than man and apple.
• It uses SVD at its core, which produces more accurate word vector representations than existing methods.
• It uses factorization which is a well-defined problem and can be efficiently solved.
• It has to be computed once and can be used anytime once computed. In this sense, it is faster in comparison to others.

Disadvantages of Co-Occurrence Matrix

• It requires huge memory to store the co-occurrence matrix.
• But, this problem can be circumvented by factorizing the matrix out of the system for example in Hadoop clusters etc. and can be saved.

Source: https://www.analyticsvidhya.com/blog/2017/06/word-embeddings-count-word2veec/ 

Fun with Python's collections library

I would like to provide few quick examples of how to use collection library in python to simplify some common tasks.  

Let's say that you have collection of some kind and you need to get basic counts to see what values you have. One way to do it is to use basic map, if you have the value increment it, if not, insert it and initialize the counter to one.

counts = {}
for item in my_collection:
if item in counts:
counts[item] += 1
else:
counts[item] = 1


This can be simplified by using collections library like so

from collections import defaultdict

counts = defaultdict(int) # values will initialize to zero
for item in my_collection:
counts[item] += 1

Now that you have the counts, how about finding top 5?

By using simple list comprehension
count_value_pair = [(count, value) for value, count in counts.items()]
count_value_pair.sort()
print count_value_pair[-5:]

or by using collections Counter library

from collections import Counter

counts = Counter(collection)
counts.most_common(5)

Obviously Counter can be used in the first example above to provide the right numbers
counts.viewkeys() # view all keys
counts.viewvalues() # view all values
counts.viewitems() # view all keys and their values




Of course, you can unlock even more power by using pandas' DataFrames

from pandas import DataFrame

df = DataFrame(collection)

counts = df.value_counts()

here you can also fill in missing values and blanks

df = df.fillna('Missing')
df[df == ''] = 'Unknown'

linux random notes

cd - (previous directory). This will take you to the previous directory you were just at.


To find out what kind of file a file is, you can use the file command. It will show you a description of the file’s contents.
$ file banana.jpg


$ less /home/pete/Documents/text1
Use the following command to navigate through less:
q - Used to quit out of less and go back to your shell.
Page up, Page down, Up and Down - Navigate using the arrow keys and page keys.
g - Moves to beginning of the text file.
G - Moves to the end of the text file.
/search - You can search for specific text inside the text document. Prefacing the words you want to search with /
h - If you need a little help about how to use less while you’re in less, use help.


One thing to note, if you copy a file over to a directory that has the same filename, the file will be overwritten with whatever you are copying over. This is no bueno if you have a file that you don’t want to get accidentally overwritten. You can use the -i flag (interactive) to prompt you before overwriting a file.
$ cp -i mycoolfile /home/pete/Pictures


Let’s say you did want to mv a file to overwrite the previous one. You can also make a backup of that file and it will just rename the old version with a ~.
$ mv -b directory1 directory2


find /home -name puppies.jpg
One cool thing to note is that find doesn’t stop at the directory you are searching, it will look inside any subdirectories that directory may have as well


alias
Sometimes typing commands can get really repetitive, or if you need to type a long command many times, it’s best to have an alias you can use for that. To create an alias for a command you simply specify an alias name and set it to the command.
$ alias foobar='ls -la'
Now instead of typing ls -la, you can type foobar and it will execute that command, pretty neat stuff. Keep in mind that this command won't save your alias after reboot, so you'll need to add a permanent alias in:
~/.bashrc
or similar files if you want to have it persist after reboot.
You can remove aliases with the unalias command:
$ unalias foobar


grep
The grep command is quite possibly the most common text processing command you will use. It allows you to search files for characters that match a certain pattern. What if you wanted to know if a file existed in a certain directory or if you wanted to see if a string was found in a file? You certainly wouldn't dig through every line of text, you would use grep!
Let's use our sample.txt file as an example:
$ grep fox sample.txt
You should see that grep found fox in the sample.txt file.
You can also grep patterns that are case insensitive with the -i flag:
$ grep -i somepattern somefile
To get even more flexible with grep you can combine it with other commands with |.
$ env | grep -i User
As you can see grep is pretty versatile. You can even use regular expressions in your pattern:
$ ls /somedir | grep '.txt$'
Should return all files ending with .txt in somedir.


Now that you know what commands to run as the superuser, the question is how do you know who has access to do that? The system doesn't let every single Joe Schmoe run commands as the superuser, so how does it know? There is a file called the /etc/sudoers file, this file lists users who can run sudo. You can edit this file with the visudo command.


/etc/passwd
Remember that usernames aren't really identifications for users. The system uses a user ID (UID) to identify a user. To find out what users are mapped to what ID, look at the /etc/passwd file.
$ cat /etc/passwd
This file shows you a list of users and detailed information about them. For example, the first line in this file most likely looks like this:
root:x:0:0:root:/root:/bin/bash
Each line displays user information for one user, most commonly you'll see the root user as the first line. There are many fields separated by colons that tell you additional information about the user, let's look at them all:
Username
User's password - the password is not really stored in this file, it's usually stored in the /etc/shadow file. We'll discuss more in the next lesson about /etc/shadow, but for now, know that it contains encrypted user passwords. You can see many different symbols that are in this field, if you see an "x" that means the password is stored in the /etc/shadow file, a "*" means the user doesn't have login access and if there is a blank field that means the user doesn't have a password.
The user ID - as you can see root has the UID of 0
The group ID
GECOS field - This is used to generally leave comments about the user or account such as their real name or phone number, it is comma delimited.
User's home directory
User's shell - you'll probably see a lot of user's defaulting to bash for their shell


/etc/shadow
The /etc/shadow file is used to store information about user authentication. It requires superuser read permissions.
$ sudo cat /etc/shadow
root:MyEPTEa$6Nonsense:15000:0:99999:7:::
You'll notice that it looks very similar to the contents of /etc/passwd, however in the password field you'll see an encrypted password. The fields are separated by colons as followed:
Username
Encrypted password
Date of last password changed - expressed as the number of days since Jan 1, 1970. If there is a 0 that means the user should change their password the next time they login
Minimum password age - Days that a user will have to wait before being able to change their password again
Maximum password age - Maximum number of days before a user has to change their password
Password warning period - Number of days before a password is going to expire
Password inactivity period - Number of days after a password has expired to allow login with their password
Account expiration date - date that user will not be able to login
Reserved field for future use


/etc/group
Another file that is used in user management is the /etc/group file. This file allows for different groups with different permissions.
$ cat /etc/group
root:*:0:pete
Very similar to the /etc/password field, the /etc/group fields are as follows:
Group name
Group password - there isn't a need to set a group password, using an elevated privilege like sudo is standard. A "*" will be put in place as the default value.
Group ID (GID)
List of users - you can manually specify users you want in a specific group

Python Decorators Examples

# Decorator without parameters
import functools

def my_decorator(func):
@functools.wraps(func)
def functions_that_runs_func():
print("In the decorator")
func()
print("After the decorator")
return function_that_runs_func

@my_decorator
def my_functions():
print("I'm the function")

my_function()

# Decorator with parameters
def decorator_with_arguments(number):
def my_decorator(func):
@functools.wraps(func)
def function_that_runs_func(*args, **kwargs):
print("In the decorator")
if number == 56:
print("Not running the function")
else:
func(*args, **kwargs)
print("After the decorator")
return function_that_runs_func
return my_decorator

@decorator_with_arguments(57)
def my_function_too(x, y):
print (x+y)

my_function_too(57, 67)

How to find all files containing specific text on Linux?

grep -rnw '/path/to/somewhere/' -e "pattern"

where pattern can be defined with exact phrase or wildcards

iptables

    sudo vim /etc/sysconfig/iptables

    sudo service iptables restart  

Commands to get Kerberos working

klist - what you have

kdestroy - destroy what you have if for example your certs are expired

kinit - create new ones for that user by proving password

kinit -p user_name - create new one for the specified user_name

Bokeh Interfaces

• Bokeh Models Interface
  • Low level - gives full control on every plotting element
  • Requires to write lots of code
• Bokeh Plotting Interface
  • Intermediate level - autocreates some default elements but also gives full control to change them
  • Requires decent amount of code
  • Used by most people
• Bokeh Charts Interface
  • High level - not very flexible and may run into shortcomings when making advanced plots
  • Requires few lines of code