Tuesday, August 30, 2016

Using with to handle exception inside of your python unit tests

I am sure you have seem something like that in one of the tests

def test_me(self):
with self.assertRaises(ZeroDivisionError):
1 / 0

What does it do? It is simply means that test_me would pass if the logic inside of it would throw ZeroDivisionError exception.

But what all that 'with' stuff?

If you ever tried to open up a file to read data, etc you certainly remember doing all error checking, catching errors, closing files, etc... nightmare, hence context manager was introduced that would allow you to open the file by using 'with' and majority of checks and file closing was done for you.

Same in this case, you are using 'with' in this case to use assertRaises as a context manager that ensures that the error in this case is properly caught and cleaned up while it is being recorded. (Same idea as dealing with files...)

So without context manager, your code would have looked like following the old format

def test_me_in_old_way(self):
import operator
self.assertRaises(ZeroDivisionError, operator.devide, 1, 0)

First argument is error type that you expect to raise, second is the operator, and last two are the args.
Much more verbose

Thursday, August 25, 2016

Python Aha's

1. To check if you have certain package in your python instance and what its version is
$ python
>> import sqlalchemy
>> sqlalchemy.__version__

2. Path at terminal when executing this file
$ os.getcwd()

3.

Tuesday, August 23, 2016

Ruby Aha's

Commenting out in ERB file

single line

<%-# commented line -%>

whole block

<% if false %>
code to comment
<% end %>

To run Rails in PRODUCTION mode

RAILS_ENV=production rails server

If you see css or javascript files being lost make sure to execute

rake assets:precompile

and then start it via

RAILS_SERVE_STATIC_FILES=true rails server -b <ip.address> -e production

Move your dev database to use posgres instead of default sqllite

Install postgress gem - Update your Gemfile

gem "pg"

Connect it via database.yml file where you to provide database information (host, user/pass, port, etc)

Migrate database scripts into it

rake db:migrate RAILS_ENV=development

Populate sample data via seed
rake db:seed

fsff

Wednesday, August 17, 2016

Linux Aha's

How to find processes running on a certain port... and kill them

Get the processes for port 5000

lsof -t -i:5000

`or` netstat -nlp | grep :5000

Now find where the process is running from...












 ls -l /proc/[pid]/exe

`Kill processes running on port 5000`

kill 'lsof -t -i:5000'

How to add trusted root certificated to the server

Install the ca-certificates package:

yum install ca-certificates
Enable the dynamic CA configuration feature:

update-ca-trust force-enable
Add it as a new file to /etc/pki/ca-trust/source/anchors/:

cp foo.crt /etc/pki/ca-trust/source/anchors/
Use command:

update-ca-trust extract

More here...

Wednesday, August 10, 2016

Git: ignore files

Let's say that you executed following lines and got whole bunch of random files that are ready to be checked in when you type git status

git init
git add .
git status

How do you remove them and keep them out of your git repo?
Create .gitignore file and add regex expressions into it to match files that you want to exclude

touch .gitignore
vim .gitignore

<
.gitignore
etc
>

What to do with the files that are already ready to be checked in? Remove them via following commad

git rm --cached FILENAME

Now check what you have going to be committed. Should be all clean by now

git status

Commit!

git commit -m "Initial commit"

Sunday, August 7, 2016

GREAT explanation on how to use Rails Devise: https://launchschool.com/blog/how-to-use-devise-in-rails-for-authentication... Copied as a back up

How to Use Devise in Rails for Authentication

Devise is a popular authentication solution for Rails applications. It provides a full gamut of features, and can be configured to meet all, but the most unusual, requirements. In this tutorial, we are going to look at how Devise is implemented on a high level, go through configueration and common use patterns of the gem, how to set up authentication with Devise, and finally demostrate how to write tests with Devise.

Devise and Warden

The Devise gem is built on top of Warden. Warden is a Rack application, which means that it runs as a separate and standalone module, and is (nearly always) executed before the chief Rails application is invoked.
Warden provides the cookie handling that verifies the identity of a logged in user via a (secure) session string, in which the id (primary key) of a particular user is somehow stored and disguised. Warden also provides a hook so your app can deal with users who aren’t currently logged in. These users will either have restricted access, or none at all, except, of course, to sign-in/sign-up pages.
Warden knows nothing about the existence of your Rails app. As a result, Warden does not provide helper methods, controller classes, views, configuration options and log in failure handling. These things are what Devise supplies.
Devise often interacts with Warden using Strategies. (a strategy is a design pattern in which an algorithm is encapsulated within a dedicated class, which implements a method with a commonly shared name. You change the class to vary the algorithm). The Strategies that Devise employs are for encrypting passwords, email confirmations, and for HTTP Authentication. If you need to extend or augment Devise, you can implement a customized Strategy class. But this is an advanced subject, for which there is usually little call.

Understanding How Devise Works

Devise’s README contains all the information you need to get started, and it also touches on its more advanced features. It recommends that you understand the basic authentication techniques (i.e. registrations and logins) before you proceed to integrate it in your app.

A Kick-Start Devise Installation

To have Devise completely installed in an existing Rails app, (where a user’s model class is named, User), the following six shell commands are required:

echo "gem 'devise'" >> Gemfile    # Or edit Gemfile and add line: gem 'devise'
bundle install                    # Fetch and install the gems
rails generate devise:install     # Creates config file, etc.
rails generate devise user        # Create model class, routes, etc.
rake db:migrate                   # Create user table
rails generate devise:views users # Creates (minimal) views

This assumes that the user model is called User.
Devise makes no assumption about the class name of the user model. Though it’s usually, ‘User’, or less commonly, ‘Admin’. These two class names can sit side-by-side in any app, as can any other names of user classes you may care to conjure up.
The rails generate devise:views users command creates the directory /app/views/users with all the devise views, such as login form, registration form, etc. If you want customize these files and have Devise use them you must set config.scoped_views to true in the configuration file, located at /app/config/initializers/devise.rb.

Devise Configuration

Once you have installed Devise, as per the above commands, the next step is configuration. This is specified in two main files:

Firstly in a global config file, config/initializers/devise.rb. If you change the settings in this, they won’t become effective until you restart the Rails server. And the settings apply to the whole site.
Secondly, in a model class representing a (registered) user, which can be tailored to suit a particular role, for example if your app has an end-user, as well as having a site administrator (more on this later).

Devise Utility Methods

Devise contains dozens of classes, including, models, controllers, mailers, helpers, hooks, routes, and views, but since Devise exposes it’s functionality in a small number of simple helper methods, It’s unlikely that you will even need to know of the existence of all of them. The most important helper methods Devise gives you to use in your own app are:

authenticate_user!
current_user
user_signed_in?
sign_in(@user)
sign_out(@user)
user_session

authenticate_user!

The authenticate_user! class method (controller only), ensures a logged in user is available to all, or a specified set of controller actions. This method is invoked via a before_filter, for example:

# This class is intended to be a superclass from which
# the other (end-user related) controller classes inherit
class EndUserBaseController < ApplicationController
  before_filter :authenticate_user!
end

If the user isn’t logged in Devise backs off, and, by default, redirects to its own sign-in page. If a user is actually logged in, then Devise, by default, redirects to the root route. The root route is the default route for an app. It’s defined in config/routes.rb with something like the following:

1	`root to: 'landing#index'`

The authenticate_user! method’s use is for applications that require a logged in user to access (a set of) their pages. In this case, it is very common to use the current_user method (see just below) as the first receiver in a method chain that accesses a database, for cases where a record’s owner is to be stored. For example:

class SentMessagesController < EndUserBaseController
  before_filter :authenticate_user!

  def index
    @sent_messages = current_user.sent_messages.all
  end
end

The before_filter :authenticate_user! line ensures that current_user is never nil, thus avoiding the fatal error: method called on nil object.

current_user

The current_user method , whose purpose is self-explanatory, simply returns the model class relating to the signed in user. It returns nil if a user has not, as yet, signed in.

user_signed_in?

The user_signed_in? query method, which really just checks if the current_user method returns a value that’s not nil.

sign_in(@user) and sign_out(@user)

The sign_in(@user) and the sign_out(@user) methods. These are useful to login (or logout) a (newly created) user.

user_session

The user_session method, which returns meta data on a logged in user.
An application programmer will normally only need to use the current_user and user_signed_in? methods regularly. These, of course, are present as helper methods. So you can, for example, have a message in the header of your app stating the name of the signed in user.
Note that if you refer to an admin (who has a dedicated Admin class), the access method becomes current_admin, not current_user.
The prefix, or sometime suffix, is the corresponding underscored class name of a user class. For example, a user class name of ‘EndUser’, when underscored, becomes ‘end_user’, so the logged in user would in this situation be current_end_user.
Devise also defines a whole host of routes, which are defined in config/routes.rb with a line like:

devise_for :users

This line is automatically inserted by the Devise generator, and the default values it applies are fine for most needs. If not, it takes a complex options hash, that can be customized – mainly to prevent name collisions, or to have different route names, should you really want them!

Devise Modules

When you invoke the Devise generator, a model class (in `app/models*) is created for you to modify for your specific requirements.
This user model class is where many of the most important configuration options are specified. Perhaps the most important of these are the Devise modules to use. These modules provide enhanced security functionality. There are ten modules in all.
The modules are included thusly:

class User < ActiveRecord::Base
  devise :database_authenticatable, :registerable,
          :recoverable, :rememberable, :trackable, :validatable,
          :confirmable, :lockable, :timeoutable
end

Note that when Devise generates a class similar to this for you, it also generates a database migration, which includes the fields on which the functionality of these modules depend. This migration file is well commented and it specifies which fields relate to which modules. It should be modified if you want to trim down the number of fields attached to the model. If some modules are not required, the fields related to them can be removed.
A number of these modules have associated HTML forms and links. In the forms, an end-user typically fills in their email address, and in exchange they are sent back an email which contains confirmation links back to Devise controllers in your app, and include an authentication to token for security. These forms are mostly to (re-)enable a particular account.
Although the Devise README lists these options, we’ll explain them a little here as well.

database_authenticatable – ensures that a user enters a correct password, and it encrypts the said password, before saving it.
confirmable – ensures newly registered users confirm their accounts by clicking a link on a (Devise sent) mail they receive. This is a defence against bots creating fake, but valid, users.
recoverable – for handling forgotten passwords safely.
registerable – allows users to register themselves, and subsequently change their login credentials.
rememberable – for transparently logging on a user, based on a cookie. This is the functionality associated with the Remember me? checkbox you often see on login forms. Since the Devise config file specifies a timeout period, this is often of limited use. And is a security risk if the user steps away from their browser, with others about.
trackable – stores login information i.e. the user’s IP address, login time, last login time, and the total number of logins. These values are generally for site admins to look at, when trying to trace unusual activity.
validatable – ensures that the given email/name and passwords conform to a particular format.
lockable – limits the number of login attempts allowed before further access to a particular account is prohibited. Access restrictions are lifted after a time span, or if a (legitimate) user requests that an email be sent that contains a link to unblock his/her account.

These modules are lazily loaded, so if you omit some, the corresponding code is ignored.

Devise’s Global Configuration File

Devise has a very well commented configuration file, config/initializers/devise.rb. This is where its site-wide settings are specified.
These are things like the minimum password length, the password encryption algorithm, a flag to have a case sensitive username lookup, a timeout period, the ‘pepper’ string, and lots more as well. There is no substitute to actually taking a look at this file. If the options seem a bit bewildering at first, their exact purposes will become clearer on re-readings.

Protecting the Mass-Assignment of User Attributes

Naturally, all of the fields associated with a Devise user contain sensitive data, whose alteration could seriously compromise security. For this reason, it is probably a good idea never to allow your app to have pages (which are accessible to untrusted users) that update (or create) a user model. Only allow these attributes to be updated by Devise, where you know it’ll be done securely.
If an app’s requirements ordain that auxiliary user information be kept, such as an address or a phone number, these details are best kept on an entirely separate table which has a foreign key back to the (Devise generated) user table. This auxiliary table would have a one-to-one relationship with the Devise user table.
If you really must add fields to the (Devise generated) user model you must absolutely make sure that Devise’s fields cannot be changed. The means by which you achieve this is different for Rails 4 apps than it is for Rails 3 apps. In the former case, it’s with strong_parameters in the controller. In the latter case, it’s with the attr_accessible or attr_protected class methods in the model.
The Devise README explains this in adequate detail.

Adding Further Authentication

Having said all that in the preceding section, there is, arguably, an exception to this “no added user model fields” principle. This is where a user model has a flag like admin?, which is used to distinguish an app’s administator (super-user) from a regular end-user.
This is not a Devise added field, but one an app developer would add in a database migration. This field should not set up in a module or page of an actual app. Instead it should be set manually in a direct database command.
For example:

rails console
>>> user = User.find(1)
>>> user.admin = true
>>> user.save

This pattern is common, and is useful in apps where site administrators need to perform tasks in common with regular users.
In some other kinds of apps, the site administrators may need perform a completely different set of tasks to regular users. In this case it is cleaner to have separate classes for a site administrator and for a regular user.

Handling Different Types of Users in Devise

In this section, we’ll take a look at some ways in which some common use-cases, relating to access restrictions, can be handled with Devise’s controller class methods (macros). (This is an area where the distinction between authentication and authorization becomes blurred.)
These class methods are invoked as before filters, and take effect for a number of the controller’s available actions.
In this first case, suppose we want to ensure that every controller action requires a logged in user, except for the login and register actions:

class ApplicationController < ActionController::Base
  # Ensures all actions invoke this (except those just below)
  before_filer :authenticate_user!
end

class AuthenticationController < ApplicationController
  # Turn off user authentication for all actions in this controller
  skip_before_filer :authenticate_user!

  def login
    '...'
  end

  def register
    '...'
  end
end

In this second case, suppose we want to grant logged in users full CRUD access, but unregistered only to have only read access for a selected group of controllers:

class CrudController < ApplicationController
  before_filer :authenticate_user!, except: [ :index, :show ]
end

class BlogsController < CrudController
end

class CommentsController < CrudController
end

In this third case, suppose we want to partition access for logged in regular users, and for site administrators, both of whom are represented by separate model classes. A regular user has a User class, and an admin has an Admin class. For example:

# All administrator controllers should inherit from this controller
class AdminController < ApplicationController
  before_filer :authenticate_admin!
end

# All end-user controllers should inherit from this controller
class EndUserController < ApplicationController
  before_filer :authenticate_user!
end

In this fourth case, suppose we want to have an end-user and a site administrator who both share the same user model. But, in this scenario, the site administrators are identified by an admin flag set to a true value on the model record. Whereas regular users have this flag switched off. For example:

class ApplicationController < ActionController::Base
  before_filer :authenticate_user!
end

# All administrative controllers should inherit from this controller
class AdminController < ApplicationController
  before_filter :ensure_admin!

  private

  def ensure_admin!
    unless current_user.admin?
      sign_out current_user

      redirect_to root_path

      return false
    end
  end
end

An added benefit of arranging the controller classes like this is that you can define private methods that are specific to a certain set of controllers. In the following example, all controllers that inherit from the CrudController will ensure that black-listed users are prohibited from further editing, but can still look at the site.

class CrudController < ApplicationController
  before_filer :authenticate_user!, except: [ :index, :show ]

  before_filer :block_abusers!, except: [ :index, :show ]

  private

  def block_abusers!
    if current_user.blacklisted?
      flash[:notice] = 'You are banned for being naughty!'

      redirect_to root_path

      return false
    end
  end
end

Devise Invitable Gem

If you want users to be registered through an invitation carried out by a site administrator, or by a group leader. (As an alternative to end-users explicitly registering themselves on provided web forms), there is a another gem to use, devise_invitable.
The Invitable gem generates a form in which a number of email addresses are filled in. Emails are sent out to the respective addresses inviting the users to confirm their registrations, which are actually performed by the site administrator.
As its name suggests, this gem must be used in conjunction with Devise. And it automatically links the invited users with the user who initially invited them.

Testing

It would be counter-productive to test Devise itself. It already has a comprehensive suite of tests that you are able run yourself, should you doubt its efficacy. However, you ought to test the parts of your app which interface with Devise, to ensure that Devise is presented with correct data, and to ensure that Devise remains in general working order.
Also, your own app’s tests are very likely to depend on your having a logged-in user, on whose behalf, the various operations of controller, acceptance, integration and view tests, are performed.
You can programmatically create a devise user as follows:

User.create!(email: "me@home.com", password: "watching the telly")

As an aside, creating users in this way is also useful in seed files in which you set up site administrators, etc.. Seeding data is done as part of a deployment step, so that you can have ready-made users as soon as your site goes live. The Rails’ provided file where you set up seed data is in db/seeds.rb, and you run it with the shell command:

$ rake db:seed

An RSpec example of a controller test that sets up a logged in user may go as follows:

describe BlogController do
  let :user do
    User.create!(email: "me@home.com", password: "watching the telly")
  end

  before { sign_in user }

  context "creating a post" do
    post :create, subject: 'matter', body: 'and soul'

    expect(assigns(:blog)).to be_instance_of Blog

    expect(:blog.try :subject).to eq 'matter'

    expect(:blog.try :user).to eq user
  end
end

Perhaps the above test has too many expectations, but it should give you a general idea, if you choose to support a suite of controller tests.
Though some might argue that the testing of such functionality would be better placed in integration tests. That, in general, controller testing is often superfluous, and has poorer code coverage, than integration testing does.
An RSpec example of an integration test could be as follows:

describe "user log in" do
  it "allows an existing user to sign in" do
    User.create!(email: "me@home.com", password: "watching the telly")

    visit "/users/sign_in"

    fill_in "Email", with: "me@example.com"
    fill_in "Password", with: "watching the telly"

    click_button "Sign in"

    expect(page).to have_content("Signed in successfully.")
  end
end

The above (smoke) test ensures that your app interfaces with Devise correctly, and in particular that, your view contains a form with the required URL and fields. If you wrote this test, you would probably write another one to test the registration process as well.

Wrapping up

If you have not done so already, your next step is to peruse the Devise README. which is probably the best single document on the practical use of Devise that’s readily available. If you want to do something out of the ordinary with Devise there’s a comprehensive list of howtos.
These two documents will, most likely, be the only ones you need to read in order to use Devise effectively.
Also if you would like more background information, take a look at the Devise wiki.

Thursday, August 4, 2016

Data Lake vs Data Warehouse

Great article is published by Martin Fowler:http://martinfowler.com/bliki/DataLake.html

But to quickly summarize, think of data lakes as data dump of raw unprocessed data that people can go back to and get the information that they might need from the raw contents

In Data Warehouse, someone would define a schema and will do some data cleaning and processing. This way, people can just grab a set from the warehouse and expect it to be a certain way and don't worry about manual processing.

So... if you don't know what you need and you don't mind looking for it, go with the lake
if you rather have sets of data ready to use for you in predefined format and you don't care about some potential missing data, go with the warehouse.

Git Aha's

To revert your changes back to the last working version, i.e. erase what you just did, use

git checkout .

To revert your changes to the version multiple commits in the past

Say you are developing a new feature and you realize after few commits that you went off to a way different route that you suppose to and you need to back it up few commits and start over... this definitely would be a cleaner way vs trying to remove what was done manually. How to do it though?

Check out what you want and get rid of all that code...

git reset --hard 0d3b7ac32

Then you would push it up

git push origin +master

Pretty simple once you know it.

To revert a single file to what's in current git use

git checkout HEAD -- <file_name>

To view the changes between staged and what's in current repo

git diff --cached

fffff