Tuesday, December 13, 2016

How to terminate PostgreSQL sessions

While putting my db admin hat on to troubleshoot unresponding postgres server, I stumbled across this great reference on how to kill unresponsive db connections, which I thought to report to preserve it for my own future use and to give the author credit well deserved.

Original article can be found at: https://blog.sleeplessbeastie.eu/2014/07/23/how-to-terminate-postgresql-sessions/

Recently, I have encountered an interesting issue, as I could not perform specific database operations due to unwanted and active sessions using the database. Thus, I will briefly note the solution for further reference.

Prerequisites

This blog post is based on a Debian Wheezy and PostgreSQL 9.1 version.
$ lsb_release -d
Description: Debian GNU/Linux 7.5 (wheezy)
postgres=# select * from version();
PostgreSQL 9.1.13 on x86_64-unknown-linux-gnu, compiled by gcc (Debian 4.7.2-5) 4.7.2, 64-bit
I have deliberately written down this information here, as there are some minor differences between PostgreSQL versions, so please be aware of potential differences.

The problem and the solution

Sometimes you need to terminate connections initiated by badly behaving client application, or just make sure nobody is querying database during a major update.
The solution is to use pg_stat_activity view to identify and filter active database sessions and then use pg_terminate_backend function to terminate them.
To prevent access during an update process or any other important activity you can simply revokeconnect permission for selected database users or alter pg_database system table.

Who is permitted terminate connections

Every database role with superuser rights is permitted to terminate database connections.

How to display database sessions

pg_stat_activity system view provides detailed information about server processes.
SELECT datname as database,
       procpid as pid,
       usename as username,
       application_name as application,
       client_addr as client_address,
       current_query
  FROM pg_stat_activity
Sample output that will be used in the following examples.
 database | pid  | username | application | client_address |                                                                           current_query
----------+------+----------+-------------+----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
 blog     | 8603 | blog     | blog_app    | 192.168.3.11   | select * from posts order by pub_date
 postgres | 8979 | postgres | psql        |                | select datname as database,procpid as pid,usename as username,application_name as application, client_addr as client_address, current_query from pg_stat_activity
 wiki     | 8706 | wiki     | wiki_app    | 192.168.3.8    | 
(3 rows)

How to terminate all connections to the specified database

Use the following query to terminate all connections to the specified database.
SELECT pg_terminate_backend(procpid)
  FROM pg_stat_activity
 WHERE datname = 'wiki'

How to terminate all connections tied to specific user

Use the following query to terminate connections initiated by a specific user.
SELECT pg_terminate_backend(procpid)
  FROM pg_stat_activity
 WHERE usename = 'blog'

How to terminate all connections but not my own

To terminate every other database connection you can use process ID attached to the current session.
SELECT pg_terminate_backend(procpid)
  FROM pg_stat_activity
 WHERE procpid <> pg_backend_pid()
Alternatively, you can simply use username to filter out permitted connections.
SELECT pg_terminate_backend(procpid)
  FROM pg_stat_activity
 WHERE username <> current_username
Every example mentioned above can be extended to include more conditions like database name, client name, query, or even client address.

How to cancel running query

It is not always desired to abruptly terminate existing database connection, as you can just cancel running query using function shown in the following query.
SELECT pg_cancel_backend(procpid)
  FROM pg_stat_activity
 WHERE usename = 'postgres'

How to prevent users from connecting to the database

Database connect privilege

To prevent connections from specific database user revoke the connect privilege for selected database.
REVOKE CONNECT
    ON DATABASE wiki
  FROM wiki
To reverse this process use the GRANT statement.
GRANT CONNECT
   ON DATABASE wiki
   TO wiki
Use the public keyword to specify every database user.
REVOKE CONNECT
    ON DATABASE wiki
  FROM public

Database user login privilege

I did not mentioned it earlier but you can also use database user login privilege to disallow new connections.
ALTER ROLE wiki NOLOGIN;
To reverse this modification use the following query.
ALTER ROLE wiki LOGIN;

pg_database system table

Alternatively, you can alter pg_database system table to disallow new connections to specific database.
UPDATE pg_database
   SET datallowconn = FALSE
 WHERE datname = 'blog'
To reverse this process use the following query.
UPDATE pg_database
   SET datallowconn = TRUE
 WHERE datname = 'blog'

How to use the above-mentioned queries inside shell script

Use the postgres user to terminate connections..
#!/bin/sh
su postgres  -l -c "psql  -c 'select pg_terminate_backend(procpid) \
                              from pg_stat_activity \
                              where datname = '\"'wiki'\"''"
Use role with superuser rights to terminate connections.
#!/bin/sh
PGHOST=localhost PGUSER=admin PGPASSWORD=adminpass psql postgres -c "select pg_terminate_backend(procpid) \
                                                                     from pg_stat_activity \
                                                                     where datname = 'wiki'"
You can read more about providing password using environment variables in my previous article: How to non interactively provide password for the PostgreSQL interactive terminal.

References

Friday, November 18, 2016

Layman's Introduction to Random Forests

From: http://blog.echen.me/2011/03/14/laymans-introduction-to-random-forests/

<Beautiful explanation...>

Layman's Introduction to Random Forests

Suppose you’re very indecisive, so whenever you want to watch a movie, you ask your friend Willow if she thinks you’ll like it. In order to answer, Willow first needs to figure out what movies you like, so you give her a bunch of movies and tell her whether you liked each one or not (i.e., you give her a labeled training set). Then, when you ask her if she thinks you’ll like movie X or not, she plays a 20 questions-like game with IMDB, asking questions like “Is X a romantic movie?”, “Does Johnny Depp star in X?”, and so on. She asks more informative questions first (i.e., she maximizes the information gain of each question), and gives you a yes/no answer at the end.
Thus, Willow is a decision tree for your movie preferences.
But Willow is only human, so she doesn’t always generalize your preferences very well (i.e., she overfits). In order to get more accurate recommendations, you’d like to ask a bunch of your friends, and watch movie X if most of them say they think you’ll like it. That is, instead of asking only Willow, you want to ask Woody, Apple, and Cartman as well, and they vote on whether you’ll like a movie (i.e., you build an ensemble classifier, aka a forest in this case).
Now you don’t want each of your friends to do the same thing and give you the same answer, so you first give each of them slightly different data. After all, you’re not absolutely sure of your preferences yourself – you told Willow you loved Titanic, but maybe you were just happy that day because it was your birthday, so maybe some of your friends shouldn’t use the fact that you liked Titanic in making their recommendations. Or maybe you told her you loved Cinderella, but actually you really really loved it, so some of your friends should give Cinderella more weight. So instead of giving your friends the same data you gave Willow, you give them slightly perturbed versions. You don’t change your love/hate decisions, you just say you love/hate some movies a little more or less (formally, you give each of your friends a bootstrapped version of your original training data). For example, whereas you told Willow that you liked Black Swan and Harry Potter and disliked Avatar, you tell Woody that you liked Black Swan so much you watched it twice, you disliked Avatar, and don’t mention Harry Potter at all.
By using this ensemble, you hope that while each of your friends gives somewhat idiosyncratic recommendations (Willow thinks you like vampire movies more than you do, Woody thinks you like Pixar movies, and Cartman thinks you just hate everything), the errors get canceled out in the majority. Thus, your friends now form a bagged (bootstrap aggregated) forest of your movie preferences.
There’s still one problem with your data, however. While you loved both Titanic and Inception, it wasn’t because you like movies that star Leonardio DiCaprio. Maybe you liked both movies for other reasons. Thus, you don’t want your friends to all base their recommendations on whether Leo is in a movie or not. So when each friend asks IMDB a question, only a random subset of the possible questions is allowed (i.e., when you’re building a decision tree, at each node you use some randomness in selecting the attribute to split on, say by randomly selecting an attribute or by selecting an attribute from a random subset). This means your friends aren’t allowed to ask whether Leonardo DiCaprio is in the movie whenever they want. So whereas previously you injected randomness at the data level, by perturbing your movie preferences slightly, now you’re injecting randomness at the model level, by making your friends ask different questions at different times.
And so your friends now form a random forest.

Tuesday, November 15, 2016

Git - tab completion

Personally, I really like to tab the commands sometimes since i am lazy and i don't feel like typing the whole thing out and sometimes out of curiosity of what other commands are available to me.

There are the steps on how to enable tab completion in git

1. Download git-completion.bash to your home directory and name it .git-completion.bash
    https://raw.githubusercontent.com/git/git/master/contrib/completion/git-completion.bash
2. Add source ~/.git-completion.bash to your .bash_profile file
3. Reload .bash_profile file: source ~/.bash_profile

Now you should be able to tab complete your git commands.

Monday, October 17, 2016

ipython notebook

Run it on a port of your choice:

ipython notebook --no-browser --ip=* --port=8889

Thursday, September 8, 2016

Git: How to check in a folder, but ignore its contents

Git does not let you to check in an empty folder, even if you are using it as a temp output location. How to work around it?

put this .gitignore into the folder, then git add .gitignore
*
*/
!.gitignore
The * line tells git to ignore all files in the folder, but !.gitignore tells git to still include the .gitignore file. This way, your local repository and any other clones of the repository all get both the empty folder and the .gitignore it needs.
May be obvious but also add */ to the git ignore to also ignore sub folders.

Tuesday, August 30, 2016

Using with to handle exception inside of your python unit tests

I am sure you have seem something like that in one of the tests

def test_me(self):
  with self.assertRaises(ZeroDivisionError):
    1 / 0

What does it do? It is simply means that test_me would pass if the logic inside of it would throw ZeroDivisionError exception.

But what all that 'with' stuff?

If you ever tried to open up a file to read data, etc you certainly remember doing all error checking, catching errors, closing files, etc... nightmare, hence context manager was introduced that would allow you to open the file by using 'with' and majority of checks and file closing was done for you.

Same in this case, you are using 'with' in this case to use assertRaises as a context manager that ensures that the error in this case is properly caught and cleaned up while it is being recorded. (Same idea as dealing with files...)

So without context manager, your code would have looked like following the old format

def test_me_in_old_way(self):
  import operator
  self.assertRaises(ZeroDivisionError, operator.devide, 1, 0)

First argument is error type that you expect to raise, second is the operator, and last two are the args.
Much more verbose

Thursday, August 25, 2016

Python Aha's

1. To check if you have certain package in your python instance and what its version is
$ python
>> import sqlalchemy
>> sqlalchemy.__version__

2. Path at terminal when executing this file
os.getcwd()

3. 

Tuesday, August 23, 2016

Ruby Aha's

Commenting out in ERB file

single line
<%-# commented line -%>
whole block 
<% if false %>
code to comment
<% end %>

To run Rails in PRODUCTION mode

RAILS_ENV=production rails server
If you see css or javascript files being lost make sure to execute

rake assets:precompile


and then start it via

RAILS_SERVE_STATIC_FILES=true rails server -b <ip.address> -e production

Move your dev database to use posgres instead of default sqllite

Install postgress gem - Update your Gemfile 
gem "pg"

Connect it via database.yml file where you to provide database information (host, user/pass, port, etc)

Migrate database scripts into it
rake db:migrate RAILS_ENV=development

Populate sample data via seed
rake db:seed

fsff

Wednesday, August 17, 2016

Linux Aha's

How to find processes running on a certain port... and kill them

Get the processes for port 5000

lsof -t -i:5000

or netstat -nlp | grep :5000

Now find where the process is running from...

 ls -l /proc/[pid]/exe

Kill processes running on port 5000


kill 'lsof -t -i:5000'

How to add trusted root certificated to the server

  1. Install the ca-certificates package:
    yum install ca-certificates
  2. Enable the dynamic CA configuration feature:
    update-ca-trust force-enable
  3. Add it as a new file to /etc/pki/ca-trust/source/anchors/:
    cp foo.crt /etc/pki/ca-trust/source/anchors/
  4. Use command:
    update-ca-trust extract
More here...

Wednesday, August 10, 2016

Git: ignore files

Let's say that you executed following lines and got whole bunch of random files that are ready to be checked in when you type git status

git init
git add .
git status

How do you remove them and keep them out of your git repo?
Create .gitignore file and add regex expressions into it to match files that you want to exclude

touch .gitignore
vim .gitignore

<
.gitignore
etc
>

What to do with the files that are already ready to be checked in? Remove them via following commad

git rm --cached FILENAME

Now check what you have going to be committed. Should be all clean by now

git status

Commit!

git commit -m "Initial commit"

Sunday, August 7, 2016


GREAT explanation on how to use Rails Devise: https://launchschool.com/blog/how-to-use-devise-in-rails-for-authentication... Copied as a back up

How to Use Devise in Rails for Authentication

Devise is a popular authentication solution for Rails applications. It provides a full gamut of features, and can be configured to meet all, but the most unusual, requirements. In this tutorial, we are going to look at how Devise is implemented on a high level, go through configueration and common use patterns of the gem, how to set up authentication with Devise, and finally demostrate how to write tests with Devise.

Devise and Warden

The Devise gem is built on top of Warden. Warden is a Rack application, which means that it runs as a separate and standalone module, and is (nearly always) executed before the chief Rails application is invoked.
Warden provides the cookie handling that verifies the identity of a logged in user via a (secure) session string, in which the id (primary key) of a particular user is somehow stored and disguised. Warden also provides a hook so your app can deal with users who aren’t currently logged in. These users will either have restricted access, or none at all, except, of course, to sign-in/sign-up pages.
Warden knows nothing about the existence of your Rails app. As a result, Warden does not provide helper methods, controller classes, views, configuration options and log in failure handling. These things are what Devise supplies.
Devise often interacts with Warden using Strategies. (a strategy is a design pattern in which an algorithm is encapsulated within a dedicated class, which implements a method with a commonly shared name. You change the class to vary the algorithm). The Strategies that Devise employs are for encrypting passwords, email confirmations, and for HTTP Authentication. If you need to extend or augment Devise, you can implement a customized Strategy class. But this is an advanced subject, for which there is usually little call.

Understanding How Devise Works

Devise’s README contains all the information you need to get started, and it also touches on its more advanced features. It recommends that you understand the basic authentication techniques (i.e. registrations and logins) before you proceed to integrate it in your app.

A Kick-Start Devise Installation

To have Devise completely installed in an existing Rails app, (where a user’s model class is named, User), the following six shell commands are required:

1
2
3
4
5
6
echo "gem 'devise'" >> Gemfile    # Or edit Gemfile and add line: gem 'devise'
bundle install                    # Fetch and install the gems
rails generate devise:install     # Creates config file, etc.
rails generate devise user        # Create model class, routes, etc.
rake db:migrate                   # Create user table
rails generate devise:views users # Creates (minimal) views
This assumes that the user model is called User.
Devise makes no assumption about the class name of the user model. Though it’s usually, ‘User’, or less commonly, ‘Admin’. These two class names can sit side-by-side in any app, as can any other names of user classes you may care to conjure up.
The rails generate devise:views users command creates the directory /app/views/users with all the devise views, such as login form, registration form, etc. If you want customize these files and have Devise use them you must set config.scoped_views to true in the configuration file, located at /app/config/initializers/devise.rb.

Devise Configuration

Once you have installed Devise, as per the above commands, the next step is configuration. This is specified in two main files:
  • Firstly in a global config file, config/initializers/devise.rb. If you change the settings in this, they won’t become effective until you restart the Rails server. And the settings apply to the whole site.
  • Secondly, in a model class representing a (registered) user, which can be tailored to suit a particular role, for example if your app has an end-user, as well as having a site administrator (more on this later).

Devise Utility Methods

Devise contains dozens of classes, including, models, controllers, mailers, helpers, hooks, routes, and views, but since Devise exposes it’s functionality in a small number of simple helper methods, It’s unlikely that you will even need to know of the existence of all of them. The most important helper methods Devise gives you to use in your own app are:
  • authenticate_user!
  • current_user
  • user_signed_in?
  • sign_in(@user)
  • sign_out(@user)
  • user_session

authenticate_user!

The authenticate_user! class method (controller only), ensures a logged in user is available to all, or a specified set of controller actions. This method is invoked via a before_filter, for example:

1
2
3
4
5
# This class is intended to be a superclass from which
# the other (end-user related) controller classes inherit
class EndUserBaseController < ApplicationController
  before_filter :authenticate_user!
end
If the user isn’t logged in Devise backs off, and, by default, redirects to its own sign-in page. If a user is actually logged in, then Devise, by default, redirects to the root route. The root route is the default route for an app. It’s defined in config/routes.rb with something like the following:

1
root to: 'landing#index'
The authenticate_user! method’s use is for applications that require a logged in user to access (a set of) their pages. In this case, it is very common to use the current_user method (see just below) as the first receiver in a method chain that accesses a database, for cases where a record’s owner is to be stored. For example:

1
2
3
4
5
6
7
class SentMessagesController < EndUserBaseController
  before_filter :authenticate_user!

  def index
    @sent_messages = current_user.sent_messages.all
  end
end
The before_filter :authenticate_user! line ensures that current_user is never nil, thus avoiding the fatal error: method called on nil object.

current_user

The current_user method , whose purpose is self-explanatory, simply returns the model class relating to the signed in user. It returns nil if a user has not, as yet, signed in.

user_signed_in?

The user_signed_in? query method, which really just checks if the current_user method returns a value that’s not nil.

sign_in(@user) and sign_out(@user)

The sign_in(@user) and the sign_out(@user) methods. These are useful to login (or logout) a (newly created) user.

user_session

The user_session method, which returns meta data on a logged in user.
An application programmer will normally only need to use the current_user and user_signed_in? methods regularly. These, of course, are present as helper methods. So you can, for example, have a message in the header of your app stating the name of the signed in user.
Note that if you refer to an admin (who has a dedicated Admin class), the access method becomes current_admin, not current_user.
The prefix, or sometime suffix, is the corresponding underscored class name of a user class. For example, a user class name of ‘EndUser’, when underscored, becomes ‘end_user’, so the logged in user would in this situation be current_end_user.
Devise also defines a whole host of routes, which are defined in config/routes.rb with a line like:

1
devise_for :users
This line is automatically inserted by the Devise generator, and the default values it applies are fine for most needs. If not, it takes a complex options hash, that can be customized – mainly to prevent name collisions, or to have different route names, should you really want them!

Devise Modules

When you invoke the Devise generator, a model class (in `app/models*) is created for you to modify for your specific requirements.
This user model class is where many of the most important configuration options are specified. Perhaps the most important of these are the Devise modules to use. These modules provide enhanced security functionality. There are ten modules in all.
The modules are included thusly:

1
2
3
4
5
class User < ActiveRecord::Base
  devise :database_authenticatable, :registerable,
          :recoverable, :rememberable, :trackable, :validatable,
          :confirmable, :lockable, :timeoutable
end
Note that when Devise generates a class similar to this for you, it also generates a database migration, which includes the fields on which the functionality of these modules depend. This migration file is well commented and it specifies which fields relate to which modules. It should be modified if you want to trim down the number of fields attached to the model. If some modules are not required, the fields related to them can be removed.
A number of these modules have associated HTML forms and links. In the forms, an end-user typically fills in their email address, and in exchange they are sent back an email which contains confirmation links back to Devise controllers in your app, and include an authentication to token for security. These forms are mostly to (re-)enable a particular account.
Although the Devise README lists these options, we’ll explain them a little here as well.
  • database_authenticatable – ensures that a user enters a correct password, and it encrypts the said password, before saving it.
  • confirmable – ensures newly registered users confirm their accounts by clicking a link on a (Devise sent) mail they receive. This is a defence against bots creating fake, but valid, users.
  • recoverable – for handling forgotten passwords safely.
  • registerable – allows users to register themselves, and subsequently change their login credentials.
  • rememberable – for transparently logging on a user, based on a cookie. This is the functionality associated with the Remember me? checkbox you often see on login forms. Since the Devise config file specifies a timeout period, this is often of limited use. And is a security risk if the user steps away from their browser, with others about.
  • trackable – stores login information i.e. the user’s IP address, login time, last login time, and the total number of logins. These values are generally for site admins to look at, when trying to trace unusual activity.
  • validatable – ensures that the given email/name and passwords conform to a particular format.
  • lockable – limits the number of login attempts allowed before further access to a particular account is prohibited. Access restrictions are lifted after a time span, or if a (legitimate) user requests that an email be sent that contains a link to unblock his/her account.
These modules are lazily loaded, so if you omit some, the corresponding code is ignored.

Devise’s Global Configuration File

Devise has a very well commented configuration file, config/initializers/devise.rb. This is where its site-wide settings are specified.
These are things like the minimum password length, the password encryption algorithm, a flag to have a case sensitive username lookup, a timeout period, the ‘pepper’ string, and lots more as well. There is no substitute to actually taking a look at this file. If the options seem a bit bewildering at first, their exact purposes will become clearer on re-readings.

Protecting the Mass-Assignment of User Attributes

Naturally, all of the fields associated with a Devise user contain sensitive data, whose alteration could seriously compromise security. For this reason, it is probably a good idea never to allow your app to have pages (which are accessible to untrusted users) that update (or create) a user model. Only allow these attributes to be updated by Devise, where you know it’ll be done securely.
If an app’s requirements ordain that auxiliary user information be kept, such as an address or a phone number, these details are best kept on an entirely separate table which has a foreign key back to the (Devise generated) user table. This auxiliary table would have a one-to-one relationship with the Devise user table.
If you really must add fields to the (Devise generated) user model you must absolutely make sure that Devise’s fields cannot be changed. The means by which you achieve this is different for Rails 4 apps than it is for Rails 3 apps. In the former case, it’s with strong_parameters in the controller. In the latter case, it’s with the attr_accessible or attr_protected class methods in the model.
The Devise README explains this in adequate detail.

Adding Further Authentication

Having said all that in the preceding section, there is, arguably, an exception to this “no added user model fields” principle. This is where a user model has a flag like admin?, which is used to distinguish an app’s administator (super-user) from a regular end-user.
This is not a Devise added field, but one an app developer would add in a database migration. This field should not set up in a module or page of an actual app. Instead it should be set manually in a direct database command.
For example:

1
2
3
4
rails console
>>> user = User.find(1)
>>> user.admin = true
>>> user.save
This pattern is common, and is useful in apps where site administrators need to perform tasks in common with regular users.
In some other kinds of apps, the site administrators may need perform a completely different set of tasks to regular users. In this case it is cleaner to have separate classes for a site administrator and for a regular user.

Handling Different Types of Users in Devise

In this section, we’ll take a look at some ways in which some common use-cases, relating to access restrictions, can be handled with Devise’s controller class methods (macros). (This is an area where the distinction between authentication and authorization becomes blurred.)
These class methods are invoked as before filters, and take effect for a number of the controller’s available actions.
In this first case, suppose we want to ensure that every controller action requires a logged in user, except for the login and register actions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class ApplicationController < ActionController::Base
  # Ensures all actions invoke this (except those just below)
  before_filer :authenticate_user!
end

class AuthenticationController < ApplicationController
  # Turn off user authentication for all actions in this controller
  skip_before_filer :authenticate_user!

  def login
    '...'
  end

  def register
    '...'
  end
end
In this second case, suppose we want to grant logged in users full CRUD access, but unregistered only to have only read access for a selected group of controllers:

1
2
3
4
5
6
7
8
9
class CrudController < ApplicationController
  before_filer :authenticate_user!, except: [ :index, :show ]
end

class BlogsController < CrudController
end

class CommentsController < CrudController
end
In this third case, suppose we want to partition access for logged in regular users, and for site administrators, both of whom are represented by separate model classes. A regular user has a User class, and an admin has an Admin class. For example:

1
2
3
4
5
6
7
8
9
# All administrator controllers should inherit from this controller
class AdminController < ApplicationController
  before_filer :authenticate_admin!
end

# All end-user controllers should inherit from this controller
class EndUserController < ApplicationController
  before_filer :authenticate_user!
end
In this fourth case, suppose we want to have an end-user and a site administrator who both share the same user model. But, in this scenario, the site administrators are identified by an admin flag set to a true value on the model record. Whereas regular users have this flag switched off. For example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
class ApplicationController < ActionController::Base
  before_filer :authenticate_user!
end

# All administrative controllers should inherit from this controller
class AdminController < ApplicationController
  before_filter :ensure_admin!

  private

  def ensure_admin!
    unless current_user.admin?
      sign_out current_user

      redirect_to root_path

      return false
    end
  end
end
An added benefit of arranging the controller classes like this is that you can define private methods that are specific to a certain set of controllers. In the following example, all controllers that inherit from the CrudController will ensure that black-listed users are prohibited from further editing, but can still look at the site.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class CrudController < ApplicationController
  before_filer :authenticate_user!, except: [ :index, :show ]

  before_filer :block_abusers!, except: [ :index, :show ]

  private

  def block_abusers!
    if current_user.blacklisted?
      flash[:notice] = 'You are banned for being naughty!'

      redirect_to root_path

      return false
    end
  end
end

Devise Invitable Gem

If you want users to be registered through an invitation carried out by a site administrator, or by a group leader. (As an alternative to end-users explicitly registering themselves on provided web forms), there is a another gem to use, devise_invitable.
The Invitable gem generates a form in which a number of email addresses are filled in. Emails are sent out to the respective addresses inviting the users to confirm their registrations, which are actually performed by the site administrator.
As its name suggests, this gem must be used in conjunction with Devise. And it automatically links the invited users with the user who initially invited them.

Testing

It would be counter-productive to test Devise itself. It already has a comprehensive suite of tests that you are able run yourself, should you doubt its efficacy. However, you ought to test the parts of your app which interface with Devise, to ensure that Devise is presented with correct data, and to ensure that Devise remains in general working order.
Also, your own app’s tests are very likely to depend on your having a logged-in user, on whose behalf, the various operations of controller, acceptance, integration and view tests, are performed.
You can programmatically create a devise user as follows:

1
User.create!(email: "me@home.com", password: "watching the telly")
As an aside, creating users in this way is also useful in seed files in which you set up site administrators, etc.. Seeding data is done as part of a deployment step, so that you can have ready-made users as soon as your site goes live. The Rails’ provided file where you set up seed data is in db/seeds.rb, and you run it with the shell command:

1
$ rake db:seed
An RSpec example of a controller test that sets up a logged in user may go as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
describe BlogController do
  let :user do
    User.create!(email: "me@home.com", password: "watching the telly")
  end

  before { sign_in user }

  context "creating a post" do
    post :create, subject: 'matter', body: 'and soul'

    expect(assigns(:blog)).to be_instance_of Blog

    expect(:blog.try :subject).to eq 'matter'

    expect(:blog.try :user).to eq user
  end
end
Perhaps the above test has too many expectations, but it should give you a general idea, if you choose to support a suite of controller tests.
Though some might argue that the testing of such functionality would be better placed in integration tests. That, in general, controller testing is often superfluous, and has poorer code coverage, than integration testing does.
An RSpec example of an integration test could be as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
describe "user log in" do
  it "allows an existing user to sign in" do
    User.create!(email: "me@home.com", password: "watching the telly")

    visit "/users/sign_in"

    fill_in "Email", with: "me@example.com"
    fill_in "Password", with: "watching the telly"

    click_button "Sign in"

    expect(page).to have_content("Signed in successfully.")
  end
end
The above (smoke) test ensures that your app interfaces with Devise correctly, and in particular that, your view contains a form with the required URL and fields. If you wrote this test, you would probably write another one to test the registration process as well.

Wrapping up

If you have not done so already, your next step is to peruse the Devise README. which is probably the best single document on the practical use of Devise that’s readily available. If you want to do something out of the ordinary with Devise there’s a comprehensive list of howtos.
These two documents will, most likely, be the only ones you need to read in order to use Devise effectively.
Also if you would like more background information, take a look at the Devise wiki.

Thursday, August 4, 2016

Data Lake vs Data Warehouse

Great article is published by Martin Fowler:http://martinfowler.com/bliki/DataLake.html

But to quickly summarize, think of data lakes as data dump of raw unprocessed data that people can go back to and get the information that they might need from the raw contents

In Data Warehouse, someone would define a schema and will do some data cleaning and processing. This way, people can just grab a set from the warehouse and expect it to be a certain way and don't worry about manual processing.

So... if you don't know what you need and you don't mind looking for it, go with the lake
if you rather have sets of data ready to use for you in predefined format and you don't care about some potential missing data, go with the warehouse.

Git Aha's

To revert your changes back to the last working version, i.e. erase what you just did, use

git checkout .

To revert your changes to the version multiple commits in the past

Say you are developing a new feature and you realize after few commits that you went off to a way different route that you suppose to and you need to back it up few commits and start over... this definitely would be a cleaner way vs trying to remove what was done manually. How to do it though?

Check out what you want and get rid of all that code...

git reset --hard 0d3b7ac32

Then you would push it up

git push origin +master
 
 
Pretty simple once you know it.

To revert a single file to what's in current git use

git checkout HEAD -- <file_name>

To view the changes between staged and what's in current repo

git diff --cached

fffff

Friday, July 29, 2016

Git: Let's back it up

Say you are developing a new feature and you realize after few commits that you went off to a way different route that you suppose to and you need to back it up few commits and start over... this definitely would be a cleaner way vs trying to remove what was done manually. How to do it though?

Check out what you want and get rid of all that code...

git reset --hard 0d3b7ac32

Then you would push it up

git push origin +master
 
 
Pretty simple once you know it.

Thursday, July 28, 2016

Running rails server... iptables and binds...

Say you want to run your rail server... you type in the command and your rails application is up and running on localhost:3000

Great.

Now you want to access this server from other machine and suddenly you can't seem to access it? Proxy? Firewall?

Well, first things first: iptables. Make sure to add port 3000 to the list

Either edit iptables directly or just add to it. The choice is yours

Option 1:
sudo vi /etc/sysconfig/iptables

Option 2:
sudo iptables -I INPUT -p tcp --dport 3000 -j ACCEPT 
sudo service iptables save 
 
Now... when you start the server, by default it would bind to 127.0.0.1 address which is not accessing to the outside. Instead, you need to specify eth0 address that is externally accessible, like so

rails server -b <ip of eth0 - using ifconfig to find out what it is>

Hope this helps someone since i personally got stuck trying to figure out why my iptables were not allowing port 3000 to be seen when I had bind issues all along :)

Tuesday, July 26, 2016

Ruby on Rails: Bundle Install vs Update

Both can install gems that are specified in Gemfile, however bundle update would also update the gems to the latest version, if no version is specified in Gemfile, or will update it to the latest patch version (i.e. ~> 1.2.3 will update to the newest version of 1.2.x, it might be 1.2.4, 1.2.5, etc)

Bundle install on the other hand, would install the latest patch version of the gem if the gem is missing from the Gemfile, but won't do anything if it is already present.

So... bundle update might surprise you if something changed between patch version or make the gem incompatible with some other gem, hence caution must be taken with bundle install command.

And as a best practice, always specify versions in the Gemfile!