Skip to content

node.js and authentication

So – after creating a first node.js app with express templates and adding database access to the node.js tagsobe hotel travel solution, we need authentication.

Call me greenhorn – I have not been able to find a concise tutorial on adding simple form-based authentication to a node.js/express/sequelize setup. So here it is, and it actually is simple (hmmm – that might be a reason… ;-)):

Setup: We want to be catched if /booking is called with no one logged in. A Login form should be displayed, and when credentials are correct, the /booking URL should be called.

First, for the code below to work, you must use two modules:

  app.use(express.cookieParser());
  app.use(express.session({ secret: "type some fancy code here" }));

Now we start be securing /booking with an express-styled routing chain called loggedIn:

app.get('/booking', loggedIn, function(req, res){
  // Actually render the booking form
});

LoggedIn is as simple as:

function loggedIn(req, res, next) {
	req.session.user!=null
	    ? next()
	    : res.redirect("/login?url="+req.url);
}

So, it redirects to /login, providing the requested URL (/booking in our case), which simply renders a login form:

app.get('/login', function(req, res){
	res.render('login', {title: 'Login', url: req.param("url")});
});

The login form can be very simple, I include it here in Jade for completeness:

form(name="f",action="../authenticate",method="post")
 input(type="hidden",name="url",id="url",value="#{url}")
   fieldset
    legend Login Information
     p
      label(for="username") User:
      input(type="text",name="username",id="username")
     p
      label(for="password") Password:
      input(type="password",name="password",id="password")
     p
      button(id="submit",type="submit") Login

Obviously, we now need an /authenticate POST target – this is where sequelize is used to get Users from the database:

app.post('/authenticate', function(req, res){
	var query = Customer.find({ where: {username: req.param("username")} });
	query.on("success", function(user) {
		if (user!=null && user.password==req.param("password") {
			req.session.user = user.username;
			res.redirect(req.param("url"));
		}
	})
});

In success case, the session is populated and the user is redirected to the originally requested URL.

Of course, this is only a rudimentary solution – credentials should be encrypted, error handling added and so on… But it should give you some first advise. Code is here: https://github.com/joergviola/tagsobe-nodejs
Advertisements

node.js and databases

So – after creating a first node.js app with express templates, it is time to add database access to the node.js tagsobe hotel travel solution.

From this stack overflow, we decided to use sequelize:

$ npm install sequelize

We decided to perform all setup-related work directly in app.’s (perhaps not really a good idea…) First, we require the dependency:

var Sequelize = require("sequelize")

After that, we define the database connection as well as the entities:

var sequelize = new Sequelize('tagsobe', 'tagsobe', 'tagsobe', {
	  host: "localhost"
});
var Customer = sequelize.define('Customer', {
	username: Sequelize.STRING,
	password: Sequelize.STRING,
	name : Sequelize.STRING
}, {timestamps: false,freezeTableName: true});
var Hotel = sequelize.define('Hotel', {
	id: Sequelize.INTEGER,
	price: Sequelize.INTEGER,
	name: Sequelize.STRING,
	address: Sequelize.STRING,
	city: Sequelize.STRING,
	state: Sequelize.STRING,
	zip: Sequelize.STRING,
	country: Sequelize.STRING
}, {timestamps: false,freezeTableName: true});
sequelize.sync({force: true});

We prohibit the creation of additional createdAt and updatedAt columns and the pluralization of table names. The last line should actually create the tables. Restarting the server show no errors…. but wait – we did not even start the db server! Obviously, sequelize is not that good at error reporting?
Ok, so we go through the standard procedure of creating a MySQL database:

$ mysqladmin -u root create tagsobe
$ mysql -u root tagsobe -e "grant usage on *.* to tagsobe@localhost identified by 'tagsobe'"
$ mysql -u root tagsobe -e "grant all privileges on tagsobe.* to tagsobe@localhost"

Then we restart the server – et voila! The database tables are there:

mysql> explain Customers;
+-----------+--------------+------+-----+---------+----------------+
| Field     | Type         | Null | Key | Default | Extra          |
+-----------+--------------+------+-----+---------+----------------+
| username  | varchar(255) | YES  |     | NULL    |                |
| password  | varchar(255) | YES  |     | NULL    |                |
| name      | varchar(255) | YES  |     | NULL    |                |
| id        | int(11)      | NO   | PRI | NULL    | auto_increment |
+-----------+--------------+------+-----+---------+----------------+
Though not exactly as we defined – sequelize always adds an id column. Nevertheless we can import the seed data now:
mysql -u tagsobe -ptagsobe tagsobe < ???/import.sql
Now it is surprisingly simple to implement the hotel search:
app.get('/hotels', function(req, res){
	console.log("hotels invoked");
	var query = Hotel.findAll({where:["name like ?", "%"+req.param("searchString")+"%"]});
	query.on('success', function(result) {
		res.render('hotels', {title: 'Hotels', hotels: result});
	});
});
Here, for the first time, the inherently asynchronous nature of node.js and his friends show up explicitly: we have to add an event listener to the query which is called when the result from the database comes in. The result is then passed to the jade template, which looks quite well:
h1 Hotel Results
p
 a(id="changeSearchLink",href="search?searchString=a&pageSize=5") Change Search
p
div(id="hotelResults")
 table(class="summary")
  thead
   tr
    th Name
    th Address
    th City, State
    th Zip
    th Action
  tbody
   each hotel in hotels
    tr
     td= hotel.name
     td= hotel.address
     td= hotel.city
     td= hotel.zip
     td
      a(href="hotels/#{hotel.id}") View Hotel

That’s it! – Next, we have to add authentication.

node.js first try with express and jade

In order to get the currently most-praised high performance server side technology, we started to experiment with node.js.

The very first look at the project page reveals a whole ecosystem of libraries and possibilities. So the first task is to fix the concrete setup for the test. express and jade seem to be quite a commonly used combination. So let’s start!

We follow this excellent tutorial: http://www.nodebeginner.org/ which, for installation, immediately redirects here: https://github.com/joyent/node/wiki/Installation. We install from source since we’ll need to do this on the test machine also:

$ curl http://nodejs.org/dist/v0.6.6/node-v0.6.6.tar.gz > node-v0.6.6.tar.gz
$ tar zxf node-v0.6.6.tar.gz
$ cd node-v0.6.6
$ ./configure --prefix ../node
$ make
$ make install
$ PATH=$PATH:/Full/Path/to/node/bin
This lasts some time but ultimately node.js is up and running. Now install express:
$ npm install -g express
$ express nodejs
$ cd nodejs
$ npm install
 Watch how Jade is installed in this last step also. Now, time to start:
$ node app.js
Express server listening on port 3000 in development mode
 Point your browser to http://localhost:3000 and look at you first page. Where does it come from? We clarify that by adding our first page, the search form. First, we need to route “GET /search” to a javascript function rendering some template. Therefore, open app.js and find the statement app.listen(3000), which starts the previously configured server on port 3000. Right before, the route for the index page is added. So this is a good place to add our new route:
app.get('/search', routes.search);
 This statement registers method search of the routes modules to the URI /search. The method itself has to be added in the file routes/index.js, where you already find a function for the index page. So add:
exports.search = function(req, res){
  console.log("search invoked");
  res.render('search', {title: 'Search'})
};
Here the request is delegated to some template engine (jade in our case, because this is pre-configured in app.js). So create the file views/search.jade and enter:
h1 Search Hotels
 form(id="searchCriteria", action="hotels", method="get")
  span(class="errors")
  fieldset
   div
    label(for="searchString") Search String:
    input(id="searchString",name="searchString",type="text",value="")
   div
    label(for="pageSize") Maximum results:
    select(id="pageSize",name="pageSize")
     option(value="5") 5
     option(value="10") 10
     option(value="20") 20
    div
     button(type="submit") Find Hotels

div(id="bookings")
Now restart node and point your browser to http://localhost:3000/search and watch your first page! It is now a tedious, but simple work to convert the other pages to jade templates and add appropriate routes. Wait – we should take a look at database access first!

Apache http components

On starting testing on EC2, we noticed the test script suddenly requiring a lot of CPU-cycles. Since we want to run the test-script on the same machine as the server to reduce network influences, this is a problem.

Looking at some stack traces, quickly apache http components were identified.

Better insights, as always, were available by using the hprof java agent:

java -agentlib:hprof=cpu=times -jar dist/tagsobe.jar  ...

A lot of calls to verify X.509 certificates for ssl connections! After using our own SchemeRegistry, CPU load was reduced:

SchemeRegistry registry = new SchemeRegistry();
registry.register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
SingleClientConnManager connManager = new SingleClientConnManager(registry);
httpClient = new DefaultHttpClient(connManager);

hprof now showed java.net.SocketInputStream.socketRead0 as the by far most active method.

Performing a production test

This is the current semi-automatic procedure to perform a tagsobe production-quality test.

  1. checkout https://github.com/joergviola/tagsobe
  2. ec2-run-instances -t m1.small -k ec_pair ami-31814f58
  3. ec2-describe-instances | cut -f1,2
    take the instance id as $EC2INST, the hostname as $EC2NAME
  4. scp -i ~/.ec2/ec_pair.pem tagsobe/aws/install.sh tagsobe/aws/test.sh tagsobe/aws/.bash_profile ec2-user@$EC2NAME:
  5. ssh -i ~/.ec2/ec_pair.pem ec2-user@$EC2NAME
  6. chmod u+x *.sh
  7. install.sh
  8. test.sh <git-repository to test>
  9. exit
  10. ec2-terminate-instances $EC2INST
  11. check mail for results
  12. publish them on tagsobe

If you need to add another user:

  1. ssh -i ~/.ec2/ec_pair.pem ec2-user@$EC2NAME
  2. sudo adduser tester
  3. sudo su tester –
  4. ssh-keygen -t dsa (return to all questions)
  5. mv ~tester/.ssh/id_dsa.pub ~tester/.ssh/authorized_keys
  6. exit
  7. sudo visudo (tester eintragen)
  8. sudo cp ~tester/.ssh/id_dsa .
  9. sudo chmod a+r id_dsa
  10. exit
  11. scp -i ~/.ssh/ec_pair.pem ec2-user@$EC2NAME:id_dsa tester.pem

Send tester.pem to the other user, tell him to:

  1. save tester.pem
  2. chmod go-rwx tester.pem
  3. ssh -i tester.pem tester@$EC2NAME

Preparing a repository for testing

Of course you have to provide a web app being accepted by the test script.

But moreover you

  • have to submit it in a public git repository (github being a good choice)
  • provide an installation script

An example of the latter one can be studied here. What are the requirements?

First, the script has to reside in the root of the repo and be called install.sh. It has to be executable. And it has to take care that your app is available locally under some URL. Your script must store this URL in the file probe.url in its working directory.

The example looks like this:

mvn install
sudo cp target/travel.war $TOMCAT_HOME/webapps
sudo service tomcat6 start
sleep 10
echo "http://localhost:8080/travel/hotels/search" > probe.url

The example project is mavenized, so it is first built (installed) and then copied to the tomcat web app directory. Tomcat is one of the pre-installed components, see below.

Tomcat is then started. After a 10 second wait, the URL is written to the file mentioned and the script terminates, initiating the actual test.

Pre-installed components currently are (ask us – more components are likely to be installed):

  • git
  • Java SDK 1.6.0
  • Tomcat 6
  • ant
  • maven

Performance benchmarking – the basics

How do you measure performance of a web application?

Of course, you write a test script, applying a sequence of requests to the application and measure the response time. You also might do this several times sequentially to get better statistics and calculate the average.

Depending on the number of parallel users your application will show one of three behaviors. To understand them, let us take a quick look on how a request is process. Put simple, it is sent from the client to the app server, processed by the app server which in turn may ask a database server. The result page is rendered and sent back. Put even more simple, a number of resources are are allocated (network, app server cpu, db connection, db cpu and a lot more). The minimum response time M in a simple model is simply the sum of all the times where resources are allocated (assuming no parallelization).

The resources with the longest allocation time L plays a special role: Its allocation time – lets call it critical) determines the response time T. Assuming only one client can use a resource.

Depending on the number of clients C , the application will be in one of three states:

  • The critical response time times the number of clients is less than the overall response time. Each client can be server within the overall response time, which is thereby constant: T = M
  • The number of clients times the critical response time is higher than the overall response time – which thereby increases linearly with the number of clients: T = C*L
  • At a given load level, some resources will start to fail. For example, app servers typically put requests in a wait queue until the request can be served. This queue has a maximum length. If the queue hits this maximum, the server responds with an error

Performing a test in EC2

Assuming you already have an EC2 account, download the command line tools and follow their setup steps.

You were successful when you can describe the aws regions:

$ ec2-describe-regions
REGION eu-west-1 ec2.eu-west-1.amazonaws.com
REGION sa-east-1 ec2.sa-east-1.amazonaws.com
REGION us-east-1 ec2.us-east-1.amazonaws.com
REGION ap-northeast-1 ec2.ap-northeast-1.amazonaws.com
REGION us-west-2 ec2.us-west-2.amazonaws.com
REGION us-west-1 ec2.us-west-1.amazonaws.com
REGION ap-southeast-1 ec2.ap-southeast-1.amazonaws.com

Now pick an ami – we use ami-31814f58, the 32 bit amazon linux, and start an instance – attention, you are billed for the running instance!

$ec2-run-instances -t t1.micro -k ec_pair ami-31814f58
RESERVATION r-cda80aac 126880832672 default
INSTANCE i-da7ab8b8 ami-31814f58 pending 0 m1.small 2011-12-18T12:01:04+0000 us-east-1d aki-805ea7e9 monitoring-disabled ebs paravirtual xen sg-cf2f15a6 default

Check startup is complete:

$ ec2-describe-instances
RESERVATION r-cda80aac 126880832672 default
INSTANCE i-da7ab8b8 ami-31814f58 ec2-184-73-70-66.compute-1.amazonaws.com ip-10-8-39-81.ec2.internal running 0 m1.small 2011-12-18T12:01:04+0000 us-east-1d aki-805ea7e9 monitoring-disabled 184.73.70.66 10.8.39.81 ebs paravirtual xen sg-cf2f15a6 default
BLOCKDEVICE /dev/sda1 vol-ff0e6892 2011-12-18T12:01:23.000Z
Now allow ssh access on the default security group:
$ ec2-authorize default -p 22
 And now –  finally – you can logon to the system:
$ ssh -i ~/.ec2/ec_pair.pem ec2-user@ec2-184-72-77-115.compute-1.amazonaws.com
 Install git on the system:
$ sudo yum install git
Install the Java SDK:
$ sudo yum install java-1.6.0-openjdk-devel

Install Tomcat 6 (you’ll find it at /usr/share/tomcat6):

$ sudo yum install tomcat6-webapps
Download and install maven (unfortunately not via yum):
curl <a href="http://mirror.netcologne.de/apache.org//maven/binaries/apache-maven-3.0.3-bin.zip">http://mirror.netcologne.de/apache.org//maven/binaries/apache-maven-3.0.3-bin.zip</a> > maven.zip
unzip maven.zip

Now checkout the first spring project:

git clone git://github.com/joergviola/tagsobe-spring-jsp.git
And then build the project:
cd tagsobe-spring-jsp
../apache-maven-3.0.3/bin/mvn install
Start the tomcat service:
$ sudo service tomcat6 start
Now clone and build the tag browser:
$ sudo yum install ant
$ git clone git://github.com/joergviola/tagbrowser.git
$ cd tagbrowser
$ ant

Now perform the test:

$ java -jar tagsobe.jar 1,2,3 1000 http://localhost:8080/travel/hotels/search
1 14 3
2 16 5
3 20 6

Of course this result data should now be stored somewhere, e.g. by logging to a file and then mailing to yourself:

$ mail -s "Test results" you@yourself.org < log.txt

And finally terminate the instance:

ec2-terminate-instances i-da7ab8b8
INSTANCE i-da7ab8b8 stopped terminated
$ ec2-describe-instances
RESERVATION r-cda80aac 126880832672 default
INSTANCE i-da7ab8b8 ami-31814f58 terminated 0 m1.small 2011-12-18T12:01:04+0000 us-east-1d aki-805ea7e9 monitoring-disabled ebs paravirtual xen sg-cf2f15a6 default
Play around a little bit with your tests. Increase length of the tests and parallel client count. We soon encountered problems using a t1.micro instance, yielding random results above only 7 clients. We did not digg deeper but believe the reason lies in the memory size of only 613 MB, which had been excessed by the standard tomcat heap allocation (though we did not observe swapping on the instance).
Therefore, we now execute all tests in a m1.small instance. Maybe other instance types should be taken into account, too.

First try – Spring travel example

Being the reference for the use case of tagsobe, a natural first step was to compile the travel example from spring.

svn export https://src.springframework.org/svn/spring-samples/travel/ spring

The export is fast and places 1238 files and directories of 5.8MB on my disk. Phew – it is a maven project – so cd to spring and try

mvn install

In more than 7 minutes, this commands downloaded 174 libraries, before, with a relief, I read BUILD SUCCESSFUL. Out of the box!

The result is target/travel.war of 20MB size. To give it a second shot, I did

</pre>
rm -rf target

mvn install
<pre>

This time of course only 22 seconds were used. To go on, I copied the result to a freshly installed tomcat:

cd $TAGSOBE_BASE
unzip unzip apache-tomcat-7.0.23.zip
cp spring/target/travel.war apache-tomcat-7.0.23/webapps/
cd apache-tomcat-7.0.23/bin/
chmod u+x catalina.sh
./catalina.sh run
<pre>

Navigating the browser to http://localhost:8080, I was surprised seeing the application already running fine. What’s going on? – A quick peek into src/main/resources/META-INF/persistence.xml and …/META-INF/spring/data.xml (obvious, eh?) revealed hibernate was switched on an in-memory HSQLDB database.

We want an in-memory-database later, but surely a technology-agnostic one – SQLite, maybe.

For the moment, we stick with HSQLDB and check that the new app can correctly be benched by the test script (should be, because the test script was written against the online version of the app).

But oh! – an error occurred in the second step:

$ java -jar tagsobe.jar http://localhost:8080/travel/hotels/search
 GET /travel/hotels/search 0parse 324load
 GET /travel/hotels/%2Ftravel%2Fhotels%3Bjsessionid%3DDE26CE13099A918A1306921EB9056ECA?=Find+Hotels&searchString=a&pageSize=5 140parse 5load
java.lang.IllegalStateException: Response code: 400
 at org.tagbrowser.api.TagBrowser.request(TagBrowser.java:99)
 ...

WTF? Error Code 400? – Of course no trace of that in the apps logs. A closer look revealed, the code https://src.springframework.org/svn/spring-samples/travel does not exactly match the online sample http://richweb.springsource.org/swf-booking-faces/spring.

After changing the TestScript – a step that should not happen too often – to:

TagBrowser browser = new TagBrowser();
browser.setStatStream(System.out);
browser.open(url);
browser.submit(0, "a", "Find Hotels");
browser.clickName("View Hotel");
browser.submit(0, "Book Hotel");
browser.submit(0, "keith", "melbourne", "Login");
browser.submit(0, "12-01-2041", "12-02-2041", "false", "OCEAN_VIEWdfg",
"OCEAN_VIEWdgf", "OCEAN_VIEW", "1111222233334444",
"KEITH MELBOURNE", "1", "1", "2010", "Proceed");
browser.submit(0, "Confirm");
browser.contains("Current Hotel Bookings");

the example could be scanned successfully:

GET /travel/hotels/search 0parse 376load
GET /travel/hotels;jsessionid=41F09BE304E905D1A7129CEA28154E7B?=Find+Hotels&searchString=a&pageSize=5 119parse 9load
GET /travel/hotels/1 8parse 6load
GET /travel/users/login 3parse 19load
POST /travel/hotels/booking?execution=e1s1 5parse 41load
POST /travel/hotels/booking?execution=e1s2 9parse 31load
POST /travel/hotels/search 5parse 36load