Load More – Node.js Messaging Server (Part 3)

Wayback Machine

OCTNOVDEC14Next capture201520162017
29 captures11 Sep 2012 – 14 Nov 2016

 About this capture


Load More – Node.js Messaging Server (Part 3)

September 6, 2012

Part 2 was introduction to Node.js and Socket.io. In this Part we will develop Twitter like functionality using Node.js and Socket.io to load new messages without requesting or pooling the server. I have given enough introduction to what we are going to develop in this part. Those who want to review can read the third paragraph from Part 2.

Previous Parts

Source Files

Installing MySQL Module

Our Socket.io based server will require connectivity with MySQL database. So before creating the server application, let’s install MySQL driver for Node.js. Use npm (Node Package Manager) to install MySQL module.

npm install mysql@2.0.0-alpha3

At the time of this writing, it was in Alpha 3 stage so the module name has postfix alpha3.

Real-time Node.js Server

Basic Connectivity

Now create a new js file for server and add the following code in it.

var io = require( ‘socket.io’ ).listen( 8000 ) , TIMEOUT = 5000;   io.sockets.on( ‘connection’, function( socket ) { console.log( ‘Client connected’ );   socket.on( ‘disconnect’, function() { console.log( ‘Client disconnected’ ); });   });

First line gets socket.io module and immediately calls listen method which returns a server object that’s listening to port 8000. This object is then assigned to io variable which will be used to add events.

On second line, a constant TIMEOUT is defined. This constant is time in milliseconds. We will check the database for new records/messages after every 5 sec (5000 ms). Obviously in production you must use 60 sec or more to reduce the load on database server.

Then a connection event is added to io object using on method. Every time a new client connects to this server, connection event is fired. A new socket object is passed to the callback for this event which is unique for every client.

Within the callback console.log method will write 'Client connected' on the server’s console. As I mentioned before, connection event will fire for every incoming client. So you will see the message 'Client connected' for every new client.

Again within the connection callback, disconnect event is added to socket object. This event will fire for every client that disconnects from the server. The callback for this event will write 'Client disconnected' on the server’s terminal.

Last Message Event

As soon as client connects to this server (client connection code will be shown later below), it will fire lastDateEvent.

io.sockets.on( ‘connection’, function( socket ) { //code…   socket.on( ‘lastDateEvent’, function( data ) { socket.lastMsgDate = new Date( data.lastMsgDate ); setTimeout( sendUpdate, TIMEOUT, socket ); });   //code… });

lastDateEvent event is fired from client to start the function that checks for new messages periodically in database because we need the posted date of most recent message that client have. So in the callback of lastDateEvent, data parameter is received which contains this date.

In the first line of callback function, new property is added to socket (parameter received from connection callback) object and assigned a Date object containing the posted date of most recent message. This date is stored in socket because it will be used later to get the messages posted after this date. Also socket object is always different for each client so every client will have its own message posted date stored in this client specific socket object.

On the second line of callback, sendUpdate function is executed using setTimeout (which will execute it after TIMEOUT milliseconds). Also note the last parameter of setTimeout is socket object which will be passed as parameter to sendUpdate.

Periodic Message Checker

Within the body of sendUpdate function, it will check for new messages in database and send it to client (if any). Then this function will call itself again using setTimeout after TIMEOUT milliseconds. Now define sendUpdate function right below connection event.

io.sockets.on( ‘connection’, function( socket ) { //code… }); //end connection   function sendUpdate( c ) { //*** db call will go here ***//   c.lastMsgDate = new Date(); setTimeout( sendUpdate, TIMEOUT, c ); } //end sendUpdate

The parameter c (client) is actually the socket object passed from lastDateEvent callback. On the first line a DB call will be added later that will check for new messages based on date c.lastMsgDate. Then the c.lastMsgDate is reinitialized to current date because after the DB call, posted date of most recent message may change (if newer messages are added in database). Finally sendUpdate is called again using setTimeout and c is passed as parameter for sendUpdate.

Now we need to create a custom Node.js module that will provide APIs to access MySQL database.

Database Module

To create a custom Node.js module, save new loadMoreDB.js file within the same directory where server file is stored. First some private data variables need to be defined.

var mysql = require( ‘mysql’ ) , loadMoreDB = exports;   var mysqlCon = mysql.createConnection({ host: ‘localhost’, user: ‘username’, password: ‘password’, database: ‘test’ });

First variable imports MySQL module which will be used for communicating with the database. Then a special exports object is assigned to loadMoreDB variable. Now loadMoreDB is an alias to exports. Both refer to exactly the same object.

NOTE: Remember that any method which you want to make public (accessible from client using this module) must be assigned as a property to exports object.

Then MySQL connection is created using mysql.createConnection method. Change userpassword, and database properties of the object passed as parameter.

Now define a private function that converts JavaScript date and returns MySQL date as string.

function dateToMySqlTimestamp( dateObj ) { return dateObj.getFullYear() + “-” + ( dateObj.getMonth() + 1 ) + “-” + dateObj.getDate() + ” ” + dateObj.getHours() + “:” + dateObj.getMinutes() + “:” + dateObj.getSeconds(); }

This method is taken from my older post Common JavaScript Date Operations.

Next connect with MySQL database using mysqlCon.connect method.


This is the coolest part in Node.js when dealing with databases. You only need single connection or pool of limited connections to database for every request from any client. The above statement will connect to database as soon as it gets executed. No more connections will be made later and also this connection will not close until the server itself is killed. This is a huge performance gain over other server side technologies like PHP, or Ruby

Here are some interesting discussions on Stack Overflow about Node.js and MySQL.

Now create a public method that gets new messages according to posted date of previous message.

loadMoreDB.getNewMsgs = function( callback, params ) { mysqlCon.query( ‘SELECT m.msg_id, m.msg_by, m.msg_txt, m.msg_date ‘ + ‘FROM msgs m ‘ + “WHERE m.msg_date > ” + mysqlCon.escape( dateToMySqlTimestamp( params.lastMsgDate ) ) + ” ” + ‘ORDER BY m.msg_date’   , function( err, rows ) { if( err ) { console.log( err ); callback( [] ); } callback( rows ); } ); };

Above method getNewMsgs accept two parameters callback and paramscallback is the function that will be called with rows passed as parameter to it after query execution. params is an object containing additional parameters like lastMsgDate that we will pass from sendUpdate function later.

There’s a single method call mysqlCon.query in the body. The query method executes SQL query passed as first parameter and it’s async method so we need to pass callback as second parameter which executes after query execution. Note in WHERE clause, mysqlCon.escape method is used to escape lastMsgDate to prevent SQL injection attacks.

In the callback which is passed to query method, err parameter (contains exception in case of error) is printed on terminal if it’s not null and a blank array is passed to callback function of client using getNewMsgs method. Otherwise client’s callback is called with rows passed to it.

Using the Database Module

Coming back to our Node.js server file created above. Database Module must be added using require statement to use it. Here is the modified code.

var io = require( ‘socket.io’ ).listen( 8000 ) , loadMoreDB = require( ‘./loadMoreDB’ ) , TIMEOUT = 5000;

Now getNewMsgs method is accessible from loadMoreDB object. Simply call it in sendUpdate function defined above.

function sendUpdate( c ) { loadMoreDB.getNewMsgs( function( msgs ) { if( msgs.length > 0 ) { c.emit( ‘newMsgsEvent’, msgs ); } }, { lastMsgDate: c.lastMsgDate } );   c.lastMsgDate = new Date(); setTimeout( sendUpdate, TIMEOUT, c ); } //end sendUpdate

First parameter for getNewMsgs is anonymous callback function which is called with rows (msgs from database) passed to it. Within the body of this callback, msgs (if any) received from database are send to client using event newMsgsEvent generated using c.emit.

Second parameter passed to getNewMsgs is an object containing lastMsgDate.

Back to Client Side

Our Node.js server is complete. Now we need to consume it in our JavaScript/Knockout.js based client which actually resides in view msgs.php of our PHP/CodeIgniter application.

Refactor Previous Code

Before creating the client consumer, we need to refactor and add new code. First within the MsgViewModel, change self.msgs from observableArray to an object and assign that observableArray of messages to self.msgs.list.

self.msgs = {};   if( _msgs !== null ) { self.msgs.list = ko.observableArray( ko.utils.arrayMap( _msgs, function( msg ) { return new Msg( msg.msg_by, msg.msg_date, msg.msg_txt ); }) );   self.msgs.firstMsgDate = Utils.parseMySqlDate( _msgs[ 0 ].msg_date ); } else { self.msgs.list = ko.observableArray( [] ); self.noMsgVisible( true ); self.msgs.firstMsgDate = null; }

You may have noticed the second change is the addition of self.msgs.firstMsgDate property which contains the posted date of first message in _msgs array. firstMsgDate will be passed to Node.js server via lastDateEvent as lastMsgDate. Actually firstMsgDate is the last message’s date in the database because query get records using ORDER BY m.msg_date DESC.

Now add a new method appendTo in self.msgs object.

self.msgs.appendTo = function( msgList, start ) { if( start === true ) { msgList.forEach( function( elem ) { self.msgs.list.unshift( new Msg( elem.msg_by, new Date( elem.msg_date ), elem.msg_txt ) ); }); } else { msgList.forEach( function( elem ) { self.msgs.list.push( new Msg( elem.msg_by, elem.msg_date, elem.msg_txt ) ); }); } }

Above method is used to append new messages received from server (PHP or Node.js) to self.msgs.list. It appends to the start of msgs.list using unshift method if start parameter is true. Otherwise it appends to the end using push method.

Now within the closure that we created and assigned to self.moreBtn, there’s an Ajax call in method loadMsgs for loading new messages. Replace the loop in success part of Ajax call that append new messages with the newly created method appendTo as shown below.

$.ajax({ url: “msgs/getMsgs”, type: “POST”, data: { lastMsgDate: Utils.getMySqlTimestamp( _lastMsgDate() ) }, dataType: “json” }).success( function( response ) { if( response !== null ) { self.msgs.appendTo( response, false ); _lastMsgDate( Utils.parseMySqlDate( response[ response.length – 1 ].msg_date ) ); }   self.moreBtn.disable( false ); });

Show New Messages Button

Show New Messages Button

A new button is required on top of messages which will automatically be displayed on retrieval of new messages from Node.js server.

<div id=”body”> <button id=”showNew” class=”btn” style=”display: none” data-bind=”click: showNewBtn.click, visible: showNewBtn.visible, text: showNewBtn.txt”>2 new messages…</button>   <ul data-bind=”foreach: msgs.list” class=”msgs”> <!– rest of the code… –>

Bindings for clickvisible, and text are associated with respective properties in showNewBtn object which is shown below.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24//code…   self.showNewBtn = { totalNewMsgs: 0, newMsgs: [], txt: ko.observable( ” ), visible: ko.observable( false ),   click: function() { self.showNewBtn.visible( false ); self.msgs.appendTo( self.showNewBtn.newMsgs, true ); self.showNewBtn.totalNewMsgs = 0; self.showNewBtn.newMsgs.length = 0; },   notify: function( data ) { self.showNewBtn.totalNewMsgs += data.length; self.showNewBtn.txt( self.showNewBtn.totalNewMsgs + ‘ new messages…’ ); self.showNewBtn.visible( true ); self.showNewBtn.newMsgs = self.showNewBtn.newMsgs.concat( data ); } };   } //end MsgsViewModel

Above code is added at the end of MsgsViewModelclick and notify methods are important.

You will see later that the new messages received from server will be passed to notify method (as data parameter). On line 17, length of new messages is added to totalNewMsgs property. On line 18, label of button is changed according to the count calculated in totalNewMsgs. On line 19, the button is made visible and finally new messages in data parameter are concatenated with newMsgs array and assigned again to newMsgs.

The click method is also note worthy. Its functionality is to hide showNewBtn, append newMsgs to observable msgs array, reset totalNewMsgs to zero, and empty newMsgs array by assigning it length of zero.

Communicating with Server

Let’s create an object which will connect with Node.js server and contain event handlers and methods for communicating with server.

function ClientSocket( evtAggregator ) { var socket = io.connect( ‘’ );   socket.on(‘connect’, function() { console.log(‘Client connected to the Server!’); });   socket.on( ‘disconnect’, function() { console.log(‘Client disconnected from Server!’); });   socket.on( ‘newMsgsEvent’, function( data ) { evtAggregator.notifySubscribers( data, ‘newMsgNotify’ ); });   this.emitLastDate = function( msgDate ) { socket.emit( ‘lastDateEvent’, { lastMsgDate: msgDate } ); }; }

The parameter received in ClientSocket constructor is evtAggregator (Event Aggregator) which is used for communication between objects in a loosely coupled manner. Actually it’s nothing more than a pub/sub pattern. You will clearly understand it when we call ClientSocket from jQuery’s DOM Ready handler.

The first line in the body of ClientSocket will connect to Node.js server that’s listening to port 8000. Then there are three event handlers. First is connect which fires when connection with server is made. Second is disconnect which obviously fires when client disconnects from server. Third and last event is newMsgsEvent which fires when newly added messages are received from server.

Then there’s a public method which when called will fire lastMsgDate event on server. Also the date of last message is passed to server.

DOM Ready

Now we have MsgsViewModel and ClientSocket which requires communication using Event Aggregator. So jQuery’s DOM Ready from Part 1 requires modification to include these changes.

1 2 3 4 5 6 7 8 9 10 11 12$( function() { var msgsVM = new MsgsViewModel() , evtAggregator = new ko.subscribable() , cs = new ClientSocket( evtAggregator );   ko.applyBindings( msgsVM );   evtAggregator.subscribe( msgsVM.showNewBtn.notify, msgsVM, ‘newMsgNotify’ ); evtAggregator.subscribe( cs.emitLastDate, cs, ’emitLastDate’ );   evtAggregator.notifySubscribers( msgsVM.msgs.firstMsgDate, ’emitLastDate’ ); });

The first statement declares three variables. Variable msgsVM which contains instance of View Model MsgsViewModel. Second is evtAggregator (Event Aggregator) which contains object that supports Pub/Sub pattern. Note that ko.subscribable is used internally in Knockout.js to support bindings between controls and View Model. Final variable is cs which contains instance of ClientSocket.

Then bindings between controls and View Model are applied using ko.applyBindings.

Now the next three lines are important. evtAggregator.subscribe is used to subscribe a method (first parameter) that exist in an object (second parameter) to an event (third parameter). So on line 8, notify method that exist in msgsVM is subscribed to newMsgNotify event. Now in other objects, we don’t need to call msgsVM.showNewBtn.notify method directly. Instead we can use evtAggregator.notifySubscribers to fire newMsgNotify event which will automatically call notify method. All of this is done to make loose coupling between objects which call methods of other objects. Our code will not break even if the method that’s called is removed, renamed or parameters are modified.

Similar subscribtion is made on line 9 for emitLastDate method. Finally notifySubscribers method is called to fire emitLastDate event and the first parameter firstMsgDate is actually the parameter for method that’s subscribed to emitLastDate event.

Hybrid Architecture

This Load More application is an example of hybrid architecture where Front End is created using HTML, CSS, JavaScript, Knockout.js. The Front End code communicates with PHP/CodeIgniter application on server to get new messages on demand (when user clicks more… button in UI). Finally to access the newly added messages in real-time is done with Node.js server.

Favorite Open Source Softwares

Most of the Open Source softwares are much better than their proprietary alternatives. So I prefer to look for Open software before going for any

Interview Experience With Google

ecently I got a chance to give an interview to Google for Front-End Software Engineer. It was a nice experience and I learned a lot