For this first post I would like to write about a chat project I started a few years ago. What does it actually do? (Here we go for the buzz words)
- Real time chat with Socket-io using the new pubsub feature backed up by redis
- Streaming Upload in MongoDB GridFs
- MVC layer over Express
- Linearly scalable architecture ?
You can try the live demo here.
1 – Chat
A chat application is the probably the simplest way of testing a distributed push application. Messages are passed from one to many like it could be in a video game or any multi-user interactive application. Thus once you have done your chat application and tested it’s performance you can confidently build your next-gen video game.
But making a performant distributed message passing application is not the simplest job to accomplish.
So, why node.js ?
I restarted the project last year and wanted to keep up with all the new technologies which had grown with node.js. I replaced my long polling by socket-io and it worked pretty well!
2 – File sharing
My first goal was to build a realtime imageboard where users could share images, videos and more. So basically a chat with file sharing capability.
To make it harder (and cooler ;)) I wanted to access the file stream directly and make it available to all users as it was uploaded. Making this work on a large scale is a difficult problem because we need to replicate file chunks as soon as they are sent to a node. I trusted an open source distributed filesystem available with MongoDB called GridFS. This latter enables you to store and retrive files in your MongoDB cluster.
I wrote a small middleware for node.js to access chunks of data when uploading a file along with a library to easily store growing files into MongoDB.
3 – MVC Layer
I wanted to see if it makes sense to follow the MVC pattern while using an evented library like node.js. For this I wrote my own little framework to :
– Autoload controllers, models and views
– Have a django style urls.js file which defines the routes
– Load a config.js file for configuration preferences such as database host, view engine or session distributed cache engine
– Define socket.io sockets mapping with the routes
I found it hard to stuck to my MVC framework implementation when integrating socket.io because it doesn’t use HTTP routes and sessions handled by Express. I choosed to write a bit of glue code to automatically load the corresponding client’s HTTP session from the session store. I haven’t load tested it but I’m afraid it can really impact performance.
To be honest, it’s not as clean as others Node.js MVC frameworks but I have the avantage of knowing the code and beeing able to tweak it to my needs.
4 – Scalability
For an application to run in the cloud you need to export everything statefull to a service known to run in cluster. I chosed to use MongoDB to save messages and files.
The problem is, file sharing requires a lot of ressources which I could not afford. I wanted to make a free or ad-paid service but the cost of bandwidth and storage involved by deploying on Amazon EC2 instances was too high. So I decided to deploy it on normal servers with unlimited download capability (see OVH offers). Then came the devops part of the project which I probably underevaluated and is the main reason I did not bring this project to the public. This will be the subject of another post!