Tag Archives: nodejs

Some life updates…

So I’ve been pretty busy with, you know, life but who hasn’t. Actually I’ve been trying to write another blog post after my last one which was sort of a post but not really (just a generic generated past-year result the WordPress offers, for people busy (a.k.a lazy) such as myself.

So far life has been pretty good. My last year performance got me a promotion a few months ago and ever since I got to be sort of Tech Lead for the Web & Mobile Team, which is in charge of all the front-end webapps, including both internal & client-facing. I’d say dealing with this domain has been very interesting as it’s something I have wanted to do for a long time, until I took an arrow… w/e it is still very fun, being the guy who can work on both back-end and front-end and sometimes applying design patterns interchangeably. Some of those have failed miserably due to differences in the nature of the 2 domains (1 being a beefy powerhouse serving large data in a timely manners to all sorts of requests, 1 being a lightweight routing layer with some minimal caching & compression of resources).

A change in domain also means a change in technology as well. Getting myself better with NodeJS, Backbone, jQuery and now Bootstrap has been really challenging, all of which I’ll write another post about later. Ideally I’d like to push for the stack to be more well-used within the team myself. This stack can easily be distributed and scaled horizontally. However we’re not really dealing with a consumer-product business the scalability of the UI sometimes is not a development bottleneck.

Other than work stuff, I got an email this morning, which kinda cheered me up a lil bit:

Hi Long,

I came across https://longho.wordpress.com/ while searching for resources around recipes and food and was wondering if this is the correct contact in regards to the content on the site. My team just created a graphic on the topic, would you be interested in taking a look? I’d appreciate any feedback.

Thanks in advance.

Peter

So someone actually reads my blog! And feedback. Based on the statistics I know someone reads my blog but normally it’s search query from Google and those can be highly irrelevant. I actually visited their site and the statistics about ramen, IMO, is indeed very interesting. Surviving on ramen has been 1 of the most important life skills in college. My Japanese neighbor was obsessed over ramen. I mean the dude was literally chewing through packs of those…

In case you guys wondered what he sent me, here it is:
We Love Ramen Infographic
Created by: Hack College

Pretty amazing isn’t it?

Tagged , , , , , , , , , ,

What stuff I’ve been working on lately

So it’s been a while since my last blog post, partially because I was swamped with work and some personal business, but mainly because my friend Haruki and I are still trying to design and set up the 1st phase of the MetaDB project. It is currently being actively developed right here on GitHub. We’ve put quite some thoughts into the data model of the project, which lead to several architecture and technology choices.

We’re still using NodeJS as the meat of the whole project. It has been the initial choice since the beginning due to various (unverified) reasons of speed, both in performance and development, scalability and lightweightness. I said unverified cause we’ve read a lot about it but technologies are sometimes YMMV kinda thing and for our specific use cases, NodeJS seems to be a good fit.

PostgreSQL-9.0.1

PostgreSQL-9.0.1

We dropped the idea of using NoSQL to using PostgreSQL as the database technology. NoSQL is great for unstructured data but there seems to be way too many relationships among our model objects that maintaining a NoSQL model and doing map-reduce just doesn’t seem to be worth it. For NoSQL if we separate some the components then hybrid objects would be a result of a map-reduce instead of a join, which turns out to be definitely not as efficient. If we store those components inside a certain object collection, the reusability would be a big mess. In the end, partially due to our lack of a solid NoSQL modeling skills, we took the easy way out which is SQL.

The framework that drives our API would be a custom in-house developed module called njrpc (Node-JsonRPC). It’s an implementation of the Json-RPC 2.0 protocol with some additional bells & whistles that allow you to do namespacing and interceptors of requests. It also exposes enough low-level calls (at least to our needs) that you can do manual response override in callbacks and such.

Part of my purpose for this blog post is also to share our development set up, mainly for experiments with distributed system and workflows.

Prod environment:

1. For our production environment, Haruki & I both have our VPSes, each of which has an instance of PSQL running and replicating master-slave. Configuring this took a while which we’ll document later but basically it’s running right now.

2. Each of the boxes would also have an instance of metadb-core serving API calls. A simple load-balancer (HAProxy for ex) will be placed on 1 box and will serve as the entry point of all API calls. This does produce an somewhat unpredictable response time for the calls but the tradeoff is redundancy, which is definitely needed for prod env.

3. UI will be set up on 1 of the boxes. It’s really lightweight right now so we don’t immediately see the need of having 2 UIs running.

4. We still have to set up database backup and archive, u know, disaster recovery stuff.

Test environment:

We’re planning to get another smaller VPS instance for our CI server (Continuous Integration). This pretty serves as the integration testing environment for both metadb-core & metadb-ui. Although njrpc is currently set up in Travis-CI, using a 3rd-party CI doesn’t allow us to do some customization and setup. Travis-CI uses RoR and allows testing of NodeJS projects but there’s version skew and database setup and all that. It’d be much less painful to have our own box dedicated to testing.

Development environment:

1. IDE: I actually use cloud9 running locally as my IDE. It doesn’t have code auto-complete and stuff but the syntax hi-lighting and JsHint are pretty decent and helpful. Interface is simple and lightweight enough.

2. Dev environment is pretty much a replicated of prod/test env so it definitely needs PSQL and NodeJS. We also maintain a separate test database with a much smaller set of data so that we can easily wipe out and dump it back in for a fresh copy.

That’s pretty what we have in mind… so much for 2 developers. It’s very time-consuming but rewarding at the same time as I’m getting much better at handling async stuff in JS.

Aight guys, have fun and keep on brogramming! Oh BTW we got our VPS from AlienVPS. They have pretty decent pricing.

Tagged , , , , ,

Yeah so my audio player in jQuery Mobile and NodeJS (part 2)

Ok so my prev post was about how to construct a playlist in jQuery Mobile with some simple NodeJS file serving. This one is to construct the audio player itself! Mine looks like this (cause I was kinda too lazy to do the styling properly):

My audio player

My audio player

Ok so it’s very basic: we got a toggle Play/Pause button, Next button, Prev button, the progress bar for download progress and a fake album art (cause I didn’t know how to extract mp3 metadata yet). The features that I implemented are pretty basic:

1. Play/Pause

2. Next/Prev Song

3. Progress bar for song buffering

4. Time Left

5. Auto-play the next one if this one ended

6. Header shows song name

The HTML structure itself is rather simple as jQuery Mobile does most of the styling for u:


<div data-role="page" class="player">

    <div data-role="header">
        <h1>My collection</h1>
    </div><!-- /header -->

    <div data-role="content">

	<div class='cover-art' style='text-align:center'>
		<audio src='music/kpop/ttl2.mp3' preload autoplay></audio>
		<img src='images/no-album-art.png' />
	</div>
    </div><!-- /content -->
    <div data-role='footer' style='text-align:center'>
	<p class='track-info'>
		<span class="song-progress">
			<input type="range" min="0" max="100" value="0" />
		</span>
		<span class="timeleft"></span>
	</p>
	<div class='playback' data-role="controlgroup" data-type='horizontal' style='text-align:center'>
<button class='playback-prev' data-icon='back'>Prev</button>
<button class='playback-play' >||</button>
<button class='playback-next' data-icon="forward" data-iconpos="right">Next</button>
</div>
	</div>
<script type="text/javascript">
	$('div.player').bind('pageshow', function(ev, ui) {
		if (!$(this).attr('data-init')) {
			Player.init('div.player.ui-page-active', $.getUrlVar($(this).attr('data-url'), 'song'));
			$(this).attr('data-init', 'true');
		}
	});
</script>
</div><!-- /page -->

So again, I bind some initialization to the “pageshow” event of the page and make sure it doesn’t get initialized twice. Since the href in each <li> points to the same page but different parameter, jQuery loads this again every single time even if it’s the same one. This only prevents the forward history button to reload the song. However, this does not prevent having multiple songs playing at the same time cause jQuery mobile loads those as different div. You can customize the changePage behavior when user clicks on a <li> but I didn’t, just to keep it simple.

The parameter is stored in the main player div (with selector “div.player”, class “ui-page-active” indicates its the active one) so $.getUrlVar just extract the parameter song from it (which indicates the song index):

$.extend({
	getUrlVars : function(string) {
		var vars = [];
		var hash;
		var href = string ? string : window.location.href;
		console.log(href);
		if (href.indexOf('#') > -1) {
			var hrefArr = href.split('#');
			href = hrefArr[hrefArr.length - 1];
		}
    		var hashes = href.slice(href.indexOf('?') + 1).split('&');
		for (var i = 0; i < hashes.length; i++) {
			hash = hashes[i].split('=');
			vars.push(hash[0]);
			vars[hash[0]] = hash[1];
		}
		return vars;
	},

	getUrlVar : function(string, name) {
		return $.getUrlVars(string)[name];
	}
});

Pretty simple, just basically splitting the data-url field into a map of parameter names and values. The Player.init function takes in the parent div selector (so that I can locate the control relative to the parent div) and the song index. I basically keep track of all the control DOM elements:

var $next = $(div + ' button.playback-next');
var $prev = $(div + ' button.playback-prev');
var $play = $(div + ' button.playback-play');
var $trackInfo = $(div + ' p.track-info');
var $songProgress = $trackInfo.find('.song-progress');
var $loading = $songProgress.find('.loading');
var $timeLeft = $trackInfo.find('.timeleft');
var $slider = $songProgress.find('.ui-slider');
var $handle = $slider.find('.ui-slider-handle');
$songProgress.find('input[type="number"]').hide();
var $title = $(div + ' h1.ui-title');
var $audio = $(div + ' audio');
var audio = $audio.get(0);

I have this habit of prefixing jQuery objects with $ to distinguish from actual DOM element ($audio is the jQuery-wrapped object of audio). Play/pause is really easy:

$play.click(function(ev) {
    var $buttonText = $(this).parent().find('.ui-btn-text');
    if (audio.paused) {
        $audio.attr('data-state', 'play');
        audio.play();
        $buttonText.text("||");
    }
    else {
        $audio.attr('data-state', 'pause');
        audio.pause();
        $buttonText.text("Play");
    }
});

Prev/Next is also straightforward:

$next.click(function(ev) {
    var state = $audio.attr('data-state');
    var current = parseInt($audio.attr('data-current'));
    Player.getSongPath(current + 1, $audio, $title, function() {
        $audio.attr('data-current', current + 1);
        if (state == 'play') {
            audio.play();
        }
    });
});
$prev.click(function(ev) {
    var state = $audio.attr('data-state');
    var current = parseInt($audio.attr('data-current'));
    Player.getSongPath(current - 1, $audio, $title, function() {
        $audio.attr('data-current', current - 1);
        if (state == 'play') {
            audio.play();
        }
    });
});

So we’ve done 1 and 2. Let’s jump to 5 cause its also easy:

$audio.bind('ended', function(ev) {
    $next.click();
});

I did 6 as a separate functionality that ping the server for the song’ path, change the audio source and also title:

getSongPath: function(index, $audio, $title, fn) {
    $.post('playlist?song=' + index, null, function(data) {
        console.log(data);
        $audio.attr('src', data.result);
        var filenameArr = data.result.split('/');
        var filename = filenameArr[filenameArr.length - 1];
        $title.text(filename);
        if ($.isFunction(fn)) {
            fn();
        }
    }, 'json');
}

For some reason I put null in the POST request data instead of the actual data (song=2) cause I wasn’t getting that data on the server side (I tried req.body, req.query and everything… didn’t seem to show up, will look into it a bit more). OK now lets get back to 3:

if (!$loading.get(0)) { //this inject the white loading bar before the handler
    $handle.before('<div class="ui-slider loading" style="width: 3%; float: left; top: 0; left: -3%; background-color: buttonface;"></div>');
    $loading = $slider.find('div.loading'); //update var
}
$audio.bind('progress', function() {
    var loaded = parseInt(((audio.buffered.end(0) / audio.duration) * 100) + 3, 10);
    $loading.css({
        width: loaded + '%' //change width accordingly
    });
});
var manualSeek = false;
var loaded = false;
$handle.css({
    top: '-50%' //somehow I think the styling of footer and handler conflicted and messed it up so I had to bump it up 50%
});

I actually didn’t know how to get the current time of the audio but after googling around and looking at audio attributes, things got a bit clearer. Here’s 4:

$audio.bind('timeupdate', function() {
    var rem = parseInt(audio.duration - audio.currentTime, 10),
        pos = Math.floor((audio.currentTime / audio.duration) * 100),
        mins = Math.floor(rem / 60, 10),
        secs = rem - mins * 60;
    $timeLeft.text('-' + mins + ':' + (secs > 9 ? secs : '0' + secs));
    if (!manualSeek) {
        $handle.css({
            left: pos + '%'
        });
    }
    if (!loaded) {
        loaded = true;
    }
});

Ok so that’s how I made a sorta functional audio player. There’re still problems with it but hopefully this DIY guide gave u some idea on how to control the audio element manually.

Tagged , , , , , , ,

What I’ve been up to… (a.k.a making an audio player using jQuery Mobile & NodeJS) Part 1

So I recently signed up for a VPS from AlienVPS at a ridiculously low price and guess what, it crashed on me twice today… -_- But $19/month is still pretty darn cheap. At least it gave me some sandbox to play around with NodeJS and jQuery Mobile.

OK so far NodeJS has been rather simple and straightforward. I actually use Express framework on top of NodeJS which ease the work a little bit. However, I can really see how this can become complicated really really fast. 1st of all, I sorta have to implement all the HTTP protocol code manually in NodeJS (except for 500 Internal Error and 200 OK I think). So that includes 404, 403, blah blah. Not that it matters that much except I wanna maximize my site traffic by taking advantage of Search Engine bots. Well I disallow everything in robots.txt right now so if those bots behave, I should be good. You can check it out at http://208.93.155.39:8080/playlist.html but please please don’t spread it around or I’m gonna have to shut it down due to my limited bandwidth. The app is still buggy since it’s a work in progress but a refresh should make it behave a bit better.

Anyway, now the 1st thing a web server should be able to do is to serve static web pages and it can be achieved pretty easily:

var app = require('express').createServer();
var fs = require('fs');
var public_path = 'public/';
var PORT = 8080;
app.get('/', function(req, res) {
        res.sendfile(public_path + 'index.html');
});
app.get('/*', function(req, res) {
        var page = req.params[0];
        res.sendfile(public_path + page);
});

Easy enuf… Now I want to query some specific stuff like, idk my KOREAN POP playlist!!

var songs;
app.post('/playlist.html', function(req, res){
        Controller.handlePlaylist(req, res);
});
var Playlist = {
        get : function(name) {
                return fs.readdirSync(public_path + MUSIC_PATH + name);
        }
};
var Controller = {
        handlePlaylist : function(req, res) {
                if (!songs) { //lazy-initialize this
                        songs = Playlist.get(req.param('playlist'));
                }
                var index = req.param('song');
                if (index && index >=0 && index < songs.length) { //if I query a specific song number, give me the path to the song
                        res.send({ 'result' : MUSIC_PATH + 'kpop/' + songs[index] });
                } else { //otherwise give me the whole list
                        res.send({ 'filenames' : songs });
                }
        }
};

Now when I hit up playlist.html?playlist=kpop with POST, I can get my playlist and playlist.html?song=1 with POST gives me the 2nd song. This is a simple enuf song serving mechanism that will help me build my audio player.

Playlist Page

Playlist Page

Since I’m not using any view rendering engine, in playlist.html I actually have to use the trick of loading the file 1st, then make an ajax call to populate the data. This gets very very tricky with jQuery mobile since it doesn’t have a lot of control events on when it’s done rendering and what not. This combines with ambiguous timing from AJAX callbacks can lead to a pretty disruptive UX (I’m still having trouble with synchronizing stuff in JavaScript). But anyway, the playlist.html has a pretty simple structure:

<div data-role="page" class="playlist">
    <div data-role="header">
        <h1>My collection</h1>
    </div><!-- /header -->

    <div data-role="content">
        <ul data-role="listview" data-inset="true">
                <li data-role="list-divider">Kpop</li>
        </ul>
    </div><!-- /content -->
    <script type='text/javascript'> 
    $('div.playlist').bind('pageshow', function() {
        var $page = $(this); // to use inside callback since "this" in the callback function is different
        if (!($page.attr('data-init'))) { // Initialize once
                $.post('playlist.html?playlist=kpop', null, function(data) { //retrieve the data
                        var i;
                        var filenames = data['filenames'];
                        var $playlist = $page.find('ul[data-role="listview"]');
                        for (i in filenames) { //populate the list of songs
                                $playlist.append('<li><a href="player.html?song=' + i + '">' + filenames[i] + '</a></li>'); 
                        }
                        $page.attr('data-init', 'true');
                        $playlist.listview('refresh'); //refresh the view
                }, 'json');
        }
    });
    </script>

</div><!-- /page -->

The HTML structure itself is simple the the JavaScript is kind of a hack. “Pageshow” event in jQuery Mobile gets invoked after page has been initialized (a.k.a after jQuery converts basic elements into its themed mobile looks). Why not “pagecreate” or “pagebeforecreate”? Because the callback is actually an AJAX call to grab the data and listview can only be refresh after it’s been initialized (also not guaranteed in the previous 2 event hooks). If I were to use a view rendering engine to populate the data, then send across the wire, I wouldn’t have had this problem so… something to look at next time.

OK so that’s the easy part, I write next time about how to actually make the player cause that took me like 3 days… >.< sleep now!

Tagged , , , , , ,

Why I’m hooked on JavaScript!!

Now I’m apparently not even an expert at JavaScript but I’ve had some exposure to it in 1 way or another, mainly through using jQuery in my last project (which I totally dig!!). I’m still working on web development in general and that project in particular right now and my friend and I have been going back and forth between the popular languages/frameworks like Ruby on Rails, Django and NodeJS. Honestly, I’m kinda a fanboy for NodeJS… I really am. Here’s why:

Now JavaScript has been mainly used for client side in web browser. The interesting thing is that each big boss (Mozilla/Google/IE) seems to develop their own JS Compiler. The Google V8 Engine (which NodeJS is built on) seems to be the fastest one right now. I was pretty amazed when JS was used as a server side technology but if you break it down, I wouldn’t say it’s impossible.

IMHO, I guess the important thing about writing a server is to handle sockets and a few protocols (HTTP, TCP… stuff like that). Now protocols are (again, IMO) rules of how information should be positioned in a packet so you can do that with pretty much any language. Traditional server technologies like Apache Tomcat, Glassfish… have been spawning threads to handle each incoming request. That’s not necessarily a bad thing but the bottlenecks seem to come down to IO so there’s been drawbacks in terms of blocking threads when u do a long query and such. Now u can overcome that by implementing distributed systems with a load balancer on top, along with some request filtering and routing, at which point it becomes kinda costly.

NodeJS

NodeJS

NodeJS uses Event Queue which puts everything in there and does event loop and uses callback functions. So technically there’re no blocking and the app is very scalable, theoretically. With that said, a long query gets run and once it’s done the callback is invoked. Now that sounds kinda cool.

OK so it can handle HTTP request, what else? Another important thing I would say is the database, which kinda comes down to separate vendor. How does the server connects to the database? Using a driver! Java gets an advantage cause it maintains an interface for vendors to write drivers based on. RoR supports a couple of database technologies and so is Django. There’re enough database vendors and plugins out there that it’s gonna take a while for NodeJS to catch up on that front, I assume.

What else? Some process manipulations like File IO and such, cause apparently the server will need to, at some point, modify some file resources, invoke some processes and stuff like that. NodeJS does have this, although I’m not sure how stable/mature it is. But that should take care of it.

Now I’ve always thought that RoR and Django were booming due to their ability to do rapid prototyping/development really really fast. You can get started on a webapp at a reasonably good pace with those, thanks to ORM and dependency injection and all the fun stuff. Honestly, I’m kinda a purist to a degree that I don’t hate abstracting 1 language over another, but magic can be disastrous sometimes. I believe that I should hand-write SQL queries and HTML/CSS instead of using ORM and compiled HTML (such as erb/jsp/asp…) cause you get a better control over it. It gets hard to manage after a while but it’s easier to identify the bottleneck and optimize it, instead of optimizing RoR and hoping that would optimize SQL.

Besides, using compiled HTML puts more load on the server than client, which in a way provides a more stable outcome since client machines/browsers vary a lot. But again, back to mixed language kinda thing, gotta be careful.

1 of the things I found that is challenging to JavaScript is actually synchronization. Now according to my knowledge, browser JS is executed in a single thread so how come synchronization is a big deal? Because browsers render different things in a different threads (you’ve been wondering how they load part by part, right??). Now what that means is, browsers render images/HTML structure/CSS styling/JS separately in different threads so its was such a pain for me to get a consistent User Experience. I honestly haven’t dug around enough to solve this issue since the nature of callback is just like that. Putting timeout and synchronization variables don’t seem to suffice…

Wait, so what about setTimeout and setInterval and stuff, aren’t they supposed to give a better control? Well FYI, setTimeout to 1000ms doesn’t mean it’ll get executed after 1000ms. It means that’ll get executed after AT LEAST 1000ms (cause it’s a queue…) so yeah, still digging around.

Back to my project which is a CMS now might turn into an RDS, I honestly think all we need are those important features listed above. Worst case is that we write a thin layer of wrapper around the tools we want to use, then invoke shell processes for those. But anw, it comes down to a light, thin server framework cause we probably don’t need all the big guns (which can take tons of time to tune). So yeah, it’d be great to set foot on that!!!

 

Tagged , , ,