Sunday, April 14, 2013

CharityVid - A new way to donate to charity


CharityVid allows anybody to donate to a charity of their choice absolutely free, no credit card required. The only thing we ask is that they watch a short sponsored YouTube video, and that's it.

I've been working on CharityVid for a couple months now, and now finally I'm putting it out there for the world to see. Building it has been extremely interesting from both a business and technical perspective, and I plan to start a mini series here about some of the key technical details that make up the site. I plan on covering:

  • using an AWS load balancer
  • server performance configurations
  • Zero downtime deploys
  • front end performance (page load) optimizations
  • logging, and server monitoring
  • unit testing


and a few more topics (I'll probably combine a few together). CharityVid has been my first large project to make it all the way from inception to production and I can't wait to see how it turns out.

Check it out! Let me know what you think.


Saturday, March 16, 2013

ZFeed

ZFeed.zolmeister.com (Source)

ZFeed is an app I wrote for the Sails.js hackathon. Basically It's a front-end that I find useful for consuming my Google Reader RSS feed. I have all of my subscriptions in one feed, and every day I go through and read/glance at every single article in it (~70 per day). However, I had a few issues with how Google was showing me my articles, so I decided to take matters into my own hands.


The key features of ZFeed are the design, integration with Google Reader (to manage subscriptions), and most importantly its page summary functionality (also the J, and K shortcut keys). I subscribe to both /r/technology and 100pt+ Hacker News, which normally do not have any description other than the title of the page. Not having a page summary is not cool, so I integrated the reader with embed.ly to give consistent summaries of almost every article.

I started developing ZFeed at my desktop, but finished it (and deployed) from here (Utah):


Which meant I had to do all my work on my netbook. This worked out ok, the biggest problem was my lack of enthusiasm for testing so it ended up being pretty buggy and crashing constantly (it's still pretty buggy).

I would go into detail on how to use the Reader API, except that Google is shutting it down (announced 3 days after I made this). I am heartbroken, but I see this as an opportunity to write my own RSS service (if I don't end up finding another alternative service with equivalent API).

Tuesday, February 19, 2013

My js1k submission: Flurry

App: Flurry

In my second blog post I wrote about javascript golfing, and how it is a great way to entertain yourself and learn some cool tricks along the way. With js1k underway (you can submit till Match 31st), I have decided to write about my submission, Flurry. Now originally it was going to use both the Full Screen API, as well as the Pointer Lock API (Mozilla misspelled their call with a capital S in screen), however after submitting it turned out that webkit blocks (be default) the fullscreen/pointerlock from within a frame (which is how js1k hosts submissions). I was disappointed but moved on to a variant that didn't use those calls.

The final code came in at 1023 bytes, here is the original uncompressed source (4.8 kB):

 /*  
  * By: Zolmeister  
  * http://zolmeister.com  
  */  
 mouseX = mouseY = 0  
 flowers = []  
 // Z  
 letter = [[.3579, .1628], [.6421, .1599], [.3512, .7326], [.6488, .7297]]  
 onmousemove = function(e) {  
   mouseX = e.pageX  
   mouseY = e.pageY  
 }  
 cHi = c.height = innerHeight - 21  
 setInterval(function() {  
   if (mouseX) {  
     cWid = c.width = innerWidth - 21  
     //with with is with  
     with (a) {  
       with (Math) {  
         rand = random  
         //Align text so that it rotates cleanly  
         textBaseline = 'middle'  
         textAlign = 'center'  
         shadowColor = '#222'  
         if (rand() > .6 & flowers.length < 2) {  
           //spawn flower  
           flow = rand() * 27 + 10025  
           dir = rand() * PI * 2  
           flower = {  
             x : mouseX,  
             y : mouseY,  
             dir : dir,  
             xSpeed : cos(dir),  
             ySpeed : sin(dir),  
             flower : String.fromCharCode(flow), //unicode flowers  
             rot : 0,  
             poptime : 20,  
             size : 90,  
             color : '#' + ['FA7A79', 'A266AC', 'E38F3D', '81A63F', '619CD8'][~~(rand() * 5)]  
           }  
           flowers.push(flower)  
         }  
         for (i in flowers) {  
           with (flowers[i]) {  
             dir %= 2 * PI  
             //wall collision in radians
             if (x < 70 | x > cWid - 70) {  
               xSpeed *= -1  
               if (x < 70)  x = 70  
               if (x > cWid - 70)  x = cWid - 70  
             }  
             if (y < 70 | y > cHi - 70) {  
               ySpeed *= -1  
               if (y < 70)  y = 70  
               if (y > cHi - 70)  y = cHi - 70  
             }  
             x += xSpeed * 16  
             y += ySpeed * 16  
             last = letter[0]  
             for ( j = 1; j < 4; j++) {  
               x1 = cWid * last[0]  
               y1 = cHi * last[1]  
               x2 = cWid * letter[j][0]  
               y2 = cHi * letter[j][1]  
               m = (y1 - y2) / (x1 - x2)  
               yy = m * (x - x1) + y1  
               if (x < max(x1, x2) & x > min(x1, x2)) {  
                 if (abs(y - yy) < 10) {  
                   x -= xSpeed * 15.9  
                   y -= ySpeed * 15.9  
                   j=4  
                 }  
               }  
               last = letter[j]  
             }  
             rot += PI / 9  
             poptime -= 1  
             if (flowers.length < 150 & poptime < 0 && size && abs(x - mouseX) < size / 2 & abs(y - mouseY) < size / 2) {  
               size *= .9  
               poptime = 30  
               dir = rand() * PI * 2  
               xSpeed = cos(dir)  
               ySpeed = sin(dir)  
               for ( i = 0; i < 10; i++) {  
                 dd = rand() * PI * 2  
                 xx = mouseX + cos(dd) * 45  
                 yy = mouseY + sin(dd) * 45  
                 flowers.push({  
                   x : xx,  
                   y : yy,  
                   xSpeed : cos(dd),  
                   ySpeed : sin(dd),  
                   dir : dd,  
                   flower : String.fromCharCode(rand() * 27 + 10025), //unicode flowers  
                   rot : 0,  
                   poptime : 20,  
                   size : size,  
                   color : '#' + ['FA7A79', 'A266AC', 'E38F3D', '81A63F', '619CD8'][~~(rand() * 5)]  
                 })  
               }  
             }  
             //draw rotated flower  
             font = size + 'px sans'  
             shadowBlur = 3;  
             fillStyle = color  
             save()  
             translate(x, y)  
             rotate(rot)  
             fillText(flower, 0, 0)  
             restore()  
           }  
         }  
       }  
     }  
   }  
 }, 33)  

I would like to point out some of the most useful things I used to save bytes in this, namely using the javascript 'with' call, along with ~~, &, |.  ~~ is used to convert a float to an integer, and the binary operators were used to replace the traditional 'or' (||) and 'and' (&&). The binary operators don't work in all cases however (from my experience it works with boolean expressions but not 'truthy' values (ie, strings or numbers !=0). The real gem in the code however is the rampant use of the 'with' call. I make a  lot of calls to the canvas context (as well as math operations sin, cos, abs) and also within loops to object properties. I was surprised to find out that you can assign values such as 'shadowBlur=3' within a 'with' block and have it work out nicely.

Other than that, the only thing worth noting is that if you want to get spinning text (in my case unicode) its easiest (and the only way I got to work) to set the textBaseline='middle' and textAlign='center' on the canvas context. This positions the text (nearly) perfectly so that it rotates flawlessly. Without that, you get weird circulating objects (which is what happens to objects that are spun offset from their center).

Finally, after hand minifying my code by renaming variables and moving a few bits of code around accordingly and also shortening 'if' blocks with ternary operators, I passed the code first through the google closure compiler service and then through JSCrush. This gave me a tidy line of 1023 bytes and concluded my js1k submission.

 _='X=Y=0;D=[];G=[[ 358`16 2`16 [ 351`73 9`73]];onmousemove= b X X;Y Y};K=c.hQZHQ-21;setIn rval(  if(X J=c.wid!ZWid!-21; a) Ma!) i   M=random, xtBasel e="mikle  xtAlign="cen r  Color="#222  6< &2 &$ ,     Xq:Y,N:N,A: $ B:z$) 90  D) D[i] N%=2*PI; >x|x>J A x (x=  x>J x=J- )  >y|y>K B y (y=  y>K y=K- ) x+?6*A;y+?6*B;P=G[0]; j?;4>j;j++)T=J*P[0 V=K*P[1 U=J* [0 W=K* [1 R=(V-W)/(T-U)*(x-T)+V,x<max^&x>m ^ 10> y-R) (x Aq B,j=4 P= ;O+=PI/9;F-?;if(150 0>F H  x-X_& y-Y_ H*= 9;F=30;N ;A= $ B=z$  i=0;10>i;i++)k ,L=X+45*  R=Y+45*z      Lq:R,A:  B:z N:k H }font=H+"px sans"; Blur=3;fillStyle=I;save( transla (xq rota (O fillText(E,0,0 restore()}}},33) ,I:"#"+["FA7A79@A266AC@E38F3D@81A63F@619CD8"][~~(5* )]}) ,E:Str g.fromCharCode(27* +10025 O:0,F:20,H: =2* *PI -  ( (k  0. in >D.leng!&     D.push({x: 70 wi!( function( ); ],@, ), && M() cos for( G[j] abs( =b.page *=-1, > -?5.9* te 3 [ 64 shadow ){!th$(N?=1@ "QeightZ= ner^(T,U)_)<H/2`, kddq,yzs ';for(Y=0;$='zqk`_^ZQ@?$!                                 '[Y++];)with(_.split($))_=join(pop());eval(_)  



Monday, December 10, 2012

How to deploy node.js applications


I am currently running 5 websites on my free (1 year) Amazon EC2 micro instace. Every one of these urls corresponds to a separate node.js service running on the server:
http://zoggle.zolmeister.com/
http://games.zolmeister.com/
http://avabranch.zolmeister.com/
http://gallery.zolmeister.com/
http://charityvid.org/

Let me preface this by saying that the way I deployed my sites before was the following:
  • ssh myserver@amazonec2
  • screen -ls (to display all the screen instances I had open)
  • screen -r XXXX (guess at which screen was the one running the site I wanted to update)
  • node app.js (launch a server process in the screen, and then Ctrl+a+d to detach from the screen)
This is obviously inefficient and slow, and by the time I got to running 5 websites (and 1 proxy service to direct traffic) I decided I needed a more efficient way of deploying and managing my sites.


Enter upstart, the /sbin/init replacement. Upstart lets you maintain just one config file per site and start and stop it as a service without the need to run everything in a screen. To deploy an app is as simple as copying the myapplication.conf file into /etc/init/ and running "start myapplication". Here is an example of one of my conf files (modified from this site):

 description "CharityVid node server"  
 author   "Zolmeister"  
 stop on shutdown  
 respawn  
 respawn limit 20 5  
 # Max open files are @ 1024 by default. Bit few.  
 limit nofile 32768 32768  
 script  
      echo $$ > /var/run/charityvid.pid  
      export NODE_ENV=production  
      cd /home/pi/websites/charityvid  
      exec /usr/local/bin/up -p 3100 -w app.js 2>&1 >> /var/log/charityvid.log  
 end script  

What this conf file says is that on service start, log the process's pid (in case we need to kill it), set node to production mode, cd into the site directory (this is necessary because of issues with express.js template loading), and finally start the app. Notice that I start the app with the "up" command, and that I am piping 2>&1 (std err to std out) and appending (>>) it to the log file. The "up" command comes from the up node module which does auto-updating and load balancing. The alternative to "up" is node-supervisor which doesn't do load balancing but can be run on any application out of the box (without the modification required by "up").

Ok, so now we have a way to start all of our applications, but how to route all the connections to the right place (remember all our apps are on one server with one IP address). Normally you would use NGINX (pronounced "engine x"), but the stable branch doesn't support websockets (used by http://zoggle.zolmeister.com) so I decided to go with the simple node-http-proxy module. The documentation is a bit lacking, but I was able to get all the sites routed, including websockets support and ssl support (for https://charityvid.org/). Here is what my app.js file looks like for my "gateway" application:

 var http = require('http'),  
      httpProxy = require('http-proxy'),  
      fs = require('fs'),  
   https = require('https'),  
   crypto = require("crypto"),  
   path = require('path');  
 var main1 = httpProxy.createServer({  
  router: {  
   'zoggle.zolmeister.com': 'localhost:3001',  
   'avabranch.zolmeister.com': 'localhost:3005',    
   'charityvid.org': 'localhost:3100'  
  }  
 });  
 main1.listen(8000);  
 var proxy = new httpProxy.HttpProxy({  
  target: {  
   host: 'localhost',  
   port: 8000  
  }  
 });  
 var server = http.createServer(function (req, res) {  
  // Proxy normal HTTP requests  
  proxy.proxyRequest(req, res);  
 });  
 server.on('upgrade', function(req, socket, head) {  
  // Proxy websocket requests too  
  proxy.proxyWebSocketRequest(req, socket, head);  
 });  
 server.listen(80);  
 function getCredentialsContext (cer) {  
  return crypto.createCredentials({  
   key: fs.readFileSync(path.join(__dirname, 'certs', cer + '.key')),  
   cert: fs.readFileSync(path.join(__dirname, 'certs', cer + '.crt'))  
  }).context;  
 }  
 var certs = {  
  "charityvid.org": getCredentialsContext("charityvid")  
 };  
 var options = {  
  https: {  
   SNICallback: function(hostname){  
    return certs[hostname];  
   }  
  },  
  hostnameOnly: true,  
  router: {  
   'charityvid.org': 'localhost:3100'  
  }  
 };  
 // https  
 httpProxy.createServer(options).listen(443);  

After running the routing proxy every site should now be functional and up, but we still have to ssh into the server to start/stop apps and to update them (with supervisor and up you should rarely have to restart the node server). This is where fabric comes in. Fabric lets you automate the ssh process and run commands on the server from your work machine. For example a simple command to send the initial application.conf upstart file would be:
 def installInit():  
      put('avabranch.conf', '/etc/init/avabranch.conf', True)
      sudo('touch /var/log/avabranch.log')  
      sudo('chown admin /var/log/avabranch.log')  

And if you're using git to manage your source (I store mine in a private bitbucket repo), updating your production code is as easy as "fab update:avabranch" with the following fab file declaration:
 def update(app):  
      with cd('/home/admin/websites'):  
           if app=="avabranch":  
                run('cd avabranch && git pull')   

Finally I keep my sites behind the Cloud Flare service, which basically means that they are protected from DDOS attacks, plus get free and efficient localized caching (cloud flare has many data-centers all over the world, which allows them to improve site cache latency times).

Monday, November 19, 2012

Avabranch

Avabranch

Avabranch (source on Github) is my entry for the github gameoff web-game challenge. It's built using Node.js and the Express framework, though most of the code is pure handwritten client-side javascript. I really enjoyed working on avabranch and wanted to take you through what I did to build it.

First of all, since the game uses canvas, we need to figure out how to clear, update, and draw our objects on the canvas. This is where 'requestAnimFrame' comes in. The boilerplate code for this function is this (it adds support for browser-specific implementations because it's not a finalized standard yet):

 window.requestAnimFrame = (function() {  
      return window.requestAnimationFrame || window.webkitRequestAnimationFrame || window.mozRequestAnimationFrame || window.oRequestAnimationFrame || window.msRequestAnimationFrame ||  
      function(/* function */callback, /* DOMElement */element) {  
           window.setTimeout(callback, 1000 / 60)  
      }  
 })()  


Before we call this function however, we need to add things that need to be drawn. For me, I add a player, enemy spawner, hud, and powerup spawner to the game object, and then call the update function on the game object.

 function startGame() {  
      keyListeners = []  
      game.play = false  
      game = new Game(canvas)  
      var player = new Player(game, null, null, null, game.speed)  
      var spawner = new BlockSpawner(game, game.speed)  
      var hud = new HUD(game)  
      var power_spawner = new PowerupSpawner(game)  
      game.addObject("spawner", spawner)  
      game.addObject("player", player)  
      game.addObject("power_spawn", power_spawner)  
      game.addObject("hud", hud)  
      game.update()  
 }  

Now, when I call game.update(), it runs this code:
 this.update = function(time) {  
           if (!this.play)  
                return;  
           this.timeDelta = time - this.prevTime  
           this.prevTime = time  
           if (isNaN(this.timeDelta)) {  
                requestAnimFrame(this.update.bind(this))  
                return  
           }  
           this.ctx.clearRect(0, 0, this.canvas.width, this.canvas.height)  
           this.physics(this.timeDelta)  
           this.draw()  
           requestAnimFrame(this.update.bind(this))  
      }  

This is where most new developers make a mistake, as they forget the critical 'time' variable that is passed into the function by requestAnimFrame, which is essential for consistent playback across all machines. In order to make sure that a slow machine, unable to run the game at the optimum 60 fps, doesn't end up playing a game that runs half as fast we must keep track of how much time has passed since we last updated the objects on the screen. This is what 'timeDelta' is for, as it keeps track of this time. Now when the Game object calls the 'physics' function on all of its children, it passes in this timeDelta value which is used in the update code for objects that move. For example the 'Line' object uses timeDelta when it updates its position:

 this.points[i].y += .05 * timeDelta * this.ySpeed  

Another thing that is important for gameplay is managing key state and key events. I actually ended up writing my own key event system for this. The basics of how I manage the keyboard are that I store all pressed keys in a JSON object, and on keypress I loop through any listeners that were listening for that keypress and run whatever function they passed in. Here is the code for that (the keys are converted from key number to letter for access in the dictionary):

 var keyListeners = []  
 var keyState = {};  
 var keyMap = {  
      13 : 'enter',  
      37 : 'left',  
      38 : 'up',  
      39 : 'right',  
      40 : 'down',  
      186 : ';'  
 };  
 window.onkeydown = function(e) {  
      try {  
           keyState[keyMap[e.which] || String.fromCharCode(e.which)] = e.which;  
      } catch(e) {  
           console.log('error converting keypress to char code')  
      }  
 }  
 window.onkeyup = function(e) {  
      try {  
           delete keyState[keyMap[e.which] || String.fromCharCode(e.which)];  
      } catch(e) {  
           console.log('error deleting keypress to char code')  
      }  
 }  
 window.onkeypress = function(e) {  
      for (var i = 0; i < keyListeners.length; i++) {  
           var k = keyMap[e.which] || String.fromCharCode(e.which)  
           if (keyListeners[i][0] === k) {  
                e.preventDefault()  
                keyListeners[i][1]();  
           }  
      }  
 }  

The last thing I want to mention is the code that I used to draw the lines. At first my approach was to draw a ton of dots as fast as I could to make it seem linear. This worked but was too CPU intensive, so I decided to try and come up with a better way to do it. I found out that you can draw lines on the canvas easily, and better yet it natively supports rounded caps. The problem was that before when you turned it was smooth, but now the edges were sharp. I tried to solve this by keeping track of the slope of the line, and when the slope changed then I would use the canvas 'quadraticCurveTo' method to round out the corner. This did not work well and so I continued to look for a solution. Turns out that canvas also lets you set the line joint type to round. Yeah, that was quite a journey, but I'm happy I got it working the way I wanted to. Here is the code:

 this.draw = function(ctx) {  
           var tail = this.points[0]  
           var head = this.points[this.points.length-1]  
           if(!tail || !head)  
                return  
           ctx.beginPath();  
           ctx.lineWidth=head.r*2  
           ctx.lineCap='round'  
           ctx.lineJoin='round'  
           ctx.moveTo(tail.x,tail.y);  
           ctx.strokeStyle = tail.color  
           for (var i = 1; i < this.points.length; i++) {  
                var point = this.points[i]  
                ctx.lineTo(point.x,point.y)  
           }  
           ctx.stroke();  
           ctx.closePath()  
      }  

Wednesday, October 24, 2012

Back-light Music LEDs


Cameras don't like taking pictures in the dark.
tl;dr
The LED lamp behind my monitor pulsates with the audio from my computer. Video is at the bottom.
The Code (warning, ugly code ahead)
Setup:
So the way the led lamp works, is that it is hooked up to an arduino which connects over USB to my tower. My computer is running a python script which sends the arduino the value of brightness to set the lamp at, based on the audio output of the music.

PulseAudio -> python -> arduino -> LED lamp

Getting the audio output into python:
This was probably the most silly issue that I had to deal with, because it seemed so simple in terms of what I needed to do, but took a ton of trial and error and lots of failing. Let me save you the trouble. You may have heard of JACK. JACK is a pain, takes a lot of work, and for me it didn't work at all. Save yourself the trouble and stay away from JACK. I use PulseAudio for my sound system as it works  with my USB headphones (they're virtual 7.1 surround sound and have drivers that only work on windows XP). Setting up my headphones is a pain, but worth it because of the fuzz (the part that goes over my ears is fuzzy which makes them extremely comfortable). Ok, enough background, here is the secret:

  • Open Pulse Audio Volume control (pavucontrol).
  • Open Sound Recorder.
  • Start recording, and go to the "Recording" tab of the volume controller
  • Click on the input, and change it to "Monitor of XXX" (where XXX is your audio device)
  • Done! (Just remember to do this for the python script while running it for the first time)
Python and beat detection:
You can check out the code here. Basically it uses the pyaudio library to record audio, write streaming data chunks to a temporary wav file (this could be improved upon), and use some magic to get amplitude data from the wav file. Then it normalizes the amplitude over the 0-255 range of LED brightness for the arduino, smooths the curve slightly, and sends the amplitude over to the arduino at 115200 baud (not sure if this is too much/not enough, but it worked for me - 9600 baud hung because it was too slow).
Beat detection snippet (thanks Mr.Doob and Dean McNamee)
 w = wave.open(WAVE_OUTPUT_FILENAME, 'rb')  
     summ = 0  
     value = 1  
     delta = 1  
     amps = [ ]  
     for i in xrange(0, w.getnframes()):  
       data = struct.unpack('<h', w.readframes(1))  
       summ += (data[0]*data[0]) / 2  
       if (i != 0 and (i % 1470) == 0):  
         value = int(math.sqrt(summ / 1470.0) / 10)  
         amps.append(value - delta)          
         summ = 0  
         tarW=str(amps[0]*1.0/delta/100)    
         sendVal(tarW)#this method normalized the data and sends it to the arduino
         delta = value  

My normalization algorithm for the value consists of storing the last 100 values of data and dividing new data by the max of the last X values (all values before a reset) to get a value between 0 and 1, which I then multiply by 255 to get 0-255. After a period of low values (ie. silence), detected by the store of last 100 values, the minimum and maximum values reset to allow for a new song which may not fit the amplitude of the previous song. I smooth the data by averaging the current point with the previous point.

Arduino LED fading (pwm):
So the arduino UNO that I have has 6 pwm digital I/O pins, which allows me to dim LEDs attached to those pins. After that, just running an analogWrite() on them with a value between 0-255 will dim the LED accordingly. One thing I did have trouble with though was sending the 0-255 number over serial from python, which I fixed by using a buffer and made the python always write out 3 character strings of data and used the atoi() command (ascii to int) to get the values back out.

The lamp:
I hooked up 12 LEDs, 6 pairs of 2 LEDs hooked up in parallel connected to one of six pwm I/O pins. After lots of frustration regarding the illumination of the LEDs (I used a dremel to diffuse the LEDs, and ended up arcing them as shown below), I finally got it to work. I just can't let the arduino fall off because the wires going to it come out easily (design issue, but hey it works).

The result:
(yes I have 5 monitors, song: Bopalong, Kinobe)

Wednesday, September 26, 2012

My Favorite Exploit


XSS (Cross Site Scripting) is my personal favorite security hole. It's everywhere, on almost every site, and it's fun to exploit! I used to spend my evening scouring the top 500 sites on alexa for XSS vulnerabilities just for fun. In addition to that I did most of my testing manually. There are tools built to automate most XSS vulnerability tests, but that's no fun. This hacking spree taught me the foundations of JavaScript and web development in general. By developing intricate tunnel exploit schemes I learned so much about how the web operates at the user level and how bugs enable hackers to get in.

My endeavours got my name here: Google Security Hall Of Fame

Which led me to create this: http://gallery.zolmeister.com/ (click the page)

That page is my XSS gallery. The images are the websites favicons, with size relative to their ranking. I have saved nearly every one of the XSS vulnerabilities I have found by hand, which is now over 200. Most of the sites are ones that were unhelpful or just pissed me off in general and so I took a few minutes to find an XSS vulnerability in them (I almost never report these vulnerabilities). Some however (the big icons) are ones where I started at the top of a list of most popular sites and moved my way down, collecting vulnerabilities along the way. It's pretty scary actually, knowing that even the most high profile sites are vulnerable.