Compress Query Strings in JavaScript

Since query strings are limited depending on the browser, I wrote this function to grab redundant data variables and consolidate them into comma-separated values. For instance:

foo=1&foo=2&foo=3&blah=a&blah=b
would turn into…
foo=1,2,3&blah=a,b

This saves some room in the query string. Of course you need to write a trivial line to parse it on the server side, but this saves you some room for more variables in a query string. Here is my attempt at a JavaScript function that does this: (don’t use this, use John’s revised function below)

function compress(query) {
  s = {type:[],value:[]}
  $.each(data.split('&'), function(n){
    parts = this.split('=');
    s.type[n] = parts[0];
    s.value[n] = parts[1];
  });
  var csv = '';
  while(s.type.length > 0) {
    value = s.value.shift();
    type = s.type.shift();
    while((pos = $.inArray(type, s.type)) > -1) {
      value += ',' + s.value.shift();
      s.type.shift();
    }  
    csv += type + '=' + value + '&';
  }
  return csv.substring(0, csv.length-1);
}

I messaged John Resig , and he wrote the function that was much smaller, and faster.

function compress(data){
    var q = {}, ret = "";
    data.replace(/([^=&]+)=([^&]*)/g, function(m, key, value){
        q[key] = (q[key] ? q[key] + "," : "") + value;
    });
    for ( var key in q )
        ret = (ret ? ret + "&" : "") + key + "=" + q[key];
    return ret;
}

UPDATE: John Resig followed up with a post of his own, titled search and don’t replace .

How John’s Script Works

You may be asking, but how does this work? – let me explain. First, the replace function accepts a regular expression as the first parameter, and the second parameter can be a string or a function.

replace(/([^=&]+)=([^&]*)/g, function(m, key, value){

As far as I’m concerned, that lines is the magic of the script so I will go furthur.

/([^=&]+)=([^&]*)/g

This is the regular expression that matches data that starts with ampersand and goes until the end or until the next ampersand. This extracts the data. The /g part loops through each match instead of just one, and does a replace on it.

The function he is passing in has the three parameters set by the regexp.“m“is the match,“key“is the unique identifier, and“value“is the entire string that the regexp is running against.

Next he sets the object based on the key, if it exists already than add a comma between after the current value followed by the next value.

q[key] = (q[key] ? q[key] +",":"") + value;

The last part of the code is simply looping through the, “q” object and outputting our new compressed query string. Brilliant.

Comments

  1. says

    Good stuff Marc. A few clarifying points if that is ok:
    m = the string that matches the pattern (for example, foo=1)
    key = first subexpression match (the result of what is matched from the rules in the parenthesis. In the case of foo=1, this would be foo)
    value = second subexpression match (the result of what is matched from the rules in the parenthesis. In the case of foo=1, this would be 1)

  2. says

    I ran out of space!
    The three arguments are not set by the regexp, but rather, these arguments are defined by what the replacement argument (in this case, a function), can accept. The first argument it can accept is the matched pattern (m), the second, etc, arguments represent matched subexpressions (there could be up to 99 of these I believe).

  3. says

    Again, my blog comments scripting sucks which is probably why I get lots of traffic but few comments. Maybe I should up the priority of reprogramming my blog.

  4. Nick Vuono says

    I had the desire to further compress querystrings with a number of records such as:
    viewIDs=6653,7258,14600,15104,6521,1436,2005,2006,2007,2008,2009

    The first thing i tried was to convert each number after the initial ID into a relative measure of distance from the first so you could get:
    viewIDs=6653,605,-7342,504,8583,-5085,569,1,1,1,1

    Then you can use javascript base36 encoding Number(“605”.toString(36)) and die happy or take advantage of the fact that javascript base36 encoding is all-lowercase and further compress the redundant 1,1,1,1 sections using the remaining 26 uppercase characters—ie: representing -1 and +1 with “Q” and “P” respectively and come up with a final string:
    viewIDs=6653,gt,5ny,e0,-6mf,-3x9,ft,PPPP

    here is a quick example where ‘result’ is an array of the original IDs

    for (var i = 0; i < result.length; i++) {
      if (encodedString.length > 0) {
        var diff = Number(result[i]) - Number(result[i - 1]);
        if (diff >= -14 && diff < = 13) {
          encodedString += strEncode.charAt(14 + diff);
        } else {
          if (diff > 0) {
            if (encodedString.charAt(encodedString.length - 1) == ",") {
              encodedString += diff.toString(36) + ",";
            } else {
              encodedString += "+" + diff.toString(36) + ",";
            }
          } else if (diff < 0) {
            encodedString += diff.toString(36) + ",";
            // diff has negative sign in front already
          }
        }
      } else {
        encodedString += result[i] + ",";
      }
    }

  5. says

    It`s very impressive technique – “function in replace” – but very slowly.
    For example, I maked faster then twice code :

    var compress2 = function (data,stuff,q) {
      stuff = data.split(/[=&]/); q={}; 
      for (var i=0,len=stuff.length;i<len ;i+=2)
        q[stuff[i]]=(q[stuff[i]]?q[stuff[i]]+", ":"")+stuff[i+1];
      stuff = "";
      for (var key in q)
        if (key) stuff=(stuff?stuff+"&":"")+key+"="+q[key];
          return stuff;
    }

    I tried to re-write flickr “contact parser” early and was defeated. I used exactly technique by John, but simply “split then run array” it were faster.
    Then I was come upon this post end benchmark your code. Unfortunately timing were as with flickr.

  6. says

    great post thank you I did also needed the inverse function:

    Which I wrote with the same techniques:

    function uncompress(_data) {
      var ret = "";
       _data.replace(/([^=&]+)=([^&]*)/g, function(_match, _key, _value) {
          _value.replace(/([^,]+)/g, function(_innerMatch, _innerValue) {
             ret = ret + (ret.length > 0 ? "&" : "") + _key + "=" + _innerValue;
          });
       });
       return ret;
    }