You are browsing the archive for Rufus Pollock.

Open Data Toolkit – Data Micro Services

Rufus Pollock - October 5, 2012 in Musings, Technical

This is an updated version of my previous diagram on DataHub / Data Micro Services. The updated title reflected a slightly different focus in which integration is somewhat looser, services may be used standalone, and there is greater emphasis on a process than a central Hub (i.e. individual services may be used together or alone without necessarily being part of an overall hub though they also can be seen as part of coherent whole). Keywords:

  • Online ETLV = extract, transform (integate)), load (validate), view/visualize
  • Small Pieces, Loosely Joined
  • Open Data Toolkit
  • Micro (Data) Web Services

Opening Up Local and National Government Open Data Conference – Talk by Rufus Pollock

Rufus Pollock - September 26, 2012 in Events, Talks

Slides from my talk today at the Opening Up project Local and National Government Open Data mini-Conference with the title Open Data and the Open Knowledge Foundation.

Javascript Timeline Libaries and Tools – A Review

Rufus Pollock - September 8, 2012 in Javascript, ReclineJS, Research

Note: a fuller, revised, version of this article was posted on OKFN Labs blog: http://okfnlabs.org/blog/2012/12/04/javascript-timeline-libaries-a-review.html

General comments. Timeline libraries consist of various components:

  • Data loading
    • Date parsing
  • Band (timeline) rendering
  • Showing render info on individual items

For me a timeline visualization library need only be the second of these but most that I’ve come across do both and more.

In fact a major issue in my opinion with most libraries is that they are too under-componentized – they don’t separate cleanly into these different components and end up doing everything.

To take one example, the Verite timeline (in my view is one of the best libraries out there) has a whole bunch of its own custom date parsing built in inside an internal utility library which are hard to override or replace and also has a large chunk of code just for loading from google docs and other data sources. (You can of course somewhat solve this somewhat — as I do in Recline by parsing the dates directly and then submitting in a standardized form). Even if library authors do want to include these things it would be good to do it in a way that allowed for a clean separation so that you could just use the parts you wanted.

Propublic Timeline Setter

  • HTML + JS
    • But Requires a build step (using ruby)
  • Very simple and compact design (nice!)

Verite Timeline

  • Very elegant design
  • Includes much more than Timeline (e.g. sourcing data from google docs etc)
  • Mozilla Public License (was GPL)

Simile Timeline

  • http://www.simile-widgets.org/timeline/
  • The original open-source JS timeline but less regularly update and maintained today: “As of Spring 2012, Exhibit is the only Simile widget seeing active development.” and the timeline control has not been updated since 2009 (see this stackoverflow question for more

Chronoline

  • http://stoicloofah.github.com/chronoline.js/
  • Recently developed and updated
  • MIT licensed

Timeglider

CHAPS Timeline

Using Verite Timeline

Some notes on how to use the VeriteCo timeline software based on experience integrating into Recline Data Explorer.

  • Date parsing: it is reasonably particular at the moment re date support and it’s default is the rather unusual yyyy,mm,dd format. ISO format yyyy-mm-dd is not supported (update: ISO format is now supported by Verite).

Notes on Organizations and Organizational Structure

Rufus Pollock - August 27, 2012 in Organization, Research

Somewhat random notes (accumulating over time – first set from early 2012) on organizations and organizational structure.

Ideas

  • Netflix: “Highly aligned – loosely coupled”
  • Common feature of the more high-tech is high / very competitive compensation
    • Netflix: we pay at the top of the range, we lose anyone who isn’t a star performer
    • Valve: our compensation is very competitive
  • CLOU = Colleague Letter of Understanding
    • Every year, each Morning Star employee negotiates a Colleague Letter of Understanding (CLOU) with the associates who are most affected by his or her work. A CLOU (pronounced “clue”) is, in essence, an operating plan for fulfilling one’s mission. An employee may talk to 10 or more colleagues during the negotiations, with each discussion lasting 20 to 60 minutes. A CLOU can cover as many as 30 activity areas and spells out all the relevant performance metrics. All together, CLOUs delineate roughly 3,000 formal relationships among Morning Star’s full-time employees.

Readings

Javascript Links – August 2012

Rufus Pollock - August 19, 2012 in Javascript, Technical

Frameworks

Dependency Management / AMD

  • requirejs
    • Would like more info on this — always found it a bit painful to use
  • yepnope.js – JS dependency manager/requirer
    • Best I’ve used so far (prefer to require.js though require is somewhat different)

Spending Story: California City Bankruptcies

Rufus Pollock - August 5, 2012 in Research, Spending Stories

This is a holding page for research on California City (and State) finances and bankruptcies. Since the start of the “Great Recession” it appears that city and municipal governments in many US states, and especially California, have faced increasing financial difficulties.

Source google doc

Next steps

  • California state budget
  • Try and acquire balance sheet (reserve plus current P&L) for all cities in California
  • Read up on Stockton

Links

Stockton

Official Chapter 9 Bankruptcy info

http://www.stocktongov.com/government/departments/manager/bankruptcy/default.html

On June 28, 2012, the City of Stockton filed a petition for chapter 9 bankruptcy protection with the United States Bankruptcy Court, Eastern District of California, Sacramento, Case No. 2012-32118. The Stockton City Council approved a Pendency Plan on June 26, 2012. The Pendency Plan is essentially the budget and the financial plan that is followed for the day-to-day operations of the City while in bankruptcy. It identifies what expenditures will be reduced or suspended. The City will continue to pay employees, vendors and service providers. The focus of the City’s plan is the restructuring of above market pay and benefits and unsustainable long term debt.

From A506 application (1st of 3): http://www.stocktongov.com/files/Stockton_RestructuringProposal_01_79pages.pdf

From The “Great Recession” has seriously damaged the City of Stockton’s finances. Record foreclosure levels, plummeting home prices, and high unemployment have cut City revenues, at the same time previous labor contracts, generous retiree medical benefits, and increasing debt obligations were driving up expenses. The City Council has declared two fiscal emergencies and negotiated or imposed labor concessions of $25 million and service reductions of $12 million in FY11-12 alone.

Since FY08-09, the City has been forced through lack of funds to reduce sworn General Fund Police staffing by 25%, Fire staffing by 30% and all other staffing by 43%. Programs and services have been reduced to minimum – or below minimum – levels. Sworn Police staffing per 1,000 residents has dropped from a high of 1.52 per 1,000 residents in 2005 to 1.16 currently, and in the face of a rising local crime rate. While violent crime rates dropped 5.5% nationwide in 2010, they were up in Stockton, which ranked 10th in the U.S. with 13.81 violent crimes per 1,000 residents. [emphasis added]

Despite four years of cost-cutting, current estimates still show a significant shortfall for FY12-13. The City has spent its reserves and cannot close the gap without further large cost reductions.
The City’s best case shows an annual General Fund shortfall of $21.3 million in FY12-13. This is a status quo “baseline budget” with no change in current positions and no COLAs. Without making new, major reductions in the City’s costs, the annual deficit will increase to an average of $34.5 million over FY15-16 through FY20-21. None of these shortfalls can exist because the California Constitution prohibits cities from deficit spending

Current year (2012/2013) budget documents

Budget proposal

Excerpts:

In my budget transmittal memo for last year I mentioned:

“The City of Stockton is near insolvency. Reserves in the General Fund are exhausted, reserves in internal service funds are near depletion and the City has leveraged itself with inter-fund loans and debt that puts the General Fund at risk for backfilling deficits in other funds. Furthermore, the City is saddled with multi year labor contracts with escalating costs that simply cannot be paid within anticipated revenues in the foreseeable future.”

Resources

BBC Archive Statistics

Rufus Pollock - June 22, 2012 in Musings

Some approximate stats about BBC holdings of different types of media from Jake Berger’s presentation today.

  • 2.3 million items of FILM and VIDEO
  • 300,000 hours of AUDIO
  • 1.1 million items of COMMERCIAL MUSIC
  • 4 million items of SHEET MUSIC
  • 4 million PHOTOS
  • 500,000 subject categories
  • 1 million programme contributor NAMES
  • 9 million searchable RECORDS (10 miles of shelves)
  • 25 million PRESS CUTTINGS
  • 70,000 BOOKS
  • 206,899 digitised Radio items
  • 22,965 digitised TV Programmes

DataHub – Small Pieces Loosely Joined

Rufus Pollock - June 22, 2012 in CKAN, DataHub

Javascript Tips and Tricks

Rufus Pollock - June 21, 2012 in Javascript, Technical

I’ve started a github repo for recording javascript tips and tricks at: https://github.com/rgrp/js-tips-and-tricks

Material there so far includes:

Deep Copy

Warning: full deep copy is hard. The following are only appropriate for “simple” objects.

JQuery

Copy (clone) in javascript with JQuery::

// Shallow copy
var newObject = jQuery.extend({}, oldObject);

// Deep copy
var newObject = jQuery.extend(true, {}, oldObject);

Underscore

Note underscore does not do a deep copy at all — it will only do copy on first level attributes::

_.extend({}, myobject);

JSON

Deserialize to and from JSON (warning: not at all performant!)::

var newobj = JSON.parse(JSON.stringify(oldobj));

Check if a variable is undefined

if (typeof variable === undefined) {
  // do something
}

Simile Inheritance in Javascript

John Resig’s original

From http://ejohn.org/blog/simple-javascript-inheritance/

/* Simple JavaScript Inheritance
 * By John Resig http://ejohn.org/
 * MIT Licensed.
 */
// Inspired by base2 and Prototype
(function(){
  var initializing = false, fnTest = /xyz/.test(function(){xyz;}) ? /\b_super\b/ : /.*/;
  // The base Class implementation (does nothing)
  this.Class = function(){};

  // Create a new Class that inherits from this class
  Class.extend = function(prop) {
    var _super = this.prototype;

    // Instantiate a base class (but only create the instance,
    // don't run the init constructor)
    initializing = true;
    var prototype = new this();
    initializing = false;

    // Copy the properties over onto the new prototype
    for (var name in prop) {
      // Check if we're overwriting an existing function
      prototype[name] = typeof prop[name] == "function" && 
        typeof _super[name] == "function" && fnTest.test(prop[name]) ?
        (function(name, fn){
          return function() {
            var tmp = this._super;

            // Add a new ._super() method that is the same method
            // but on the super-class
            this._super = _super[name];

            // The method only need to be bound temporarily, so we
            // remove it when we're done executing
            var ret = fn.apply(this, arguments);        
            this._super = tmp;

            return ret;
          };
        })(name, prop[name]) :
        prop[name];
    }

    // The dummy class constructor
    function Class() {
      // All construction is actually done in the init method
      if ( !initializing && this.init )
        this.init.apply(this, arguments);
    }

    // Populate our constructed prototype object
    Class.prototype = prototype;

    // Enforce the constructor to be what we expect
    Class.prototype.constructor = Class;

    // And make this class extendable
    Class.extend = arguments.callee;

    return Class;
  };
})();

Usage::

var Person = Class.extend({
  init: function(isDancing){
    this.dancing = isDancing;
  }
});
var Ninja = Person.extend({
  init: function(){
    this._super( false );
  }
});

var p = new Person(true);
p.dancing; // => true

var n = new Ninja();
n.dancing; // => false 

Steffen Rusitchka version

/* Steffen Rusitchka version
 * <http://www.ruzee.com/blog/2008/12/javascript-inheritance-via-prototypes-and-closures>
 * Creative Commons Attribution licensed
 */
(function(){
  var isFn = function(fn) { return typeof fn == "function"; };
  PClass = function(){};
  PClass.create = function(proto) {
    var k = function(magic) { // call init only if there's no magic cookie
      if (magic != isFn && isFn(this.init)) this.init.apply(this, arguments);
    };
    k.prototype = new this(isFn); // use our private method as magic cookie
    for (key in proto) (function(fn, sfn){ // create a closure
      k.prototype[key] = !isFn(fn) || !isFn(sfn) ? fn : // add _super method
        function() { this._super = sfn; return fn.apply(this, arguments); };
    })(proto[key], k.prototype[key]);
    k.prototype.constructor = k;
    k.extend = this.extend || this.create;
    return k;
  };
})();

Usage::

var X = PClass.create({
  val: 1,
  init: function(cv) { 
    this.cv = cv; 
  },
  test: function() { 
    return [this.val, this.cv].join(", "); 
  }
});

var Y = X.extend({
  val: 2,
  init: function(cv, cv2) { 
    this._super(cv); this.cv2 = cv2; 
  },
  test: function() { 
    return [this._super(), this.cv2].join(", "); 
  }
});

var x = new X(123);
var y = new Y(234, 567);

Overview of Open Knowledge Foundation Data Projects

Rufus Pollock - June 19, 2012 in Musings, Organization

NB: this is an in-progress diagram. It may not reflect current (or even future) reality!

Can be hard to engage or contribute directly to larger projects as they have got more complex. Need a good way to cluster activity before this point — both at coding level and at data wrangler level. Natural place may be labs — though we may want a dedicated space for data wranglers as activity grows.

Source on Google Docs