Two worlds collide – ScraperWiki recap

Big problems take big data. On one side, you have the people who can make sense of all those spreadsheets — the computer programmers, the web developers and, overall, the “techies.” Once that data’s been sorted, you need people to tell the stories in the data — enter journalists.

That, in its simplest explanation, is the idea behind the ScraperWiki workshop March 31 in Washington. In theory, it’s a really great idea. In practice (from this journalist’s perspective), it was bit like people from two countries meeting — without a translator.

The first problem was I don’t know a lick of programming. I know HTML. I know CSS. I know a teeny bit of JavaScript. I know of PHP and Python and Ruby on Rails. Ask me to explain the differences, well, I won’t even bother to fake it.

The day before I attended the workshop, ScraperWiki had held a training session — a kind of “How to Scrape Data 101.” (My first question was, What is data scraping? The question was preceded by the disclaimer: This is going to be a stupid question.) The introduction to data scraping would have been key to being able to jump into a project already in progress.

Talking to some of the programmers, I got a better understanding of the potential for collaboration between us (the word geeks) and them (the computer geeks). When President Obama took office in 2009, he vowed to make his administration the most open and transparent of any previous administration. To some extent, that has been true with the launch of various websites that track government stats, such as contract spending and Recovery Act grants. With the availability of so much federal government data, working with programmers will make accessible the raw data that is put online.

The reality, though, is unless you are a New York Times or a Washington Post, you will not have the resources to devote to these projects. Unless you can find a superstar coder/programmer/journalist in one person, news organizations will have to partner with organizations such as ScraperWiki or Sunlight Foundation.

For the realistic short-term, I decided it was best for me to learn as much programming as I can. I ordered a programming how-to book … for children (and beginners too). I figure, at the very least, it will teach me the language I will need to communicate with the other side.

Posted in Uncategorized | Leave a comment