Josh Tauber (the guy behind GovTrack) recently wrote a book on Open Government Data. I think the chapter where he walks through the process of scraping and building a visualization is great, this kind of thing should be required reading for all non-coders who are interested in technology and transparency issues. It walks through really practical examples of why formats matter. Also liked the figure on the right for locating the strengths of various types of data formats with respect to intended use. Makes the great point that a “high-quality” electronic document isn’t always better for some purposes.
In the book he also discusses at length his concerns with watchdog groups jumping too quickly–and with too shrill a tone–to cry corruption at every correlation that appears with campaign finance data. This is concern I share, especially since I work with a bunch of that kind of data, and often feel like we are not telling the most interesting stories, or that we might be finding relevant stories to tell if we had a better statistical models (or any at all!) of the processes underlying the data and clear definitions of the effects and behaviors (good and bad) that we are searching for. However, I would like to offer an alternative explanation for the failure of money-in-politics groups to highlight party effects as key explanatory variables in their analyses: many of the groups doing the research are 501c3 non-profits which, for legal and strategic reasons, steer away from any reporting that could be construed as partisan.
Overall, I thought the most important point in the book, which Josh attributes to John Wonderlich, is to remember that all the real influence in politics tends to hide just offstage, slightly outside the scope what is formally reported. The more we tighten the transparency laws, the more the deals take place in back rooms, and hide in smoke-filled language. Of course, I still get excited sometimes to think that we might be able to infer some of these offstage process by the traces the leave in the data we do have access to. Transparency may be a necessary precursor for moving towards more democratic and egalitarian societies, but its not really a shortcut. We still have to do the work to get there.