Presentazione per DJS2

Ecco la presentazione del mio intervento di oggi alla Data Journalism School a Milano

Data Scraping, Data Organization from Marco Montanari

Sull'utilità dei dati aperti

In seguito ad una discussione su twitter seguita all'articolo su CheFuturo dell'amico Napo, mi sono posto alcune domande: Perchè pubblicare dati se tanto, come dice il governo inglese, l'interesse è poco, le apps pure, ecc? Il problema, a mio avviso, è duplice e in entrambi i casi il fattore tempo è il fattore preponderante. Il primo problema è quello della qualità. I persiani utilizzavano il liquido scuro che usciva dalla terra per farci le lampade ad olio. Oggi sappiamo che la qualità di quel liquido scuro è molto bassa, e da quella esperienza abbiamo imparato che è necessario passare attraverso una fase di raffinazione che porta il greggio ad essere ripulito e quindi più facilmente riutilizzabile per realizzarci tutti i possibili prodotti dalle gomme delle auto alle cover degli iphone.
Il secondo problema è di tempo di risposta. Chi politicamente si impegna per l'apertura dei dati ha bisogno di un riscontro a breve/brevissimo tremine per giustificare l'azione o l'investimento. Il problema è che l'informatica ha illuso di poter dare immediatezza a tutto. Ma a questo livello di immediato c'è poco. Per creare una filiera di uso dei dati ci vuole del tempo, ci vogliono fallimenti, ci vogliono successi e ci vogliono copiatori. Non nasce in pochi anni una filiera del genere, a meno che una grandissima azienda non inizi a spingere sull'argomento. Ma a questo punto ci si trova davanti allo scoglio di qualità, quantità e freschezza. Come prima, torniamo al modello del petrolio. Chi investirebbe in un pozzo petrolifero senza fare i controlli del caso? Dove si trova il petrolio? Quanto ce n'è? Quanto va raffinato? La stessa operazione cinica e bara va fatta sui dati, soprattutto se chi ci vuole lavorare è una Startup.
E quindi in questa discussione, chi pubblica dati pre-aggregati non sta giocando a favore dell'uso dei dati. Chi pubblica pochi dati gioca a favore ma non cambia i pesi sulla bilancia. Quello che cambia i pesi sulla bilancia sono progetti come quello di Torino, con le API che danno i dati sui parcheggi in tempo quasi reale, e infatti proliferano apps. Quello che cambia i pesi sulla bilancia è il set di API fornito dalla MTA di NYC, che espone le informazioni sulla metro in tempo reale, e proliferano le apps. Poi possiamo discutere sul fatto che alcuni dataset sono più o meno complessi da utilizzare ed analizzare. E questo è fuori discussione. Il traffico è immediato, magari l'impatto delle decisioni della giunta del comune sul rimboschimento delle aree verdi della città meno. Ma questo non significa che non ci siano opportunità. E le opportunità arrivano con l'aumento di almeno due delle tre dimensioni. Qualità - Quantità - Freschezza.

Data Journalism School

Un po' di pubblicità alla Data Journalism School organizzata dal 13 al 15 Dicembre a Milano da Istat e dalla fondazione <AHREF. Qui maggiori dettagli.

Gov.UK principles for good government website (and service) design

Tim Brown posts the link to this interesting website: https://www.gov.uk/designprinciples. The principles exposed are very interesting and takes the discussion to an amazing new level. This explains how far forward  UK government is in this area, compared with other local governments...
[via Linkedin]

Prez for Open Data Lab

Last week's Laboratory on open data refining and usage at the Smart City Exhibition in Bologna

Open data from Marco Montanari

Open Transport Data Manifest

From a napo's blog: http://de.straba.us/2012/11/05/open-transport-data-manifesto/

On a new wave of Data Warehousing

During my trip to Helsinki at the OKFestival, I happened to talk to some new friends about how data has to be analyzed. So in the end what I came home with is a concept, that has been following me for some time now, and that sooner or later has to be analyzed deeply. In the end the two main references in the datasets are Time and Space. This means that we have a basically 2D point that matches a given time and a given location. The problem is that geographic positions are only a recent acquisition of mankind. Before that, there were only location names like my hometown, Castel Maggiore, that was previously called Castaniolo then Castagnolo Maggiore, and in each period the form of the location was completely different. This basically implies that the only granted dimension is time, while "Administrative space" is time-dependent and physical point-based location is not. Is there a way to express this in a meaningful manner?

Analysis en course...

And finally time's up...

... for the vacation. In the end the two weeks passed incredibly fast. I sincerely would need one more week to end some of the things I have begun, but time is limited, so we'll see.
The balance is overall very positive:

• VivaCity project for DjangoDash (single-developer) got top votes for the innovation/programming parts, while being a bit penalized for the lack (or almost) of UI. I'm very happy about that. Based on the data in VivaCity I made a small app for WP7 to find the BikeRacks anywhere in the world based on the open data exposed by the city governance
• SWPatterns was a sideproject which exploded to become a main project of one of the days and got a pleasant companion app for WP7
• GentleNerds will become the core for all the social apps coming to Android soon enough (HYMD and Loco)
• Nerderie already had plans and will soon get the code behind it and some action will start.
• CineCit (which is at the moment under Nerderie) is going to grow with Lorenzo
• Signage Studio is growing fast and strong and will soon be in private beta. At the moment there is some fighting going on with Canvas events causing ajax operations to be called.
• SemBase and FlowLy are going to become parts of the other instruments (SemBase is the back-end part of VivaCity) and are going to be re-launched in a more "web-oriented" manner

Things I learned from these two weeks: 

• asmack, smack, asmackservice for Android... Most of them are in part a PITA, but in the end, using a sometimes "violent" approach Android DOES talk to an ejabberd server
• ejabberd is a memory hog, which is tragic when you have only one VPS for many domains and apps.
• Linux hard limit for open files is a PITA in a server where you have lots of apps running but there's always a solution (ulimit -Sn 2048)
• OpenVZ does not like SwapFiles..... (ejabberd + apache2 ==> TOO much memory used)
• Considering I only use WSGI apps, there must be a way to avoid the many thousands of plugins of apache2 in favor of a Virtualhost-aware wsgi-only webserver... I'm considering making something with Twisted, but who knows...
• Time and space ARE a problem... the unsolvable kind...

New things that will be in next to-do lists

• With the arrival of Open Data, GreenIsland can be once again very interesting. 
• In an optic of data-driven information, even the old WildLife project can be cool...

So, that's about it.

From tomorrow it's usual job again! 

[Cross-posted on TheITGuys.biz]

Data Maps - Part 2

Always from the Sim City Blog, the interesting analysis of the data-driven city.

"For example, when the player places a power plant, they will see unpowered roads and transmission lines. Once the power plant is placed, power will start flowing along power conductors to the buildings. If there’s a break in that transmission, the player will immediately see the break. If the power plant isn’t big enough, the player will see the power getting eaten up before it reaches the farther buildings. That information is there in the world to see so the player doesn’t have to look for it – as if they had X-ray goggles that could show electricity flowing. Through the data layers, we’re teaching the player the mechanics of a complex system without them being aware of it. With data layers as a backbone we’re able to take complex simulation-driven information and provide a visual that all players will understand and find accessible."

That is why Sim City is THE tool for the Smart City
http://www.simcity.com/en_US/blog/article/Data-Maps-in-SimCity-Part-2

David Osimo on CheFuturo! [ita]

http://www.chefuturo.it/2012/05/david-osimo-8-strumenti-gratuiti-per-migliorare-la-politica/?utm_source=twitterfeed&utm_medium=facebook

Data Maps

And in the end the data is everything for a city simulation. The point where real and virtual become one. The point where the real city gives the simulation a background and enables both the administration and the citizen a way to understand and appreciate the complexity of a real city.
http://www.simcity.com/en_US/blog/article/Data-Maps-in-SimCity-Part-1

Sim City...

this is the Smart City Tool I want. It's a tool created for the people, not for politicians or managers. For people by engineers and designers. No politics involved. No mediations except for data complexity. This is the tool that can leverage the best of the civic spirit and enables the citizen to interact in the most intuitive way with the very fabric of the urban living and interacting. And many point-oriented experiences are already distributed in the world. Will we be able to achieve this tool? Or will we be always depending on IBM Tivoli to see and manage the city?

I had a dream...

... A dream where the governance had a real time dashboard about the city not far from the SimCity interface. A dream where the decisions about the structure of the city were taken based on this data. And where the information displayed was basically public and not politically filtered. Then i understood. It was basically an instrument for the very fashionable Smart City. The problem is that this instrument does not exit. It's not a strange IBM Tivoli adaptation (you don't want to manage your city through gauges and sliders), and it's not a funny tool custom-designed for a specific city. It should be a set of tools maybe completely independent from each other and only referencing the same backend. What kind of tools should these be? Well, The many open source projects around the world show an enormous interest in those small tools. Would there be a possibility to merge these projects into a larger view on how cities can be managed and transparency in that management can be gained? Let's see the various projects:

• From Italy comes the political part through OpenMunicipio, a tool developed by the developers of OpenParlamento, enabling the tracking of every political decision and every absence and presence of a politician during the various sessions of the council.
• From Canada comes the django-version of FixMyStreet, a british open source tool, clone of SeeClickFix. The idea is good, but it depends highly on google's geocoding system, while using 
• From USA comes OpenTreeMap, to track both the positions of trees in the city and their problems regarding risks for the people below them
• From Brasil comes the webtool enabling Urban Acupuncture, to deliver pressure points to the city and making a better city through those points and activities coming from the people
• From Ukraina comes a non-web tool enabling the programming of the bus schedules, while...
• From France comes a non-web tool to evaluate the various tracks of the bus map
• From the OKFN come all the webapps enabling data management for the opendata-oriented city, and for example the OpenSpending app to display the usage of money.

And in all of this Open Data is both the sideproduct of the new age of public information management, and the source of the whole power of such a tool...

What do you think? Could a non-for-profit create such a tool and start delivering it to governance? Could we nerds finally save the world?