There is often a lot of debate about how effectively Trusts like Indigo spend their money and it is something all our staff and trustees spend a great deal of time thinking about! Questions concerning how we measure success are very complicated and difficult.
One area of our work that can produce a whole plethora of doubts and insecurities is our work in making government more open, accountable and democratic. Philosophically, this work is so important and we have strong feelings about why we should continue to fund it, but we are aware that in the field of open data, transparency and accountability, it can be hard to directly measure the impact of a project initially and it can take quite a while to see definite social change. Are citizens more engaged? Have politicians changed their ways as a consequence of projects and is service delivery actually improving?
Fundamentally, whilst it can be difficult to measure, we believe that enabling citizens to access information on how they are governed and to actively participate in democratic processes will positively impact on societies. We know people care about the way in which they are governed and the kind of society they live in. What we have also discovered is that patience is key – sometimes it can take time for an intervention to stimulate change and it can also be difficult to monitor impact. We are convinced our work makes a difference, but for those of you with doubts about projects looking to strengthen democracy across the world we thought we would try to change your mind!
mySociety is one of the leading organisations trying to engage and empower society in government. In Britain, their websites TheyWorkForYou, WhatDoTheyKnow and FixMyStreet have become tremendously successful and their open source software model is spreading these exciting ICT initiatives across the world. We asked Paul Lenz, Chief Operations Officer, some questions to find out more about the success of their model and some of the difficulties in driving forward democratic change.
1) Why is it so hard to measure the impact of projects like those of mySociety and how long do you think it takes before real change starts to happen?
I can answer this in the context of a number of different projects – but perhaps the most relevant is looking at Mzalendo, Kenya’s parliamentary monitoring website. First of all, I think that there is clear distinction that needs to be made, up-front, between outputs and impacts. Neither are likely to be delivered quickly, but in every case the impact will be delivered after the outputs.
In the case of Mzalendo getting the initial, core output – the re-launching of the site – took something like a year from the initiation of the project.
The amount of time an intial output takes is almost always specific to the project in question. For Mzalendo we had to recruit a local lead to manage data sourcing, outreach and awards and we also had to build a new software platform from scratch (although based upon the principles that we had already established with TheyWorkForYou). There are also some factors common to the establishment of pretty much any new parliamentary monitoring site:
a) It takes time to identify local partner, agree the areas of responsibility between the organisations and initiate the project. In some instances the partner will be seeking funding for their side of the project separately which can take even more time.
b) Parliamentary monitoring /democratic engagement sites require a significant amount of data to make them viable. Frequently the very value these sites provide is a product of the fact that this data is not readily and publically available. It therefore takes time to gather, assess, clean and format the data required for the site. The starting point is normally a ‘wish-list’ of ideal information which gets whittled down into a minimum viable launch site based upon availability and accuracy.
For Mzalendo this work included going through a 40,000 row data-set, line by line, to correct it and building a bespoke tool to download work with Kenyan Hansard. Some areas are still the focus of work – I have been attempting to make an education data set usable on and off for eight months now!
Once the project has created its initial output, there is then the question of impact. This can follow on swiftly behind output but such instances will largely be dictated by electoral calendars rather than any specific action on the part of the project managers. If the launch coincides with a significant electoral event then it can ride on the coat-tails of the election which will usually see a spike in public interest about politics. Our sites can become very popular during elections as they provide voters with specific information that will be of use to them.
I am perhaps taking a fairly critical view when I think of impact in this context. Some measures will almost immediately show that ‘things have changed’. With Mzalendo, we saw a large spike in re-launch traffic and since then the site has continued to show a significantly higher number of visitors and page views per visit than previously. Clearly, with a totally novel site all such traffic is an upside.
True impact though, I think, should be measured in terms of behaviour changes – are citizens more engaged (and how does this manifest itself)? Are politicians more accountable? Creating behaviour change is both harder to measure and less likely to happen.
We do, however, have a number of reasonable proxy indicators: the number of people making comments on their politician/place pages, the number of people downloading information (for instance constituency handouts), the number of politicians who contact us themselves about the site.
The reasons that it can take a long time for these behaviour changes to emerge are many and will vary from country to country. Initially there is simple awareness, both awareness of the site and understanding that it is a credible, impartial source of information. This latter point is one we always work hard to ensure is made clear.
Then there is confidence. The types of locations in which these sites can have the most impact are often those with a history of disengagement by voters – or worse – by their politicians. Citizens are slow to comment or contact their MP, because they believe that nothing will be done or that they might suffer reprisals. Such situations can only change through an organisation taking the first step – the first site being created – and then working over a period of potentially years to build that public confidence.
2) What are some of mySociety’s greatest impacts to date?
FixMyStreet: Over 250,000 problems reported, with 50% of users having never reported a problem to their local authority before
TheyWorkForYou: Over 1.5m unique users a year. From surveys we know that three out of five users had never looked up information on what their MP was doing in Parliament before using TheyWorkForYou and about 90% of users said that TheyWorkForYou had improved their knowledge of their political representatives.
WhatDoTheyKnow: Over 130,000 Freedom of Information requests issued so far.
Personally, I think that the relaunch of Mzalendo has been a great success for just one specific reason. One of the candidates running for President in the 2013 Kenyan Election contacted the site personally to query the data behind their scorecard rating. If Presidential candidates care about how they are being represented on the site, then it is having an impact – not least because awareness that they will be held to such a level of scrutiny in the future could help drive higher standards of behaviour.
3) A key feature of your organisation is its decision to make open-source software for these public engagement sites. Can you give us some examples of people who have taken mySociety’s model and applied it in their own context?
There are lots of examples of this, but the best ones I would choose are:
a) The Irish ‘FixMyStreet’ has only just launched, but they have used our code, without our help and look like they have done so successfully.
b)The Norwegian ‘FixMyStreet’ is a site we built in association with a Norwegian group – it’s doing very well.
c) Hungarian Alaveteli, or freedom of information request platform, was made re-using our code
d) Parlamany is the Egyptian version of Mzalendo. It was made using our code, but completely independently of us – we didn’t even know about it until after the launch! This really shows how re-useable our code it!
We look forward to watching some of the other democracy projects which we are supporting across Africa grow and stimulate change across the continent. We know this takes time, but we have faith that, if we’re patient, citizens will gradually become more informed and better able to actively participate in their societies.
mySociety is currently identifying potential partners across Africa who are interested in replicating a Mzalendo type of site in their country. If this is of interest to your organisation, please let us know.