Opening big data for urban innovation

Writer: Tan Chui Hua

The advancement in big data analytics allow cities to gain insights, and catalyse urban innovation. The emergence of huge data sets, however, poses challenges in access and security, and deriving relevant insights and applications. To mitigate these challenges, open data initiatives are gaining traction.

Cities mentioned in this article:
Santander, Singapore

Image © Kris Krüg 2013

Technology company IBM estimates that the world produces some 2.5 quintillion bytes of data every day ↗ (opens in new tab). Sensors, mobile phones, machines, wearable technology and social media – these are just some of the diverse sources feeding the data torrent. As technology and the Internet of Things become increasingly intertwined with work and life, the volume of data generation likewise is set to grow in tandem.

Large data sets are not new to urban planning. Municipal authorities have always worked with disparate and voluminous data sets such as utilities usage, traffic patterns, household information and income distribution etc. What has transformed the landscape now is two-fold – the exponential growth in data volume, and the extensive coverage of data arising from the surge in number of connected devices and sensors in the city.

The city of Santander in northern Spain illustrates how connected devices increase the volume and scope of precise, location-specific data. Residents may use their smart phones to send physical sensing information such as GPS coordinates, environmental information on issues such as noise and temperature to the public SmartSantander platform. With their phones, they may also tag points of interest in the city such as parks, shops and attractions to distribute location-based information.

For urban planners and city governments, such granular views into living and working in the city are important data sets that can form the foundation of policy formulation and execution. Coupled with big data analytics, today’s big data sets open up new and exciting possibilities in gaining data-driven perspectives, precise mapping of behaviour and trends, and developing data-driven urban solutions and innovations.

Challenges in utilising big data

In utilising big data, city governments have to contend with a number of common challenges posed by big data’s characteristics, such as the rapid generation of vast sets of real-time, continuous data, the variety of data formats ranging from textual and visual to video, as well as the problem of noise, biases and authenticity in data.

Traditional databases and tools no longer suffice to cope with the increasingly complex and unwieldly data sets. Given that cities are the engines for human development, it is critical that governments leverage advanced computational methods and analytics to explore the city’s data and find data-driven solutions for urgent urban issues. At the same time, while governments are the traditional owners of data, expertise and innovation in big data analytics usually reside in the industry rather than within the government. Tapping on resources in the private sector is thus pivotal to the effective utilisation of big data.

Engaging or partnering technology companies directly to develop end-to-end solutions, from data collection and integration to designing real-time responses, is one possible approach. The partnership between the city of Rio de Janeiro and IBM to develop a large-scale sensor-based system is a case in point.

Nonetheless, these approaches still grant limited, or rather, specific access to the expertise, innovations and solutions available in the industry. Open data initiatives, however, allow city governments to cast their nets wider and crowd source innovation.

“Given that cities are the engines for human development, it is critical that governments leverage advanced computational methods and analytics to explore the city’s data and find data-driven solutions for urgent urban issues.” — Writer

Open data initiatives: for and against

According to the World Bank, more than 250 governments at national, subnational and city levels and almost 50 countries have launched open data initiatives to date ↗ (opens in new tab), wherein the data is both technically and legally open. This means that users are able to download raw, machine-readable data, and reuse and repurpose it legally.

The arguments for such projects are well-established. The government’s data is funded by taxpayers and therefore should be publicly accessible. Opening raw data has its extolled benefits. It can help catalyse urban innovation, create business opportunities, improve urban governance and living conditions, inform policy making and bring about greater transparency and accountability in governance. It also allows governments to access the expertise of a much larger, more diverse set of users as they explore and use the data.

However, making city data accessible and useable is a complex operation. Such data sets are often non-standard, lacking in contextual information and varied in formats. Considerable financial and manpower resources are needed to standardise big data sets while retaining the granularity to support analytics.

In releasing data sets, city governments also need to walk a fine balance between providing useful, detailed raw data and protecting confidential data that may compromise national security and the privacy of individual citizens.

Despite these pressing concerns, city governments worldwide are beginning to see the benefits of open data initiatives realised. The trend in opening up data is further boosted by growing societal expectations of participatory governance. That the open data movement is the wave of the future can be seen in the growing number of such projects and the launch of toolkits by organisations such as World Bank to help governments start open data initiatives.

Catalysing data-driven solutions

When it launched in 2009, the United States of America’s open data project had only 47 sets of data. Six years on, there are over 130,000 sets of government data ↗ (opens in new tab) and users range from civic hackers (citizens who collaborate with others to solve civic problems using open data), designers to researchers and developers.

The litmus test of open data projects lies in the application. In the case of the United States of America, they found a slew of companies harvesting the data to create services and products, from tracking and comparing price of organic produce across states, visualisation of agricultural yields and job matching to geospatial analysis and travel deals sites. Companies who have benefitted from the open data spread across a spectrum, from multinationals such as LinkedIn and Foursquare to localised, smaller entities.

Over at United Kingdom, the government’s open data initiative has been reaping results as well. Technology companies have found business in cleaning and repackaging data from data.gov.uk. The packaged data is then purchased by developers and start-ups to create relevant services and applications.

Economic benefits are only part of the story of open data’s impact. For instance, using data from United Kingdom’s Environment Agency, a software company had developed an app which allows residents living in flood-prone areas to monitor local river levels and assess the risk of floods. In 2012, using transport data from the government, the American Red Cross launched the Hurricane app, which helps people find safe ways to move around using open roads, rails and airports in case of disasters. When Hurricane Sandy struck, the app was downloaded by some 700,000 during the first two days alone.

“Governments need to be open to innovation from the private sector, because the industry drives innovation.” — Donald Pressley (Senior Vice-President, Booz Allen Hamilton)

Towards collaborative data-driven innovation

For open data projects to be successful in catalysing urban innovation, there are several factors at play. Strong government leadership in releasing data, setting the terms of use and providing public access lay the foundation for any open data initiative.

Leveraging big data analytics capabilities and know-how in the private sector is another key component. Donald Pressley, Senior Vice-President at technology consulting firm Booz Allen Hamilton, which has collaborated with numerous governments on smart nation projects, shares, “Governments need to be open to innovation from the private sector, because the industry drives innovation. Dialogue between the two can be beneficial as the private sector has industry expertise and information that governments do not necessarily have access to, and vice versa. Sharing the data helps to generate more synergy and make for more effective smart nation projects. The private sector can also help government officials shape regulations and policies to reflect ground reality and the government can open the policies to the public for further dialogues.”

To harness collective intelligence and crowdsource innovation, city governments also need to engage with more stakeholders, including its citizens and civil society, to see open data as a common resource. Hackathons and datapaloozas have been shown to be effective in garnering ground enthusiasm and generating new and exciting insights into a city’s data. For example, the American Red Cross’ Hurricane app was the result of one such hackathon where developers, businesses, non-profits and government personnel were invited to work on solutions using the open data, to better the lives of Americans. Worldwide, there is now a growing movement of community-driven data hackathons, such as DataKind, innovating for better solutions.

Open data initiatives are relatively new to the scene and their impacts are only beginning to be felt in recent times. Early results in open data pioneering countries such as the United Kingdom, United States of America and Sweden have been demonstrated that their initiatives have not only stimulated the economy and catalysed innovation, but have also brought communities together in collaborating for a better data-driven future.

The recognition of big data’s potential is leading some governments to recruit data scientists and experts to drive smart nation visions. Singapore, for instance, has been putting in concerted efforts to appoint data scientists as part of its bid to become the world’s first Smart Nation. Through a government data portal, vast public data sets are also released to the public. Regular civic hackathons focusing on an array of urban issues from public transport to shipping are organised with the support of the government and the industry. In such events, one sees the future of big data and the city – where public participation, government initiative and enterprises collaborate for urban innovation.