Opening industry data: The private sector’s role in addressing societal challenges

Abstract

This commentary explores the potential of private companies to advance scientific progress and solve social challenges through opening and sharing their data. Open data can accelerate scientific discoveries, foster collaboration, and promote long-term business success. However, concerns regarding data privacy and security can hinder data sharing. Companies have options to mitigate the challenges through developing data governance mechanisms, collaborating with stakeholders, communicating the benefits, and creating incentives for data sharing, among others. Ultimately, open data has immense potential to drive positive social impact and business value, and companies can explore solutions for their specific circumstances and tailor them to their specific needs.

Policy Significance Statement

Society is facing crucial issues such as climate change, poverty, disease, inaccessibility to education, and limited job opportunities. The private sector has the potential to play an important role in addressing these problems, in part by advancing more open and accessible private sector data to spur innovation to combat them.

1. Private Sector Engagement in Open Data for Societal and Business Benefit

Our world faces a range of complex societal challenges, but we have the power to create positive change by working together. Collaboration is the key to unlocking innovative solutions, and this requires the participation of all actors, including the private and public sectors, academia, researchers, and civil society.

Sharing data is a powerful way to drive scientific progress and tackle societal challenges. Private sector companies have a unique opportunity to contribute to this effort by making relevant data more accessible to researchers and other stakeholders. By prioritizing data sharing and breaking down barriers to access, companies can help accelerate scientific discovery and promote collaboration across different fields. This approach can ultimately lead to more effective solutions to complex problems, driving positive social change in the process.

The benefits of open data are not limited to societal impact; they can also create long-term business success for companies that adopt this approach. By leveraging their data assets and collaborating with external stakeholders, companies can achieve greater efficiency and competitiveness while contributing to the greater good.

In recent years, there has been a positive shift in the private sector’s attitude toward open data and data sharing. While it is true that a majority of companies have been hesitant to share their data in the past, an increasing number of private enterprises are now recognizing the benefits of making their data more accessible (Russo et al., 2021).

Among others, Lufthansa (Lufthansa Developer Center, n.d.), Swisscom (Swisscom Open Data, n.d.), Microsoft (Microsoft AI, n.d.), Uber (Uber Movement, n.d.), UK Power Networks (UK Power Networks, n.d.-a), and Walmart (Walmart Open Data, n.d.), are some examples of the shift toward open data, with all of these companies establishing open data portals. It is worth noting, however, that the adoption of open data practices in the private sector is still in its early stages.

Realizing the full potential of open data necessitates a commitment on the part of the private sector to open up their data, not necessarily in its entirety, but for the purpose of advancing societal welfare and expediting the development of solutions to real-world predicaments.

Opening and sharing data can drive business and social value by enabling better insights, fostering innovation, and promoting transparency to address complex issues. Partnering with policymakers, academics, civil society organizations, and private companies to leverage diverse datasets can enable organizations to gain a better understanding of complex societal issues, identify new market opportunities, and drive economic growth. When data is open and has clear licensing arrangements, researchers, policymakers, and innovators can collaborate across industries and disciplines, drawing on each other’s expertise to develop new solutions that address societal challenges while driving economic growth.

Reputational benefits for companies arise from being transparent in their work in social good areas, which builds trust and loyalty with their consumers and attracts top talent. In addition, investors are now placing more importance on environmental, social, and governance (ESG) factors to determine investment decisions (Hale and Thomas, 2019). By opening their datasets to solve specific issues, companies can demonstrate their corporate social responsibility, leading to greater transparency and trust.

In addition to enhancing their reputation, companies that share their data can also benefit from increased stakeholder engagement and collaboration. By working with external partners, companies can leverage their data to cocreate solutions that are more effective and responsive to the needs of society. This can lead to greater innovation, as external partners bring new perspectives and expertise to the table. Moreover, collaboration can help to build relationships with stakeholders, including customers and investors, who share the company’s commitment to social impact.

Sharing data can also help companies to improve their internal decision-making and performance. By making data available to employees, companies can improve their ability to measure progress, identify areas for improvement, and make informed decisions. This can help to drive innovation and improve operational efficiency, while also contributing to broader social goals.

Another benefit to companies for openly sharing data is generating new revenue streams and business models. For instance, McKinsey suggests that economies that embrace data sharing for finance could see GDP gains of between 1 and 5% by 2030, with benefits flowing to consumers and financial institutions. These benefits include increased access to financial services, greater user convenience, improved product options, increased operational efficiency, better fraud protection, improved workforce allocations, and reduced friction in data intermediation (Kerner et al., 2021).

Companies that share data on their products and services with external partners may be able to create new value-added services or develop new products that are tailored to the needs of their customers. This can help to drive revenue growth and create competitive advantages, while also contributing to broader social goals.

Overall, companies that openly share their data for the betterment of society can derive a range of benefits, both societal and commercial. These include enhancing their reputation, increasing stakeholder engagement and collaboration, improving internal decision-making and performance, and generating new revenue streams and business models. By prioritizing their social responsibilities and contributing to efforts aimed at addressing social challenges, companies can build a more sustainable and inclusive economy, while also creating long-term value for their stakeholders.

2. Real-World Examples

When the private sector opens or makes their data more accessible, society and business benefit as illustrated in the real-life examples below.

2.1. Identifying the economic impacts of the Covid-19 pandemic in the United States

By combining anonymized data from 11 private companies and making it available in real-time, the Opportunity Insights Economic Tracker (TracktheRecovery.org, 2021) can help policymakers and businesses in the United States to better understand the current economic landscape. By analyzing the data from the tracker, researchers are able to uncover insights that help to inform.

For example, the data revealed that the implementation of the Paycheck Protection Program provision of the Coronavirus Aid, Relief, and Economic Security Act cost the taxpayer an estimated $377,000 per job paid (Garcia and Smith, 2020). The lack of data hampered the government’s response to target aid to those that needed it the most; instead, the loans were processed based on expediency. Policymakers can now design future programs with this new insight and provide more targeted economic relief.

2.2. Assessing the risk of noise pollution for United Kingdom developers

UK Power Networks, a United Kingdom electricity utility, shared data and built a mapping interface to show the locations of their substations and predicted noise contours around them (Institute of Acoustics, 2022). This map and data were then proactively shared with local government planning officers and acoustic engineers to assist with assessing the potential impact of any proposed development projects in the surrounding area. By sharing this data, the company enabled local authorities to fully assess development plans based on the risk of noise pollution, potentially saving all parties involved from costly retrospective mitigation measures and legal disputes.

2.3. Unlocking retail banking competition in the United Kingdom

In Europe, the Open Banking sector has been driven by strong regulatory initiatives. The UK’s Competition and Markets Authority (CMA) played a key role in kickstarting Open Banking in 2017 after its market investigation into retail banking revealed that the sector was dominated by a small number of banks. The CMA wanted to “increase rivalry in the provision of retail banking service” (Update on Open Banking, 2019). By enabling banking customers to opt in to sharing their banking and credit card transaction data with trusted third-party service providers, new innovations and challenger start-up banks and companies have emerged. This increased competition in the sector, which has brought benefits to consumers.

2.4. Decreasing the accidental strikes on underground pipes and cables in England and Wales

Reports estimate accidental strikes on underground assets cost the UK economy over £1.2 billion a year and put workers in danger. By reducing the number of strikes, industry can decrease the risk to its workers and minimize the disruption to city activities while realizing an estimated total monetized benefit of £3.4 billion, which is £347 m per year over 10 years.

At the Northumbrian Water Innovation Festival in 2017, key industry stakeholders had a common problem to solve and discussed the need for a single map that identified underground assets in order to avoid accidental strikes, streamline their operations, and improve their decision-making capabilities.

To realize this solution, they came together and engaged with the UK Geospatial Commission to develop the National Underground Asset Register (NUAR), which will use the buried asset data from 30 companies to create a digital map of underground pipes and cables. Given the complexity of the project and the number of stakeholders, there were major barriers to overcome such as determining who bears the wider societal costs of utility strikes and who bears the upfront investment, along with addressing concerns about the commercial sensitivity of sharing data among the stakeholders.

The NUAR project demonstrates that when stakeholders are working toward the same purpose, once seemingly insurmountable barriers can be overcome, with benefits to society and business. Of how industry-led open innovation and collaboration effectively catalyzed a policy change at the national level once it was picked up and supported by the Geospatial Commission.

2.5. Incentivizing London utility companies to share data for infrastructure development

The Mayor of London’s Infrastructure team created the Infrastructure Coordination program (Mayor’s London Infrastructure Group, n.d.) to better coordinate infrastructure development and maintenance activities across the city. The program focused on streets infrastructure coordination and utilized a “Dig Once” approach, which incentivized data sharing among stakeholders to reduce disruption caused by the maintenance or delivery of new infrastructure.

Through the program, utilities, highway authorities, telecoms companies, and other stakeholders collaborated to design, deliver, and evaluate a wide range of Street Works projects. The Dig Once approach, which utilized long-term investment data shared by utility companies within the Infrastructure Mapping Application (IMA), resulted in numerous benefits for London’s infrastructure, including reduced road congestion, fewer days of disruption, improved well-being of residents, fewer business losses, lower emissions and air pollution, and increased resilience.

To incentivize data sharing, the British energy regulator Ofgem¹⁴ set financial and reputational incentives for UK Power Networks to collaborate with other utilities. UK Power Networks has made its current and future Street Works datasets available on its Open Data Portal, which is open to the public and updated every 2 hours (UK Power Networks, 2023). The purpose of this move is to encourage collaboration with stakeholders. Overall, the Infrastructure Coordination program demonstrated the importance of incentives and the benefit of leveraging shared or open data to better coordinate infrastructure development and maintenance activities across London and maximize benefits for Londoners.

3. Moving from Challenges to Solutions

Achieving a balance between generating business and contributing to the betterment of society while avoiding risk is a crucial consideration for companies seeking to make their data open or more accessible. Companies can derive reputational and economic benefits, however, concerns regarding data privacy, security, and commercial sensitivity as well as legal and regulatory barriers present obstacles to opening data (Bughin and Manyika, 2013; Fassnacht et al., 2023).

3.1. Challenges

The challenges companies face can be categorized into four areas: privacy and competitive advantage concerns, lack of perceived value, technical challenges, and lack of leadership support.

1. 1. Privacy and competitive advantage concerns

   Companies may be hesitant to release their data, especially if it contains personally identifiable information that could compromise the privacy of their customers or employees. Additionally, they may fear data breaches or unauthorized access to their sensitive information.

   Companies may also view their data as a competitive advantage and may be hesitant to share it with potential rivals, fearing that doing so may provide them with an advantage in the market. They may worry that providing valuable insights to their competitors could cut into their profits.

2. 2. Lack of perceived value

   Companies may not see a clear benefit to releasing their data, particularly if they do not see a direct financial or reputational reward. They may not fully understand the potential additional value that could be derived from their data, especially in the context of social good. Legal or regulatory barriers may also be a concern, particularly in industries that are heavily regulated or where there are comprehensive data privacy laws. In these cases, companies may determine that the risks outweigh the potential benefits of releasing their data.

3. 3. Technical challenges

   Companies may have data stored in different formats, using different standards, or with varying levels of quality, making it difficult to prepare the data for release in a standardized and useful format. Companies may need to integrate data from different sources or systems, which can be time-consuming and require significant technical expertise, which is often difficult due to the talent gap (Mikalef and Krogstie, 2019; Janssen, 2022) in data science and engineering. Additionally, companies may need to secure their data in an enclave for trusted data sharing.

4. 4. Lack of leadership support and investment

   A lack of leadership support and investment can be a significant barrier to companies opening or sharing their data for societal benefit. Without leadership support and investment, concerns over data privacy, legal risk, and competitive advantage may be viewed as insurmountable hurdles. Furthermore, without leadership investment in technical infrastructure and expertise, it may be challenging to release or collaborate around data effectively and efficiently.

3.2. Solutions

While opening and sharing data faces legitimate and perceived barriers as described above, the potential for open data to drive business value and positive social impact is immense. As companies think about whether to share or open their data, it is helpful to explore solutions for their specific circumstances. To address these challenges, companies could consider the following solutions, noting that one solution could address multiple challenges.

1. 1. Anonymize or deidentify data to remove personally identifiable information from data sets, which can minimize the risk of exposing sensitive information. By doing so, companies can contribute their data to initiatives for social good, while safeguarding individual privacy.

2. 2. Build a robust infrastructure for data sharing including secure data repositories, cloud-based data storage, and data-sharing platforms that enable efficient and secure data access and sharing. Also, implement data privacy and security measures to protect sensitive information. Companies can use secure data-sharing methods, such as encryption or secure data transfer protocols, to protect the privacy of their customers and employees. This can help ensure that data is shared safely and responsibly, while also preventing hacking or theft.

3. 3. Developing policies and procedures for data governance is crucial to ensuring responsible and secure data collection, storage, and sharing. By establishing standards and protocols for storing and sharing data in a standardized format, companies can streamline the preparation process while maintaining data quality and integrity. Data quality assurance is essential for building confidence in data and encouraging sharing. Furthermore, developing policies and procedures around data governance, including data access, sharing, and reuse, can help build trust with customers and employees while reducing the risk of data breaches or unauthorized access.

4. 4. Conduct a holistic risk assessment when opening a dataset to the public. For example, UK Power Networks adopted a data triage process which identifies a dataset’s risks across various categories including security, commercial sensitivity, and data quality (UK Power Networks, n.d.-b). This then informs what risk mitigations need to be considered and implemented before a dataset is published.

5. 5. Partner with technical experts, such as data scientists or data engineers, to help prepare data for release. These experts can also provide training and education to employees to build technical capacity within the company and bridge the talent gap in data science and engineering.

6. 6. Consult legal experts to understand the regulatory environment and identify any potential legal risks associated with data sharing. Companies can also create agreements to establish clear guidelines for data sharing and define the roles and responsibilities of each party involved. This can help ensure that data is shared responsibly and in accordance with legal and regulatory requirements, while still protecting individual privacy.

4. Demonstrating the Value of Opening or Sharing Data

As described in the previous section, lack of leadership support and investment can be a significant barrier to a company opening or sharing its data to benefit society. It is crucial to illustrate the value, which can persuade business leaders to support and invest in opening or sharing their data for social benefit. This can involve identifying the societal problems that the company wants to help solve, assessing the value of the data, and communicating the potential benefits of open data to internal and external stakeholders.

1. 1. Identify the societal problems the company wants to help solve by examining the company’s core values and mission, as well as by engaging with company leadership and external stakeholders, such as academic institutions, nonprofit organizations, and government agencies. This process can identify opportunities for collaboration and build trust with stakeholders.

   Also, assessing the value of the data and how it can contribute to solving societal problems will help evaluate the potential benefits of releasing the data, including improving the company’s reputation and brand, generating new business opportunities, and fostering innovation.

2. 2. Make the benefits of open data clear by communicating to internal and external stakeholders the potential benefits of open data, such as increased transparency, improved decision-making, and potential new opportunities for innovation. Companies can develop broad-based public engagement and education programs that raise awareness of the benefits of data sharing, promote data literacy, and encourage public participation in data-driven initiatives. By engaging the public, companies may be able to build support for data-sharing initiatives. It can also be helpful to showcase successful examples of companies that have made their data open to encourage others to follow suit.

3. 3. Create incentives for stakeholders in the data ecosystem to share their data.

   • ○ For companies: financial incentives, such as tax credits or subsidies, or through recognition and awards for companies that make their data open.

   • ○ For employees: financial incentives, such as bonuses or promotions, or through internal awards and recognition programs.

   • ○ For researchers: collaborate with researchers to share data and work together on projects, fund the costs associated with making data openly available, recognize researchers through awards, citations, or other forms of public recognition or provide access to proprietary data that is not publicly available.

4. 4. Build partnerships with external stakeholders (public sector, academia, researchers, civil society, and other companies) to amplify the benefits of opening private sector data. By participating in discussions and debates around data sharing and open science, industry can help identify the most important research questions and the types of data that are needed to address them as well as work to create incentive structures that benefit researchers in academia. Industry can also help to identify areas where data sharing can have the greatest impact, such as in drug development, healthcare, or environmental research.

5. 5. Consider creating a corporate data philanthropy (George et al., 2019) initiative to voluntary donate data, data resources such as analysts, and/or data technologies such as software, computing power, and storage. With such an initiative, companies can set their philanthropic priorities and determine where to focus their resources. Engage with internal stakeholders to determine if there is an opportunity to integrate data philanthropy into a company’s existing corporate social responsibility priorities, which would incentivize leaders to invest in building the infrastructure needed.

5. Industry Partnerships

Creating industry partnerships with like-minded companies, such as the Industry Data for Society Partnership (IDSP) (Industry Data for Society Partnership, n.d.) or the Development Data Partnership (The Data Partnership, n.d.), is a powerful way for companies to harness and compound their collective impact. By pooling resources, knowledge, and expertise, companies can achieve greater impact and accelerate progress in areas such as research, innovation, and social responsibility. Industry partnerships can take many forms, from sharing data and insights to jointly developing new products or services. Whatever the nature of the partnership, it requires a willingness to collaborate, open communication, and a shared commitment to success.

Launching a partnership like the IDSP required careful planning and execution to ensure its success. The partnership launched in December 2022 with seven organizations including GitHub, HPE, LinkedIn, Microsoft, Northumbrian Water, R2 Factory, and UK Power Networks with the aim to bring together like-minded companies to use data to improve outcomes in important social impact areas. Based on the formation of the IDSP, below are considerations for companies who are interested in joining or creating a partnership that focuses on putting data to work.

1. 1. Develop the partnership’s vision, objectives, and responsibilities

   A critical first step in establishing a successful partnership is creating the partnership’s vision, goals, and responsibilities. The vision should articulate a shared purpose and a compelling reason for the partnership’s existence. The vision will help inform the ideal number of partners needed to initially form the partnership, as well as help determine the member profile, including their skills, expertise, and resources required to achieve the partnership’s goals. Generating a frequently asked questions (FAQ) document can help clarify thinking and be a useful tool when reaching out to potential partnership members.

2. 2. Identify like-minded companies and make connections

   Identifying possible partners can be challenging. The lack of a designated person or team responsible for open data initiatives or complex decision-making processes within a company can make it difficult to identify the right person to contact. Additionally, personnel changes, such as when key decision-makers either leave or are new to a role, can disrupt ongoing initiatives, and strict gatekeeping processes to leadership can be barriers to identifying potential partners. To overcome these challenges, consider the following strategies.

   • • Conduct research to identify companies that have a strong reputation in corporate social responsibility, review annual reports, and look for individuals who have written articles or blog posts related to the partnership’s vision.

   • • Leverage industry networks such as industry associations, conferences, and trade shows.

   • • Use social media platforms such as LinkedIn and Twitter to engage with companies and individuals who are active in this space.

   • • Activate personal networks and ask for recommendations from colleagues, industry contacts, or other identified partners.

3. 3. Evaluate partnership models and determine the structure

   It is important to determine the most suitable structure that aligns with the goals and objectives of a partnership when establishing one. There are different options to consider, ranging from formal to informal, depending on the partnership’s needs. Creating a legal entity, such as a nonprofit, may be appropriate, or an agreement such as a memorandum of understanding (MOU) may suffice. Alternatively, a forum, alliance, or pledge could be established without a formal agreement. It is essential to evaluate the partnership’s vision to determine if the selected structure will enable its mission. Seeking legal advice is also recommended to determine if a formal legal entity is necessary to achieve the partnership’s vision.

   Another key point is to establish the partnership’s governance structure, including the appointment of a secretariat or governance body responsible for managing the partnership’s operations and overseeing its implementation as well as the roles and responsibilities for each partner. Additionally, having an understanding on the level of participation required and the specific contributions each member will make to the partnership is critical.

4. 4. Build trust with potential partnership members

   To establish successful industry partnerships, building strong relationships and fostering trust is crucial. While a clear partnership plan and supporting materials like a slide deck or FAQ are essential for effectively communicating the vision and objectives to potential partners, leaving room for their feedback and ideas is also important. Engage in a dialogue that enables potential partners to contribute their expertise to the proposed plan, discuss their internal approval processes for external engagements, and share any concerns they may have. Encouraging open discussion can help create an environment for a productive partnership that is flexible enough to accommodate input from all parties involved.

5. 5. Consider member retention and growth

   Encourage member engagement and partnership ownership by creating workstreams to develop a program of work. For example, the IDSP members identified two bodies of work for the first year of the partnership: launching a Data for Local Environments Challenge (Open Data Institute, n.d.) and working with the National Academies to plan a U.S. Research Data Summit (National Academies, n.d.) to be held in Fall 2023 at the National Academies in Washington, D.C.

   Partnerships may also experience attrition for a variety of reasons. To sustain partnership growth, it is essential to leverage the collective expertise, resources, and connections of each member. This approach can create a pipeline for growth by identifying mutual interests, exploring collaboration opportunities, and sharing data and insights.

6. Conclusion

More examples of open data in the private sector highlight a positive step toward finding innovative solutions to complex problems. The benefits of opening and sharing data are not limited to societal impact but can also lead to long-term business success. By opening up their data, companies can contribute to the greater good while achieving greater efficiency and competitiveness. However, concerns around data privacy, security, and competitiveness remain significant barriers that must be addressed to realize the full potential of open data.

To overcome these barriers, companies can find a balance between business and social value, and identify the societal problems they want to help solve. They can also address the challenges and concerns that prevent them from opening up their data, and create incentives for stakeholders in the data ecosystem to encourage collaboration and data sharing. Building partnerships with external stakeholders is also crucial in achieving these goals.

To fully realize the potential of open data, it is essential to continue addressing these challenges and building trust with stakeholders, including the public, civil society, and other private sector companies. Ultimately, by working together, society can harness the power of open data to create both business and social value, contributing to a more sustainable and equitable future.