GovPro

Analysing institution-company relationships
Categorical analysis of institutional spending
Mapping out the food purchases of institutions

Inspiration

The 'tragedy of the commons' refers to a situation where individual users act selfishly and deplete shared resources, resulting in communal loss. Flipping this concept, we envision a 'triumph of the commons' in the realm of public procurement data. Inspired by this vision, we sought to create a platform where open, accessible data acts as a shared resource, inviting collective stewardship and innovation. By encouraging government institutions to make procurement data readily available, we aim to tap into the collective intelligence of data scientists and the public, unlocking a wealth of insights and fostering a continually evolving landscape of enhanced strategies and solutions.

What it does

Our app is a user-friendly tool designed to make Iceland's public procurement data transparent and easily analyzable. Users can select a company and view its transactions with various government units over time, providing a detailed look into spending patterns. Additionally, the app categorizes government spending, allowing users to discern what types of products or services institutions frequently invest in, such as software or food and drinks. A unique mapping feature also visualizes where institutions make food purchases, offering a practical way to identify opportunities for consolidation and cost-saving. In essence, the app is a practical, hands-on tool that invites ongoing improvement and innovation, aiming to make public procurement data more accessible and usable for everyone.

How we built it

We built the app prioritizing efficiency and effectiveness. Using Apache Arrow, we managed the data seamlessly for on-disk and in-memory operations, despite its manageable size. The data cleaning process was methodical: initially, data was downloaded and complemented with scraped internet sources. Next, it was tidied into clear tables corresponding with each of the app's three user interaction points. Using Apache Arrow simplified uploading the cleaned data to Amazon S3. Finally, the app was configured to download the data directly from Amazon S3, ensuring that users always interact with the most accurate and up-to-date information. This structured approach streamlined our workflow, making the data management process smooth and reliable.

Official procurement data was available but not conveniently aggregated. So, we created a script to download data month-by-month, navigating through individual xlsx files, and then amalgamated them into a single table—a process taking around ten minutes of runtime. As for obtaining institutions' addresses, surprisingly, there wasn’t a straightforward method. Funnily, we discovered that the necessary details were available in the member list of the Association of Directors of Government Institutions (Félag Forstöðumanna Ríkisstofnana), enabling us to accurately link each institution to an address in our application. We were then able to use this data along with Staðfangaskrá from HMS (Reykjavikurborg actually hosts a version of this on their github as well) to connect each institution to a WGS85 coordinate for drawing maps.

In constructing our Shiny app, practicality and ease of understanding were key. We employed the rhino framework from Appsilon, which significantly improves the app’s navigability and learnability, making it more accessible to outsiders or those wishing to contribute or modify the application. For enhanced reproducibility and version control, we used the renv package in R. This allowed us to meticulously manage the versions of the packages used, ensuring that our app runs consistently and reliably, mitigating the risks of discrepancies or conflicts in the package versions. These strategic choices were aimed at promoting a smooth, user-friendly, and reproducible application experience.

All our code is available at: Data: https://github.com/bgautijonsson/gagnathon_rikiskaup_data App: https://github.com/bgautijonsson/gagnathon_rikiskaup_app

Challenges we ran into

One of the hurdles in our project was sourcing the addresses of various government institutions. While one might expect such information to be readily available, it proved to be a considerable challenge, lacking a direct, consolidated source. We had to think outside the box, eventually finding the needed data within the membership list of the Association of Directors of Government Institutions. This discovery was somewhat unexpected but crucial for enhancing the functionality and reliability of our application by enabling precise location mapping and analysis.

Another challenge was managing the data’s vast scope. Early in the process, we made a strategic decision to narrow our focus, concentrating only on the specific data essential for our objectives. This approach allowed for a more efficient workflow, enabling us to commence the implementation phase more swiftly. It ensured that our energies were channeled productively, focusing on data that would genuinely enhance the app’s functionality and user experience, aiding in our goal of making procurement data more accessible and analyzable.

Accomplishments that we're proud of

We’re proud of several key accomplishments in our project. Successfully navigating the complexities of data gathering and structuring, creating tidy data that is easy to plot and tabulate. Our strategic approach to honing in on specific, goal-oriented data streamlined our workflow, allowing for efficient progression from concept to execution. The data are vast, but we were cool under pressure and managed to create a finished product. Technologically, we effectively utilized tools like rhino from Appsilon and the renv package and created a user-friendly application that encourages collaborative improvement and transparency. By making our code available on GitHub, we’ve embraced openness, fostering a community-oriented approach to public data analysis.

What we learned

We learned the importance of strategic scoping in handling vast data sets, choosing to focus on specific data relevant to our objectives. We learned to think outside the box when looking for data sources, particularly in sourcing institutional addresses. Technically, our proficiency grew through the practical application of tools like Apache Arrow, Rhino framework from Appsilon, and the renv package in R, as well as the use of leaflet for plotting spatial distributions.

What's next for GovPro

The app is designed for easy data refreshes, allowing it to stay current with the latest public procurement activities. Its modular code structure facilitates straightforward updates, ensuring that the app can quickly adapt to new requirements or improvements. The evolution of GovPro is also community-driven; the level of interest and engagement from users will guide its development. With significant interest, we can enhance the app further, focusing on making the data and code more accessible, fostering a user-friendly platform that encourages broader exploration and analysis of public procurement data. As public data becomes more easily available, we will be able to update this application with more utility.