Unlocking the Power of Government Data: Moving from Siloed Information to Smart Public Services

Unlocking the Power of Government Data: Moving from Siloed Information to Smart Public Services

Government agencies are collecting unprecedented volumes of data, yet much of it remains untapped, trapped in silos that prevent effective decision-making and service improvements. This data holds immense potential to transform public services by enabling more accurate, timely insights into service delivery, effectiveness, and efficiency. However, the way data is currently collected, stored, and structured often renders it under-utilised or completely unused.

In this blog, co-authored by Ali Nicholl from IOTICS and Nick Turner from CACI, we explore the critical user requirements for a data-driven smarter state and propose a scalable, federated approach to data discovery, access, and sharing. By enabling real-time data access at the point of need, this approach not only empowers better public services but also provides a coherent AI-ready workflow that leverages existing legacy systems without disruptive centralisation, duplication, or increased complexity.

The Challenge: Making Data Work for Everyone

In today’s environment, where both government and industry are under pressure to do more with less, reduce complexity, and comply with stringent regulations, several pain points persist:

  • Breaking down data silos: Data is often trapped within departmental or organisational boundaries.
  • Improving data quality: Data accuracy and consistency are compromised without a coordinated approach.
  • Addressing data custodianship concerns: Worries around GDPR, security, and data misinterpretation hinder sharing.
  • Ensuring controlled access: Striking the right balance between open access and secure controls.
  • Managing costs: High expenses related to data transit, hosting, and maintenance.
  • Overcoming budget constraints: Investment in new systems while maximising returns on legacy assets.
  • Becoming AI-ready: Adopting new technologies without costly overhauls.

For any system to be truly data-driven, it needs a minimum standard for quality, availability, consistency, and interoperability—without sacrificing security and appropriate access control. It’s the organisations closest to the data sources that have the best insights into managing quality and availability. However, leaving consistency and interoperability solely in the hands of data owners can lead to fragmentation, while expecting any single organisation to manage all data ownership is unrealistic.

The Evolving Solution Space: Technology, Policy, and Attitude Shifts

Recent advances in technology and shifts in policy have begun to address these challenges. Nearly two decades ago, the Reuse of Public Sector Information Regulations (2005) and the Transparency Agenda (2010) laid the foundation for more open attitudes towards data sharing in the UK. The evolution of cloud technology and API-driven architectures has further improved data accessibility by reducing latency and enhancing interoperability.

For example, the UK Transport Sector has effectively used open data APIs to share real-time transport information with developers and service providers, resulting in over 600 apps that benefit millions of Londoners every day. However, while these methodologies improve access, they do not fully solve the “data silo problem”—where data remains fragmented and lacks context, limiting its utility for broader insights.

A Federated Approach: Keeping Data in Place While Maximising Its Value

Our combined experience at IOTICS and CACI has only reinforced how unsustainable current approaches are. A smarter state needs a smarter approach. A federated approach. A federated approach allows data to stay in situ within its existing silos, accessible through a controlled, consistent, and extensible framework. This approach eliminates the need for costly mass data migrations while still unlocking insights at the point of need. Creating a more equitable democratisation of decision-making by ensuring that the right data is available at the right time.

This methodology aligns with how Health Services in the UK have approached data integration in recent initiatives. Within Social Care Networks, for example, connecting existing systems rather than centralising all data has ensured the Healthcare sector maintains flexibility to access relevant information while adhering to security and privacy requirements.

Data visualisation

Understanding Stakeholder Needs: Tailoring Solutions for Maximum Impact

Different stakeholders have different goals and challenges when it comes to leveraging data. Here’s how a federated approach such as ours addresses their specific pain points:

  • CIOs need timely, reliable data for informed decision-making. Our solution ensures up-to-date insights without the need for complex data migration, helping CIOs set policies and make strategic decisions with confidence.
  • Heads of Data and CDOs seek to maximise ROI from data assets. We provide enhanced data discoverability and governance, ensuring that those who need access can find and use data efficiently.
  • Service Owners focus on delivering policy or strategic outcomes. Our approach reduces the under-utilisation of data, enabling service improvements without significant operational disruption.
  • Data Analysts require consistent and high-quality data for accurate analysis. By maintaining data integrity and enabling seamless integration across sources, we empower analysts to deliver actionable insights.
  • End Users demand real-time access to relevant data without navigating multiple platforms. Our solution brings data closer to its source, maximising relevance and minimising inconsistency.

Building a Data-Driven Smarter State: The Path Forward

Creating a data-driven smarter state requires lowering the barriers for departments, organisations, and individuals to surface their data and enrich it with context, turning it into actionable insights. A federated approach represents a scalable, flexible, and low-risk path towards unlocking the full potential of government data. The journey from siloed information to integrated insight is not just about technology; it’s about creating an ecosystem where data flows seamlessly, fostering collaboration, innovation, and smarter decision-making across the public sector.

To build this future, we must prioritise accessible, context-rich data and scalable collaboration across stakeholders. The smarter state of tomorrow is within reach if we embrace these principles today.

This blog was submitted to TechUK as part of their “Building the Smarter State Week” and can be found on their website here.

Why data insight projects fail and how to succeed – Part 4

Why data insight projects fail and how to succeed – Part 4

In this 4-part blog series, we look at some of the common reasons data projects fail and the key underlying solutions that can help businesses improve project success rates. Part 1 dealt with outcome misalignment and the problems of a project failing to align with business strategy. Part 2 dealt with estimates and what problems they can cause. Part 3 dealt with project scopes and why they are so important. This time we look at what underpins any successful project, time and resources.

Time and resources – without people, there is no project

As we mentioned in part 3, your estimates will also include estimating the time and resource you’ll need on a data insight project.

Building any project delivery schedule relies on realistic timelines, which can be difficult to produce for projects your team may not have executed before. While elements may be familiar, more and more large projects such as cloud implementations put IT teams on unfamiliar ground, making accurate timelines incredibly hard to estimate (even if everyone is available for your project 100% of the time!).

Time and resources

Reaching out to 3rd parties that specialise in the kind of project you are undertaking can help you set a realistic project time estimate and schedule, helping you better manage your internal stakeholder’s expectations of time to delivery and not feeling burdened by unforeseen delays.

This can be especially useful on Digital Transformation projects where data is a critical point, as specialist 3rd parties can support you on understanding the resource requirements based on your data maturity and current BI tools.

It is important to select the right supplier for your organisation. There are three key areas to validate that the 3rd party are the right fit.

Firstly, a strong technical acumen is important; a common trait amongst 3rd parties will be their ability to deliver through a competent delivery team who are well skilled in the chosen technology.

Secondly is a clear awareness within your industry. Understanding the subtleties of your industry is vital to being able to create and support effective solutions. There is no one-size-fits-all for data, so it is important that you are able to maintain and enhance your reputation by developing outputs which are focused and relevant.

The third, and most important area, is that the arrangement is a partnership. The best projects are those where collaboration is built into the core of the delivery. Each business has unique nuances that mark it out from the competition. Being involved throughout means these nuances are guarded and the context of the project remains relevant and specific to your business. When considering procuring the services of a 3rd party, it is important that you are not dictated to, particularly with complex data projects. On conclusion of a project, you want your staff to feel confident to be self-sufficient in the management of the developed solutions, so it is imperative the project looks at developing staff skills as well as delivering high quality outputs.

Alongside this, it goes without saying, people are the key resource on any project – without people, there is no project. Many IT teams report being overstretched, often working on too many projects at the same time, or on projects with too few resources available to meet deadlines.

The initial scoping process should give the chance for an open discussion about which resource is needed and when on a project, as well as how much of their time each project team member (often with a workload outside of the project) will need to spend on the project on a weekly basis.

Defined roles in projects allow for clear differentiation of responsibilities. The previously accepted approach for data insight projects to be delivered was by those within IT. There is more of a need now than ever before that the driver of data projects is the business users who will utilise the output. Ensuring subject matter experts are available to feed into the requirements phase to contextualise complex data constructs and also to provide their feedback within user acceptance testing to prove the validity and usability of the output is essential.

Clear differentiation of responsibilities

Risk

Underestimate time or manpower on a project and your project might not fail completely, but it’s far more likely to run dramatically past it’s delivery date by weeks, months or in some cases even years. These delays create other bottlenecks around the organisation and can cause a break down in trust between you and stakeholders from not keeping your promises. Avoid this entirely by being realistic with your time estimates and seeking outside insight into requirements where your organisation has a knowledge gap.

Reward

Understanding the current state of play with the moving parts in your project helps you understand what resource you need. With this insight, you’ll be able to meet your deadlines by securing the right people for your project at the right time.

Our customers often reach out to us at the beginning of new data projects for aspects such a current data maturity assessment or infrastructure assessment that supports them in structuring their projects and allocating the right resource. Once into a project, our customers know that as a trusted partner, if they need some extra support from us, they can pull from our bank of services at any time to ensure they deliver and meet their data goals.

Conclusion

Throughout this series, we’ve covered outcomes, scope, estimates and resource planning, and it’s clear that when it comes to big IT projects, there are a number of risk factors that can cause a project to fail, run over-time or over-budget.

When we begin to mitigate these risks by taking the appropriate steps at each stage, projects are more successful and other benefits such as improved stakeholder relationships and employee job satisfaction begin to filter through.

Through planning, detail and communication, projects are infinitely more likely to be successful, however this can be easier said than done for busy IT departments who are usually juggling multiple projects in parallel.

This is why CACI’s Business Insight Group partners with customers and can support them at any point of their data journey, from aligning their outcomes through to post implementation training.

Find out more about our services that help you succeed in Digital Transformation projects.

Why data insight projects fail and how to succeed – Part 3

Why data insight projects fail and how to succeed – Part 3

In this 4-part blog series, we look at some of the common reasons data projects fail and the key underlying solutions that can help businesses improve project success rates. Part 1 dealt with outcome misalignment and the problems of a project failing to align with business strategy. Part 2 dealt with estimates and what problems they can cause. This time we look at why a project scope is so important.

Scope of the project – the devil is in the detail

In order to deliver a solution that everyone understands and is happy with post project completion, a clearly understood project scope is essential. This details what you are and are not going to deliver.

Project teams encounter challenges frequently during projects with additional requirements creeping in that were outside the initial scope. Through the previously mentioned outcome alignment and relevant stakeholder engagement during the scoping process, the risk of unforeseen “new requirements” (that either cause your project to run over time or budget) can be significantly reduced. This is not to say that the project cannot be agile to change but having a laser-focus on the outcomes and scope provides a solid foundation to effectively control such change and manage stakeholder expectations.

Scope of the project

This process is often rushed in a bid to see projects kicked off faster, but without a well-defined scope there is a risk the development team may not have the full picture of what needs to be delivered and stakeholders may have mis-aligned expectations as to what is in the plan.

When considering how to scope a data insight project, there are some key considerations that will allow for effective planning:

Inputs

Data: clarity on WHAT information is needed. This element needs to consider whether or not the requirement information currently exists; if it does, where is it? If it does not, how can it be obtained?

Quality: the delivered solutions will only be as accurate as the data feeding them. It is important to understand within the scoping exercise how that data quality can be managed.

Knowledge: there are a large number of people who can manipulate and utilise data. The value in your business is those that can appreciate the CONTEXT of the data. These are key stakeholders who can explain the ‘why?’ and ensure the final output is relevant.

Outputs

Delivery: some projects will require a single output produced at a particular point in time, others will benefit from outputs delivered at regular intervals to inform the users. The method of delivery to those users is also important – do they have to retrieve the information from a location or will it be delivered to them via email for example.

Audience: it is important to consider two things with the audience – knowledge of the data and the position held in the business. More senior stakeholders will require strategic information to make decisions, service teams will require operational level data to be able to perform their day jobs. It is also important that enough information is provided. Some consumers will have a more intimate knowledge of the data so will not require as much explanation, but others may need additional detail for clarity and confidence in making decisions.

Intended use: If the purpose of the development is to be able to forecast future activity, it is not enough to provide the audience with just the current position; there is a need for historic data to map trends and start to look at how that may be influenced moving forward. Conversely, if the purpose is to performance manage a member of staff, only providing a forecast of potential future performance is not adequate to be able to analyse historic activity.

By aligning your outcomes at the very beginning, this should help inform the scope of the project, as there will already be a level of agreement on what the project needs to help the business achieve.

Business Woman conducting a meeting

Risk

Without a full project scope, delivery may not resemble what everybody expected. This could be the result of a 3rd party not having the full project requirements, or an internal stakeholder expecting something different than what was planned. Mitigate this risk by producing a full and detailed scope of your project.

Reward

Fully detailed project scopes keep everyone on the same page, specifically your project team and internal stakeholders, as well as any 3rd parties involved in your project. Scoping provides an element of project transparency that ensures you’ll be able to better manage expectations.

Project structure and process, especially in the beginning phases of a project are critical to set the tone for what is to come, which is why our team of experts have developed a project framework with criteria and process at each stage to support delivery of successful projects. Find out more about the Fusion framework here.

In part 4 of this series, we’ll be covering the all-important areas of time, resource and people on your project.

Why data insight projects fail and how to succeed – Part 2

Why data insight projects fail and how to succeed – Part 2

In this 4-part blog series, we look at some of the common reasons data projects fail and the key underlying solutions that can help businesses improve project success rates. Part 1 dealt with outcome misalignment and the problems of a project failing to align with business strategy. This time we look at estimates and what problems they can cause.

Estimates – Balancing the books and the benefits

Almost every organisation would like to deliver more for less in the shortest space of time – after all, who wouldn’t?!

When undertaking any project, you’re going to need to provide estimates on the time it will take, the resource you will need and how much it will cost. This will likely first be highlighted in your business case or business analysis for the project.

Estimates can present challenges for many IT departments, where often the focus is on “how much is it going to cost?” and pressure on how to keep that number as low as possible. While some projects have a relatively low investment, larger enterprise projects will require a significant investment from the business.

This is an area where your business case is absolutely critical for your project to secure the stamp of approval. Many projects are not approved at this stage due to a poorly developed and formulated business case.

The very best project can be delivered but, if the business case was not correctly targeted, the outcome delivered will miss the mark.

There is a need to pivot the conversation within businesses. The conversation should not look at “how much is this going to cost?”. The question should be “how does that add value to what we do?”. The best business case is written from a perspective of the reward far outweighing the risk. Being able to articulate business benefits will automatically shift the narrative from the negative to the positive.

Project cost is relative. Delivering 10 cheap projects that all fail is not good business. Delivering a single project with a greater level of investment that improves the outcomes of customers, whether directly or indirectly, should be far more preferable.

To avoid the possibility of your project not being approved – especially in the case where the size of investment may give stakeholders the need for pause – it’s crucially important to:

  • Define the need within the business: current problems or gaps your recommended solution will solve
  • Determine the desired outcomes: describe the positive improvements to the business when your solution is implemented

Risk

Without a well-developed business case, your project may not be approved – meaning it’s failed before it’s begun. Lowering costs on your estimates in order to gain sign off may increase project failure risks or mean you need to request further investment halfway through. Overcome this challenge by creating a persuasive business case which shows your stakeholders why the need for this solution is business critical and highlight the valuable outcomes your solution will help them achieve.

Reward

Well developed business cases are not only more likely to be approved, but also facilitate a successful outcome delivery with the most business benefit.

Our team of Business Analysts work with organisations across the UK at the beginning of data journeys to build impactful and persuasive business cases that help secure investment in a solution best fit for purpose. Find out how our team could help you.

Next time, we’ll look at how project scope can impact delivery, benefits realisation and affect managing expectations on your projects.

Why data insight projects fail and how to succeed – Part 1

Why data insight projects fail and how to succeed – Part 1

Reports across every industry tell us that a very high rate of IT projects will fail to deliver on one or more aspect.

On average, only around a third of projects are delivered on time, in scope and without budget overrun. Data Insight initiatives are no different. As data analysis solutions become more and more important to organisations if they want to stay competitive, manage processes and improve operations, the drive for these projects to be successful is on a steep rise.

In this 4-part blog series, we look at some of the common reasons data projects fail and the key underlying solutions that can help businesses improve project success rates.

Reason 1 – Outcome Alignment. What does the future hold?

The first and most important part of any data analytics project is to map out its real purpose and create alignment with the business strategy.

Organisations are often presented with requests for such solutions accompanied by the case “we need to bring our data together to create more value” or “this will solve the data transformation problems we currently have”. However, these reasons don’t address the true benefits that could be realised and this makes return on investment (ROI) measurement very difficult.

You’re about to allocate valuable time, resources and investment into your next solution, so in order to deliver a fit for purpose solution and measure ROI you should always begin at the end. This means mapping out what improves in your organisation once your project is delivered. Alongside this, do these improvements align to your overarching business strategy, goals and objectives?

A common error is to confuse the features and benefits of a solution with the outcomes delivered through implementation or its use.

One such example of this scenario is where businesses believe procuring a model toolset is an outcome. In the world of analytics, standing still is no longer sustainable and businesses must continually evolve with the ever-changing technology landscape. By recognising that the outcome from the analytics project is to maximise the value that can be derived from data assets, the roadmap is set for a solution to be sought. This may involve the use of modern data tools, a strengthening of the data governance or could be a change in the approach to how data is utilised.

The following represents an example of a well-constructed outcome:

  • Feature: Embedded analytics in selected solution
  • Benefit: Users are able to easily analyse data and generate insight
  • Outcome: Company saves 15% in costs from identifying operational inefficiencies

By establishing the desired outcomes before undertaking a project, you can better manage expectations and are more likely to deliver a solution that drives tangible, positive improvements and experiences.

Risk

Without aligning the outcomes of a data project to your business goals and objectives (as well as your IT strategy), you run the risk of failing to deliver tangible improvements. When ROI is evaluated at a later date, if there are no measurable improvements, it can create challenges when seeking approval of future projects.

Reward

Proposing a solution mapped to outcomes that align to your business goals creates a direct correlation between your project and a positive impact. By taking this approach you build better relationships between IT departments and senior stakeholders, as well as ensuring everything you do is continually helping your organisation. This is a virtuous circle, which generates an appetite for continual improvement through the use of data insight.

In order to put data at the heart of a business, it must be used to deliver outcomes that shape, develop and advance the business in all areas.

Our team of experts work with organisations across the UK running workshops with stakeholders to help align solutions with clear business outcomes. Find out more about our data discovery process.

In part 2 of this series, we look at estimates and how to build a well-developed business case to secure investment for your project.

Data catalogs: are they important and do I need one?

Data catalogs: are they important and do I need one?

In the age of big data and self-service analytics, data catalogs have become the standard for metadata management, helping analysts and other users find the information they need and get the visibility required for good data governance. But what is a data catalog, how do you know if your business needs one and crucially – how do you go about implementing a successful data catalog?

What is data catalog?

A data catalog is a collection of metadata, combined with data management and search tools. It maintains an inventory of data assets through the discovery, description and organisation of datasets. Through providing context to the relevant user (be that an analyst, data scientist or any other key stakeholder), they are able to find and understand datasets in order to extract business value.

A good data catalog should act as a single version of the truth through unification of all sources of metadata which can be disseminated to the right people at the right time, with the right permissions in place – making it both collaborative, compliant and secure. When implemented correctly, a data catalog will provide you with a clear understanding of your datasets in order to unlock the value you in your data and improve business intelligence and insight.

How do I know if my business needs a data catalog?

If your business is getting serious about a data-driven strategy, you’re going to need a data catalog.

The drive to implement a data-driven strategy is usually born from a company’s desire to transform the way they do business through use of data analysis and interpretation to make more effective and informed decisions.

The natural first stage of this strategy is to identify, combine and manage multiple sources of data. It’s not unusual for a company’s data to be siloed across multiple systems and sources such as data warehouses, data lakes, legacy systems and cloud-based repositories. Bringing these sources together into a single version of the truth helps solve both the challenge to the user of seeing the full picture, as well as the data governance issue that fragmented data presents.

If you’re looking for your users to gain effective and meaningful insights from the data held across your organisation, an optimal way to achieve this is by using a data catalog.

A data catalog can help organisations who are struggling with manually identifying data assets that deliver value as well as improving data efficiency, creating more context around your data and reducing the risk of errors in your data.

But by far the biggest benefit we see in companies with a data catalog is the impact on self-service analytics. With a continued increase in the number of stakeholders across an organisation that use analytics to support their activities and decision making, it becomes critical for data to be transparent and accurate, as well as accessible at the point of need, without the admin required to simply find and understand the data.

Through use of a data catalog, business and data analysts can search and find the data immediately, with access to all of the relevant datasets, helping them to evaluate and make informed choices. Quality of analysis is then improved by users spending sufficient time on the analysis over the admin of data preparation.

Our experience

Through CACI’s work with one of the world’s leading global financial services organisations, our team have deployed an enterprise data catalog that allows full visibility of all streams of data. This has supported improvement of data quality, as well as ensuring GDPR compliance.

By being able to see the full lineage of the data – where it has come from, where it is going and who will have visibility to what datasets, this institution is empowered to implement a level of data governance that can only truly be achieved through cataloguing to the most granular detail.

Any changes to the data are immediately highlighted so that appropriate action can be taken, or it can be used for future insight.

If you are looking for a data catalog, but aren’t sure on where to get started, why not speak to one of our team who can guide you through the steps of the process.