General enquiries :
+44 (0)20 7602 6000

Revealing the hidden troubles and journeys in education - 7 big data tips

Wednesday 26 September 2018 Data Insight & Analytics

Marc Radley's picture
By Marc Radley

Producing better outcomes

The big common themes across communities, schools, the voluntary sector and public services involve complex issues regards the challenges of creating a safe environment and a services system that enhances life chances.

I have been working on applying learning about new data analysis tools and approaches using “system dynamics” to make predictions and “artificial machine learning and optimisation” to reveal hidden opportunities. These can be used to manage capacity in a local authority service area e.g. to prevent violence and violent crime such as murder. Insights are generated that equip local partnerships with greater confidence to invest in preventative approaches, putting in place a mix of changes to policy and practice to tackle issues earlier. Further, lower the rates of entry to systems and to know quickly when to adapt as demands change. The methods help quantify the costs of fragmented services and include how these can also damage, as well as they reveal hidden dynamic indicators of transformed cost and performance.

“System dynamics”, “Artificial machine learning” and “Optimisation” methods have been used in the risk and financial services industries for some time. However, because they rely of the availability of industry knowledge, data and information they are also typically a secret ingredient. A recent Nesta report confirms the methods are particularly useful for children’s services because much of the work of commissioners or frontline professionals including teachers and support workers etc. involves complex decision-making with lots of information which is increasingly available to share.

7 TIPS for building Data, Evidence & Insight

How can partnerships strengthen the use of data and information? Nesta identifies that organised multi agency partnership group interaction with data can lead to tangible improvements. But, with so much data available, what can you do to improve outcomes? How can you collaborate to achieve this through multi agency partnership?

1. Use a problem-oriented mindset:

Data is not in and of itself useful, but using data analysis to test hypotheses or solve problems can ensure value is created.

Speech Language and Communication, Special Educational Needs, Disability and Neuro Disability screening in Youth Justice services in England & Wales is beginning to show that understanding underlying unmet needs are significant individual factors in reducing persistent offending behaviour. We are now beginning to show how information about life experiences indicates specific areas for reinvestment in earlier individual support and how these can be dynamically calibrated against reduced risk and the community services costs of child sexual exploitation, knife and drug crime and murder etc. 

2. Integrate data into a data warehouse to enable deeper analysis and use:

Linking data creates a fuller view of issues or individuals, making problem solving or pattern spotting easier.

Linking data from a perspective of understanding multiple Complex Needs highlights how small numbers of  outlying individual child journeys lead to poor outcomes and very high costs (e.g. in Youth Justice). These patterns are not seen in the fragmented snapshot of performance data across operational systems. We can use tools like the ChildView Hub to automatically match data across multiple operational systems; use ChildView youth justice screening and life events data to match against historical school exclusion data to validate the value of cost shunting in our system models. This can led to reinvestment and service targeting and new order performance indicators (typically rates). Other tools such as ChildView QlikSense enable rapid flexible prototyping of joined up data sources without the cost of structured data warehouses and the result is accelerated pattern spotting involving problem solving groups rather than analysts.


Personalised care is driving information to be joined up around an individual, so the person can be understood as a ‘whole’.

3. Enable data sharing through use of case-oriented information governance protocols:

Being specific about the circumstances and purposes for which data can be shared makes it easier to unlock data and integrate it.

Personalised care is driving information to be joined up around an individual, so the person can be understood as a ‘whole’. However, our challenge is that we are still learning what information represents the real lives of people with needs. This means separating out what we can from our existing operational system records and joining datasets together. This is so we can see more of the gaps, overlaps and complex impacts on individuals and not just a process management view. We can now make advances in ‘whole learning system’ approaches by making visible the hidden challenges of understanding and responding to human diversity and disadvantage and through this sustain reduced demand on the Looked After, Criminal Justice and Mental Health Systems.  

4. Support the use of data from the top:

Senior managers and politicians can create a data oriented culture through asking for data as part of decision and policymaking processes.

Partnerships between the key Police, Social Care and Health agencies are supported in generating insight and learning from individual data and evidence concerning lived experience and outcomes and the specific impacts of unintended policy and practice misalignments. Service learning and adaptation can be made in partnership decision making group such as community hubs, fair access panels and other problem solving groups.

5. Invest in the data science capacity needed to perform analysis and integrate large data sets:

Data work increasingly requires data scientists and programmers who are currently rare in the local government workforce.

Data Science is an interdisciplinary field of scientific methods, processes, algorithms and systems to extract knowledge or insight. This is not to be confused with skills in using SQL or using existing database to monitor performance. I am collaborating with data scientists to facilitate tools, dynamic models and dashboards to explore hidden dynamics in interconnected data and systems and also to promote confidence in using insights to make small changes that have an accumulative impact over time, typically counter intuitive changes that increase system capacity whilst reducing overall cost.


6. Take an agile approach to working with data: 

Rapid prototyping, testing and iteration improves the quality of analysis and tools, and helps build momentum.

A partnership problem solving mindset will facilitate and enable teams to experiment with new thinking and adaptation of existing services using evidence and data (new and old). Enabling conversations and culture creates a virtuous cycle of improvement, getting familiar with getting representative data and using knowledge of front line practice to gets source recording that makes contribution to tracking overall system results e.g. how admissions data can be reused to indicate improved response to inclusion which in turn indicates reduced upstream costs.

7. Ensure that infrastructure enables integration of data and analysis:

Without high-speed broadband, data storage options and the right software, data approaches can be held back.

Key to making progress is flexibility to connect and work with large datasets across an easy to reconfigure and appropriate secure and resilience network environment. This flexibility and security is now easy and cheaper to achieve in a Cloud hosted environment where the costs of continuous adaptations and resilience is lower. This can also be available across shared services. 

In addition, network bandwidth and processing speeds can be rented and expanded and contracted when needed, saving time on extract and loading data. The support for prototype data project work can also be more responsive and extra costs of managed contract arrangements minimised.


Revealing the hidden troubles and journeys in education. Seven tips for building big data, evidence and Insight.

Revealing the hidden troubles and journeys in education - 7 big data tips


Add new comment