The social value sector is at the very beginning of working out what AI might be useful for. We know all too well how good large language models (LLMs) like ChatGPT are at making broad assumptions and presenting them as facts, and this presents a real risk when you’re tackling complex topics. If the user is inexperienced – or just too trusting – findings can easily lead to serious mistakes.
However, if we take a sensible approach, there should be scope for LLMs to help us manage the complexities of social value, whether that’s by sharpening our processes or digging up angles we hadn’t previously considered.
We were curious to see if AI could help us put together a local needs analysis – traditionally a labour-intensive task involving desk-based research, data gathering and stakeholder engagement.
After all, there’s more open source data available than ever before, and generative AI (while still making people’s faces and hands look weird) is becoming ever-more adept at carrying out fact-finding and analytical tasks.
We went into this experiment clear eyed – you can’t press a button and expect miracles – but also hopeful that there might be some useful findings we could learn from.
We used two people to put ChatGPT to the test – one expert (a CHY social value consultant) who has conducted local needs analyses before and one lay-person who has never performed a local needs analysis, to see whether they would both learn anything new.
Here’s what we found.
We started simple, so there was no surprise that the information ChatGPT returned from this prompt was weak. It merely explained what a local needs analysis is, with very little local context.
Pros: ChatGPT gave seven out of eight of the key themes that CHY would recommend exploring as part of a local needs analysis – not bad.
Cons: It picked a different seven themes for each user – one user’s list was missing ‘environment’ and one was missing ‘deprivation’ – very bizarre! Regional detail provided was generic – most would apply to any major UK city.
Conclusion: Not much use beyond giving key themes to start exploring.
Next, we attempted to make ChatGPT work a bit harder – actually searching for data sources that might be useful in compiling a local needs analysis for Leeds specifically.
Pros: It gave us some slightly more relevant local data under each heading, and named some sources we could look into (e.g. for Health & Wellbeing it pointed us to the Leeds Health & Wellbeing Strategy and Leeds Tackling Health Inequalities Strategy).
Cons: A lot of the reports it recommended were outdated or so vague you wouldn’t know where to start looking for them (NHS CCG “plans”?), and none of the suggested information was linked. We would definitely advise double checking all the stats it gave too!
Conclusion: This gave us a better idea of the kind of resources that are out there, which non-experts may not be aware of.
Next, we attempted to get the AI to drill down and give some direct links so we could find the data sources it was talking about. It produced links, both for national statistics and local.
Pros: Some of the links worked…
Cons:... but the majority no longer existed. The handful of links that worked tended to go to very general landing pages (e.g. the DFE homepage) and it wasn’t clear where to find the data from there. Basically, it couldn’t do what we asked it to do at all.
Conclusion: ChatGPT struggled to locate the accurate data required to make a confident and informed decision about anything local, or anything “needing”. This made us question whether any of its claims were worth including – as they would be challenging to substantiate (it couldn’t even do it by itself!)
Having given up on data sourcing, we wanted to see if the AI could at least guide us in how to structure the information once we’d retrieved it ourselves.
Pros: This is the stuff that ChatGPT is stronger at – it gave us a clear framework (executive summary, methodology, overview, key findings and gaps, strategic alignment etc.) covering most of what you would need to include.
Cons: It made up a wordcount for each section, but didn’t indicate these were only suggestions (bids can be any length!), which could be misleading.
Conclusion: a pretty good use case for AI – could provide a template for SMEs who haven’t had much experience using a local needs analysis as part of a bid.
A final attempt to get AI to give us some actual content we could use! We asked it to write a section of the (imagined by itself) bid it had just provided an outline for.
Pros: The draft it gave on housing provided a pretty comprehensive framework – with sections covering key challenges in Leeds including student housing, urban and brownfield regeneration, homelessness etc.
Cons: Again, there was very little information beyond the section headers. The AI either avoided talking about specific local needs, gave vague information, or cited outdated or irrelevant sources, meaning we couldn’t trust what it said.
ChatGPT gave us some helpful guidance in putting together a generic local needs analysis, providing structure, themes and suggesting where we might find data. Useful for a beginner!
However it was unable to provide any real content specific to the local area and struggled to retrieve up-to-date, relevant data – often giving inaccurate or outdated information as fact. It didn’t draw many conclusions from the data it did find either… so the analysis part was lacking too.
We can see how it might provide a useful guide for SMEs answering a social value question for the first time.
But when it comes to researching local context, finding detail and performing analysis – we reckon human research is still absolutely essential for everyone. For now!
Setting rules for your AI (whichever one you’re using) can be a really good way to get around some of its shortcomings.
We’d recommend teaching your AI to evidence its claims wherever it can by linking back to a source, and where that is not possible to flag that the information cannot be substantiated. That way you can more easily check facts and spot fictions. You can do this with the “memory” functions of ChatGPT. Try “always provide a direct link to any information referenced in a response".
If you or your team have successfully used AI to speed up or improve a social value-related process, we’d love to hear about it! Please do drop us a comment or a DM over on LinkedIn.
.jpg)
-min.png)
.jpg)
.jpg)
.png)
.jpg)
.png)
.png)
