Automating assessments with AI
Checking, marking, critiquing, assessing, scoring
We recently reflected on a common theme we’re noticing in the work we’re delivering. The projects had different names like “accreditation certifier”, “clause validator” or “skilled migrant assessment”, but they were all doing the same thing, assessing something.
What we also realised is that AI is often great at a first draft, or a last pass. In this case, an initial assessment.
Assessing contracts
These were tasks that in the past were hard for computers to assess because of the text content. Take a contract - in the screenshot below is an example from a recent projects where AI has been told to find discrpencies from rules in a contract. If you look at the rule and relevant exerpt columns, you’ll notice that they are completely different in language but the same in meaning, and the tool recognises this. With rule number 4, it’s picked up that the minimum liability should be $2 million not $1 million, flags the discrepancy and then allows you to view the source.
In this example the problem was that for non-disclosure agreements, most lawyers were typically looking for the same things. IP protection, liability and a few others.
Video and course content assessments
Another example was assessing regulated training content to see if it meets professional standards. In the past someone would manually review every minute of every course to make sure that it met professional standards, but what we discovered is that what they were looking for was very achievable to instruct to an LLM agent.
So now a solution reviews all the training content for markers on professional standards, readability, quality. This process used to take thousands of hours per year, now reduced by 90%.
Another example is automating the review of the content provided by a candidate for a skills assessment. What often happens in a complex assessment, like the skilled migrant assessment performed by many industry associations, is that most of the time is taken up by reviewing academic certificates to see if they meet what was mentioned in the application, or that both sides of the driver’s licence are scanned, or that they’ve provided evidence of employment.
The skills assessment is still performed by a specialist, but all the initial prep work validation is done by the bot.
10 ideas for automating assessment
Tender response meets the requirements
User story quality
Contract criteria
Job application with job description
Tests
Complaints submissions (assess severity and type)
Compliance audits
Customer feedback
Supplier performance
Statements of work
Go through your current process and see what takes you long to review. Find where the bottlenecks are - who are you waiting on, and could this just initially be reviewed by AI.
See you next week
Dawid
Join our next workshop or book in a discovery call with us.