How we recruited assurance talent in BJSS

From no experience to decades, with the same three excercises.

How did we identify genuine aptitude and talent in assurance candidates, regardless of experience level? BJSS no longer exists, so there is no more need for secrecy.

And, in 2026, how capable are LLMs of emulating that talent?

For several years, I was a frequent interviewer for BJSS' assurance capability (testing and more) in our south-west region - something I really enjoyed. I helped to shape a set of exercises that proved very effective at identifying raw ability and mindset, for any experience level.

The soft skills are pretty obvious after a couple of interviews. The tricky part is finding evidence of high ability, of the sort that would make someone great at leading the wide range of 'assurance' activities on modern projects. We wanted to see things like critical thinking, analytical thinking, an instinctive eye for quality and an affinity for code. The expectations were high.

Assurance specialists needed to go beyond traditional testing and be advocates for product quality and risk reduction across the whole SDLC. Yes, they would largely own the technical side of QA, like automation and performance testing, but they would do a lot more besides. They might scrutinise UI designs. They might strengthen the guardrails built into our processes and pipelines. They might analyse user behaviour in prod. They might coach other engineers to become better testers. They might grow into more cross-functional engineers themselves and many did - why not? Plus, they might interview our future joiners and the high standards must be maintained.

Experience alone is not necessarily enough to infer ability - we saw plenty of 'senior' candidates who could talk the talk, but could not walk the walk. We saw very junior candidates who didn't know the industry jargon (yet), but were clearly very able, with huge potential.

We evolved three simple exercises, each taking from 5 to 15 minutes, depending on how deeply the candidate would dive into them. To be successful, we would expect an experienced candidate to have plenty to say on all of them. A junior candidate would probably only need a strong performance on two out of the three, as long as their potential was clear. The exercises were:

Test approach - analytical thinking, experiment design, test mindset.
Design scrutiny - eye for quality, user empathy, critical thinking.
Code review - more critical thinking, more 'test mindset', affinity for code, logic.

The effectiveness of these exercises lies in their simplicity and in their open-endedness. They might look easy on the surface, compared to the hard logic puzzles that some companies use. But, these exercises can initiate deep, insightful conversations with real-world relevance, which allow a strong candidate to show how their mind works.

One could easily make small changes to the exercises to keep them fresh and effective, even if candidates have previously seen the exact examples listed here.

The sample answers here are thorough, but don't represent a realistic expectation. They list many things that candidates might say, but we wouldn't expect all candidates to give such exhaustive responses, under pressure, in the short time available for the interview. As long as they were making good progress on each exercise, we would have plenty of evidence of their ability and would get into some interesting conversations, which was the point.

We stopped using a traditional code test. More on that below!

Abandoning the traditional code test

A note on CV reviews

The 'test approach' exercise.

1) Test approach

This looks like a simple exercise that should be a walk in the park for testers, but it does catch people out. It was amazing how it revealed gaps in some experienced candidates' capabilities. Strong candidates, however, have a lot of freedom here to show us how they think. Even if they are completely new to testing, a bit of logic and a bit of imagination can still produce a strong answer.

The exercise

The candidate is told the following:

There is a piece of functionality with two inputs, 'A' and 'B', and an output 'C'. A and B can each be a number or text. Here are the requirements:

If A and B are numbers, then C is the sum of A and B.
If A or B is text, then C is a concatentation of A and B.

How would you approach testing that?

The answer

They might note that the requirements are vague - good. Bonus points for design scrutiny. It doesn't stop us though, it just means that there is some exploratory, investigative work here to find out how the functionality really behaves. Maybe we've inherited a legacy system full of mysteries. Sometimes, we need to do detective work.

Main permutations

A good candidate would usually start with a systematic, all-combinations approach to the main happy-path permutations:

| A | B |

| Num | Num |

| Num | Str |

| Str | Num |

| Str | Str |

f(A,B) -> C

(1,2) -> 3; (1,"b") -> "1b"; ("a",2) -> "a2"; ("a","b") -> "ab"

A great start, implying a logical, systematic, testing mindset. They might have included 'empty' as one of the options for A and B, giving 3^2 = 9 permutations. They would probably demonstrate that they could work out the expected results for simple scenarios, as above, but that's not really the interesting part...

Exploratory

Next, they should go on to consider the wide range of possible forms that a number or a piece of text could take, beyond integers and short a-z strings:

It's a great sign if someone thinks about the limits and boundaries that might exist. Data types have maximum sizes - what are they? What does a boundary test look like? Maybe there will be validation rules we can expose and test, or maybe missing validation rules will be need to be flagged.

It would be interesting if they touched on all-combinations or all-pairs testing, noting that it would be unfeasible and probably unnecessary to explore every possible permutation, now that we have lots of possibilities for A and B. Seeing the code might clarify whether there is risk there - are A and B handled identically, or differently in some way? Are we using established, native methods (String.concat()) that are low-risk and don't need thorough testing, or something more custom and unproven?

Ideally, a candidate wouldn't stop there, but they would keep coming up with creative ideas about how to investigate and maybe break it.

Security and more

We really liked to hear them mention code injection and we would eventually bring up 'security' ourselves if they didn't. SQL injection is something that an experienced person should have heard of. They might mention other types of code injection too, such as malicious use of keywords or characters that might break JSON, XML or scripts if not handled properly: ", ', ; , /> , }, etc.

They might also bring up performance, permissions, throttling, anything! We would end up with a list of all their ideas - the longer the list the better.

Deeper conversation

There are lots of interesting directions the conversation could take after that: TDD, BDD, security, requirement definition. If they are experienced, one option is to chat about which tests they would want to automate at what level. If it were me, I'd say something like:

'We won't need to automate every exploratory test that we've imagined in this exercise - some of those will have served their purpose once we've resolved our uncertainty about the functionality and settled on desired behaviour, like boundaries and validation. We only need to automate the tests that we'll probably run in our pipeline for the foreseeable future, mainly for regression protection. What we want is the necessary and sufficient set of tests to demonstrate the expected behaviour and prevent us accidentally diverging from it in future.'

'We should have low-level unit tests that cover the main permutations (str/num combinations), the boundaries (char sets, str lengths, num ranges), the validation rules and any security protections (code injection prevention). Those might be sufficient, or we might need to unit-test some additional edge cases, if we have additional risks for any reason.'

'At a higher level (API/UI, depending on context), we'll have a much leaner set of automated tests for key user journeys that involve this functionality, but we won't need to retest every little detail that is already covered in the unit tests. The higher-level tests will mainly prove that the functionality is properly integrated into the wider product, throughout the stack and across user journeys.'

'E.g. Perhaps one high-level test will be enough to prove that a validation message appears on screen when an input is unacceptable, while many low-level unit tests will prove that every validation rule exists and works. The next step in the same high-level test will show that a valid input is accepted, the output has its effect and the user can proceed to whatever comes next. Many low-level tests will prove that the function also works for the range of possible inputs, noting boundaries.'

'This provides the regression coverage we need in an efficient way, because lower-level tests are quicker/cheaper. Other valid approaches and test levels may exist though, depending on the project. '

The 'design scrutiny' exercise, showing a questionable UI design.

2) Design scrutiny

This was my favourite exercise. It might look simple and even a bit silly, but it proved to be very revealing. Here, we're looking for naturally critical thinkers and user-empathisers who know 'bad' when they see it and instinctively want to question things and suggest improvements, even subtle ones.

Professional experience might help a tiny bit, but it's a fairly level playing field - we all have experience of being software users outside of work and we've all seen good and bad. The question is who notices and who cares? Surprisingly, some don't!

Apart from the obvious importance of UX to users, it's also something that can disproportionately influence our clients' opinions of us, because it's the tip of the software iceberg that they can actually see. It doesn't matter how brilliant the back end is, if the client has a bad experience with the front end. It's definitely something an assurance specialist should think about.

The exercise

The candidate is shown the (bad) UI design. We ask:

'What do you think about this?' (Deliberately open-ended. Hopefully that's enough to get them going.)

We might follow up, if needed, with:

'What would you do if a client/designer/BA provided this as a design?'

'What would you do if you joined an existing project and testing this was your first task? (The code has already been written.)'

The answer

Some bad answers would be:

'That's the design, so my job is to check the product matches it.'
'I'll just test whether I can log in with a valid password.'
'I can see it's a login screen and it seems OK'.

Those answers might tell us the candidate has deeply ingrained habits that aren't at all what we're looking for. Maybe they think product quality should be somebody else's concern and it's not their place to question things. Or maybe they mean well, but in practice they are just not a very observant, critical thinker. We want people with both the ability and the attitude to be constructive critics.

Good candidates will have plenty to say about it, without much prompting, and might like to draw their improvements on the (virtual) whiteboard.

Some things jump out straight away: the title is mad, 'login' is ambiguous and we seem to be missing a button. We'll keep picking holes in it and by the end we'll hope to have touched on various topics, not limited to accessibility, mobile responsiveness, and the benefits of familiarity with respect to modern, mainstream UX.

Here are some more thoughts, starting from the top...

What is this design style?

This looks like a basic wireframe, is that right? If so, fine, but we might need a more refined design next.
If this is the finished design, why is it so plain and colourless? Maybe there are reasons, but colours can be helpful and this looks strange, so let's check that.
Do we have a design system that this is meant to follow? If not, could we consider taking heavy inspiration from a mainstream one? Let's not reinvent the wheel (badly).

Banner and branding

Is this a whole page? If so, where's the logo or other branding? How does the user know what they are logging in to? Let's add something suitable.
If it's not the whole page, but some sort of pop-up modal, can we see it in the wider context of its parent page?
It's helpful to have an informative or instructive title, such as 'Sign in to your account', to tell users exactly what action they can achieve on a page. (Helpful for screen-reader software, plus users who are low on IT literacy and confidence.)
Websites and apps often have a menu somewhere. Do we not need one?

We need to talk about 'GOOD MORNING!'

This is a very strange page title. It feels amateur.
If we really want to make our software friendly and warm with a personable greeting, then we can have that conversation, but shouting at the user in all caps is a bizarre way to do it. What if we make it smaller and not all caps? We know that standard 'sentence case' is more readable.
What if we just discard it? Minimalism can have value too. Minimising text to only the bare essentials can be very helpful for some users.
What if it's not the morning? What would we show at other times of day? What about a timeless greeting (if we must), like 'Welcome back'?

The credential fields

The proportions of the fields are strange - will long values fit? What do those fields look like with realistic values in them? Might we need to make those fields longer?
With good reason, modern UIs put field labels above fields in a single column, not next to them in a second column. This helps to make the layout neater and more responsive to different screen sizes. It allows more space for longer input fields, even on narrow screens.
'LOGIN' as a field name is ambiguous. Is it email address, phone number, username, or something else? We should use an explicit term like that.
As with the title, let's use sentence case, as it's more readable than caps.
Would it be relevant to show an example of what the user identifier field ('login') should contain? Like an arbitrary email address or username? Some websites do this, so perhaps it's helpful.
What appears in the password field when we type? Is the password hidden? What about a show/hide password option?
What about a 'remember me' option?

The buttons

Obviously, we need a 'Log in'/'Sign in' button. Has it just been forgotten, or is something else going on? Can we be more conventional?
N.B. it should be 'Log in' on the button (the verb 'to log in') or 'Sign in'. Sometimes we might use 'login' as an adjective or a noun, which is ok, we just need to use it correctly or it looks amateur.
What on earth does the 'cancel' button do? It isn't particularly intuitive or familiar in this context. If we need an option to get back to a different page without logging in, let's make it more explicit: 'Go back to <some page>'.

Unhappy/alternative paths

What if the credentials are invalid? What validation messages are we showing and where? Let's think about that and try to follow familiar patterns.
What if the user has forgotten their credentials? It's common to find a link for that, close to the credential fields. Again, let's follow a familiar pattern unless we have reason not to.
What about a link to register if you are new user?

Modern patterns

'One thing per page.' Login flows are sometimes split into steps, so the user only has to think about one thing at a time: username first, then some subsequent steps, which might include password entry, code entry, or maybe something else...
MFA? It's common to get a login link via email or to use an authenticator app.

General accessibility

We need to think about how screen-reader software handles this page. On-screen text should explain what's going on concisely. Fields should have whatever descriptive tags they need - we'll test this later.
We're going to add some validation messages and the 'forgotten password' and 'register' links - we need to be careful that any text we add is in clear, plain English and is not excessive.
Colours and font sizes are among the accessibility considerations we'll come back to when we have a more refined design.

Security

We're getting away from UX here, but are we sending those credentials to the back end in a safe, encrypted way?
As mentioned, MFA?

Dealing with this scenario in a project team context

There's no specific right answer, but it's interesting to hear how candidates might take their questions and suggestions to colleagues. Do you go straight to your peer, or your manager or wait for a retro in two weeks' time? It might be a good idea to ask a friendly colleague where the design originated and get a feel for any political issues. Maybe a business stakeholder considers themselves a bit of a designer, but clearly isn't great at it - a tricky situation. (I've been there.) Or, maybe the team has been struggling without any real UI design input and would love your vocal guidance, starting immediately. Maybe you can help, or maybe you want to ask around for a UX expert somewhere in the business.

What if the code has already been written? You need to form good relationships with developers, so tread carefully. Maybe they'll say it's just a prototype, or go-live is a way off so they're not worried, but it should still concern you that something pretty simple isn't being done well first time. With a bit of luck, the devs will be grateful for your input and you'll work together closely on improvements. If there's some resistance, you might need a more gradual approach while you build trust and credibility. What if you suggest some user research and allow its findings to educate the team and build consensus for improvement?

3) Code review

This is a terrible piece of UI test automation code, written in Java (probably involving Selenium).

If it looks so bad you couldn't make it up, that's because it is indeed not made up. It is a real thing that someone in the history of BJSS was shocked and amused to encounter (before completely rewriting it, presumably).

There are lots of little problems with it, but the prize is being able to infer what the whole method is (not) achieving and how it is fundamentally flawed in multiple ways.

The exercise

A junior candidate with little/no programming experience would be talked through the code so they roughly understood what's going on, but we'd let them work a bit out for themselves, if they could. Hopefully they would be able to spot a few problems, or at least demonstrate understanding of problems that we drew their attention to.

A candidate claiming any test automation experience would be mostly on their own! They would only need to be asked 'what can you tell us about this?' If they needed steering, we might ask them to walk us through their understanding of each line, then ask them if they have any review comments.

The answer

It shouldn't matter if someone has prior experience in Java - the code is pretty self-explanatory and key concepts are not language-specific. If someone has a specific knowledge gap, like what the webdriver is, we'd be happy to explain that. Knowledge isn't really under examination here, but ability (and a little bit of test wisdom).

Fundamental flaws

This is a high-level summary of some major problems that candidates will hopefully spot. There is more to say on both of these points.

The intention appears to be to retry a procedure once, if something goes wrong. But the two attempts do not involve the same steps - a flaw. If we need a retry, it should probably be consistent.
It looks likely that the exception-retry mechanism is pointless anyway. We probably just need to find whatever sequence of events does work and run that once.
- If the recurring problem comes from an important step that is repeated in both attempts, then it could still fail in attempt 2. In this case, the retry isn't protecting us, so it's not the answer. We need to get to the root and fix it.
- Or, if the recurring problem is only in attempt 1, but attempt 2 is somehow reliable, then let's keep the steps from attempt 2, cherry-pick any additional necessary steps from attempt 1 and merge them into one successful sequence.

Entry-level candidate

We would help them understand the context and what each line does. Hopefully they would ingest that without a problem, so they would then be able to ask some questions, like:

Why are there three or four different kinds of 'wait'? And what does the mysterious one just called Wait() do?
Why is SendKeyEnter() commented out? It's misleading that it then prints a message saying it did press enter.
And why is the test hitting the Enter key at all?
What's a RuntimeException?

They might have other questions too, which is great. Curiosity is very positive and shows they haven't got stuck at 'I can't read code'.

Experienced candidate

Elaborating on the fundamental flaws...

The problematic waits: The bad waits might be the source of the problem in the first place. If we need to wait for something to load, we should use an explicit wait for something to meet some condition (usually page elements loading) before proceeding. There is one good, explicit wait, although the timeout might be too short. There are three bad, implicit (static) waits for fixed amounts of time. These are flawed, because something might take longer to load than that arbitrary amount of time. Even if the static wait is longer than the loading time, these waits are still bad, because our tests are waiting around for no reason, running more slowly than necessary, literally delaying our project.
Retries: If we have a good reason to allow retries, we can use a better mechanism (maybe a loop), so we have more control over the number of retries and the exit from the retry mechanism. A try-catch might be involved somewhere in the solution, but it isn't really a solution itself.
DRY: The two blocks of the try-catch are repeating lines of code unnecessarily (and not actually repeating accurately, for some reason). Code reuse (DRY) would be better, if something genuinely needs to be repeated.
Broad exception catching: The catch of a RuntimeException is worrying. It's too generic and will catch a lot of different things (although actually it won't catch every kind of throwable). When an exception does get caught, we have no idea what it is or which line threw it. If a catch is needed at all, we probably need to catch some specific exception from a specific line in the method and do something more specific to handle it. Hopefully we don't need the catch at all - we can fix our underlying logic so we don't routinely run into this problem.
Exceptions when page elements don't exist (yet): This is a common problem and likely to be related to the try-catch madness. There are simple things we can do - essentially well-placed explicit waits for the right elements to load in the right order, with a long enough timeout to handle foreseeable performance variations. The problem might even be further back in the test, not in this method. Maybe we're not even on the right page. After any action that changes what's displayed on screen, wait for the new things to load and reach the expected state. Before any action that requires an element to be in a certain state, check it. Do these things and the test will either work, or valid exceptions will surface and fail the test in a way that shows you exactly where the problem is.
Breaking the test state: With the current retry approach, you could get a situation where the text is entered into the field twice, depending on where the exception was thrown. That means we don't know for sure what we've entered into a field, so we can't make reliable test assertions later. We are probably ditching the multi-attempt approach anyway, but if we keep it for some reason, we need to make sure we avoid this problem.

Below are some more general complaints, which may well surface before the candidate has got their head around the entire method and identified the fundamental flaws.

The method name should be sendKeys() not SendKeys(). Same problem with the SendKeyEnter() method that it calls.
Questionable style in a few places: brackets, line wrapping, etc.
Where is this mysterious locator defined? Is it at the class level? Which element is it for? In the common 'page object model', a class represents a page with lots of elements, not just one.
The text printed to the console is a bit messy. If we need it at all, we might want clearer, more relevant information, like what field we're entering text into.
Printing "Click Enter" is strange. It doesn't appear to be mouse click. It appears to be a keystroke.
Strange to put "\n" at the start of a printed string, because all the prints use println(), which adds new lines anyway. You might need to add your own new line characters if using print().
Basic problems with the exception catch - we never see what the exception was or where it came from, plus it might just happen again in attempt 2.
Implicit (fixed time period) waits are usually bad.
The mysterious Wait() definitely raises questions. Where is it coming from and what on earth is it waiting for? Another naming style problem.
Etc.

Bonus point

There was a typo in the filename (something like 'exampl.java'), which was visible to the interviewee while the code was on screen. They usually wouldn't notice this little easter egg, but occasionally someone might!

Can AI do these?

When these exercises were born, generative AI was not. Just out of interest, I tested them with ChatGPT-5.6 Sol on 'high' effort. This might not age well.

1) Test approach: 🟡 7/10 on first attempt, 🟢 9/10 when asked to go further.

It nailed the systematic approach to the key input permutations and the subsequent variations, earning a solid 7.

Only when pressed to go a bit further did it get more imaginative, mentioning things like performance and security. Compared to a good human response, it was a bit too keen to accept the premise as legitimate, rather than to question it and point out the uncertainty around it, but it really wasn't far off a perfect answer.

2) Design scrutiny: 🔴 3/10 on first attempt, 🟢 9/10 once ordered to scrutinise.

Starting as we would with a human, asking only 'what do you think of this?', the bot was useless - it complimented the poor design first, then gently pointed out a few of the obvious, screaming issues.

The next prompt was 'Assume we're aiming for a modern UI to please our clients. Your job is to scrutinise this design. Go for it.'

It did follow its new orders and ripped the design to shreds, reporting many of the issues very well. However, it still stopped short of fundamentally questioning what the design is for, where it originated and how it fits into a wider app. It proposed plenty of changes and additions, but nothing radical - it didn't mention the modern pattern of one thing per page, or MFA.

3) Code review: 10/10 ⭐

Perfect.

What does this tell us?

Unsurprisingly, it's amazingly good with code and scored top marks on the code review exercise, even though it was only asked the very open-ended 'what can you tell me about this?' We know that code review is a common use for these models, so we shouldn't be surprised that their default behaviour is optimised for it, without needing a steer.

The other two exercises were more interesting, as it did not do so well with the initial open-ended prompts. These exercises are a bit more abstract. It definitely revealed itself to be a robot, obeying enthusiastically but not really thinking outside the box. It did much better when explicitly ordered to go further, but didn't score 10s for me. Where it fell short of a brilliant human was in displaying critical thinking, judgement and creativity, without being asked.

I'm sure there will be good reasons for that - tokens aren't free and there's no point in the model working much harder than it has been asked to. It would be annoying if it constantly questioned everything and overstepped in its initiative. I may be wrong, but I don't think this is one of those temporary AI weaknesses that will be irrelevant in five minutes, because it isn't really a weakness...

With only a little extra prompting, responses were almost perfect.

I believe the model is capable of scoring a 10 first time, if prompted in the right way. It needs more context about the behaviour I want to see, but that means I'm not fully delegating 'intelligence' in the way we expect to with human colleagues. It's a reminder that if we're designing agents to play roles on our projects, we must be thorough in their instructions, or they might fall well short of human performance in some tasks, despite being theoretically capable of matching or exceeding it. I do want you to question tasks, criticise my work brutally, suggest better alternatives, etc.

These exercises are arbitrary and narrow in scope.
- A good human does not need much instruction to tackle all of them well.
- A good LLM needs some additional instruction to tackle all of them well.
Real-world problems are much bigger and more complicated.
- We can extrapolate that the good human's abilities and attitude will allow them to do the job well.
- We can extrapolate that the LLM would need much more context and much more careful instruction in order to do well.

So, the exercises still have relevance for us humans, as we need the kinds of people who exhibit the thinking that these exercises aim to reveal, but without being told. We need people who can impart their wisdom into an agent's instructions and then evaluate whether the agent does their tasks to their high standard. Plus, there are reasons why teams might actively choose to keep some amount of work in human hands, like rising token costs and the risk of losing institutional wisdom through AI overreliance. Even in highly agentic setups, there's probably an element of human oversight and accountability somewhere in the loop and that human needs very good critical thinking skills, as AI mistakes are often subtle ones, surrounded by a very convincing commentary. (It was tricky finding what it had missed in its very confident, mostly excellent responses to these exercises, but the gaps were indeed there.)

Another thing to bear in mind is that, in real life, the most valuable problems we solve are not usually arbitrary exercises like these. Yes, we're using the kinds of cognitive abilities that these exercises explore and that AI can easily replicate with a narrow scope, but we're often applying our abilities to bigger, real-world problems and taking into account vast amounts of context in forming our judgements. We deal with nuanced things like trade-offs, prioritisations, risks, strategies and tactics. Our context might include unquantifiable knowledge about ourselves, our teammates, our stakeholders, our users, our product, our budget, our contract, something that happened last year, something that might happen next week, the weather, the world cup and something we just heard on a tech podcast. That mega-context judgement is, for now, something that AI struggles to replicate, because it's unfeasible for any of us (let alone all of us at the same time) to give such vast context to the models and even if we tried (some do!), it would probably cost us a fortune. This, however, is a point that might be out of date in five minutes.

Abandoning the traditional code test

We used to ask anyone who claimed automated testing experience to complete a code test, but dropping it was a good move, partly made possible by the interview exercises.

The code test was something along the lines of: here's a basic, incomplete test framework, get it working and add some test coverage for a specific website.

A good response would be a working test suite that was easy for someone to download and run straight away, with little messing about. It would have enough well-designed tests to demonstrate that the author knew what they were doing, but they didn't need to go overboard. Well commented test code is obviously easier to review and was appreciated.

There were a few problems...

It was often very tedious and time-consuming for whoever had to review it. (E.g. It might not be straightforward to run.)
A lot of what we received was not very good, so it wasted time for us and them.
Plagiarism was a risk, as there is plenty of code to copy from the internet. (Following examples and adapting them is ok - that's realistic work. But, the exercise loses value if almost-ready-made solutions are publicly available, such as from previous candidates.)
Good candidates needed to be hired fast, before someone else found them. The code test is a delay.
It is reasonable to imagine we might have screened out some good people by accident. If someone is committing all their mental energy to their existing job in working hours, then living a busy and fulfilling private life on evenings and weekends, they might not be able to spend two days writing pointless code for us, unpaid. If they are good, they will probably get an offer elsewhere and we will miss out.
We could generally reach a confident 'yes' or 'no' from all the other aspects of the recruitment process anyway, including the interview exercises described on this page.
In the unlikely event that someone remained a 'strong maybe' at the end of the multi-stage recruitment process, with our only doubt being their test automation ability, then the code test could still be an option. It wouldn't be needed often.
The probabtion period is the real test anyway...

So, we gladly took the code test out of the default recruitment process, retaining the idea only as an optional add-on, if it were ever needed. For the reasons above, it was absolutely fine.

What about generative AI?

At the time, we didn't know that powerful generative AI was right around the corner. It's interesting to consider how AI might have changed things...

On the one hand, using generative AI to assist in writing our test code is perfectly valid and commonplace - it's what people do on real projects now. There's still some degree of skill and human judgement involved when we do AI-assisted or agentic development, so maybe it's a worthwhile assessment to see what people can achieve with the contemporary toolset.

On the other hand, we might just receive endless, almost identical test suites, which don't tell us much. Instead of candidates directly copying from the internet, AI tools might just be doing it indirectly on their behalf (but more neatly). Instead of being frustrated by a large volume of bad code projects to review, we'd be frustrated by a large volume of projects that are just fine, all very similar, but not outstanding. Essentially, we'd be back in the same situation we were before, with a lot to review and reject.

Furthermore, AI makes it easy for people to generate much, much more code very quickly. The time cost of a review might become absurd, unless... more AI. It might be valid to use AI to do the heavy lifting in the review process, identifying standout projects in a sea of mostly similar, mediocre ones. Or, it might just be a big waste of tokens, similar to how the manual review was often a waste of human time.

Perhaps the answer is that AI makes code tests feasible, but they'll only be valuable if we change them a bit. They could become an interview exercise, like the ones we used to do. I quite like the idea of setting the candidate a quick challenge, seeing if they can wield AI tools effectively, then discussing their product with them immediately. Have they produced something genuinely good, and can they explain what's good about it? Or have they produced something generic that they can't really explain?

They could be limited to a short time window (30 minutes?), sandwiched between some time with an interviewer. The 'coding knowledge' aspect might be less than before, which is ok, because the main purpose of the code test was always to assess the candidate's end product - a thoughtfully-designed test set - not to test their syntax recall in a single programming language. Anyway, Google searches and clever IDEs have been helping us with the syntax for many years.

A note on CV reviews

It's hard to really understand someone's aptitude from their CV, but if the volume of candidates is high then, realistically, we can't interview everyone. We have no choice but to do some filtering at this stage, so we need to think about how we interpret CVs well. LLMs might lighten the load, but they still need us to define what we're looking for.

Like with the interview exercises, I would be looking for evidence of abilities and mindset, over specific existing knowledge. Existing knowledge and skills are still relevant on the CV, but mainly because they show someone's learning trajectory. A keen, fast learner, even with current knowledge gaps, has high potential and high value. An experienced person with some relevant knowledge, but a flat trajectory, brings risk - will they be suited to project work? Every project is different and the industry is always changing, so we need to be adaptable and versatile. In some organisations, you might be able to work on the same product with the same tech stack for a decade or more, but consultancy is very different.

So, what kind of evidence can we look for on a CV?

General

Good writing is always a good sign. Writing is thinking on paper and the CV is the first opportunity to show people how you think. Bad writing is a bad sign and obvious AI-generated writing gives us nothing to go on.
Good formatting and layout is telling. A neat CV tells us that someone cares about presentation and detail.
A CV should offer us interesting talking points to follow up in the interview. It's interesting if a candidate is aware of that and has given us some interesting bait - a cutting edge tool, a novel way of working, an unusual project. If the CV makes us excited to meet the candidate and find out more, then we're off to a great start.

Entry-level candidates

This is difficult, because there aren't many data points on their CV to indicate their trajectory. I'd be more likely to give an entry-level candidate the benefit of the doubt and interview them, because, if they do have high potential, they are a very high-value hire. An influx of junior talent is really important.
One potential source of evidence is education. We didn't strictly require a university degree, but obviously it is relevant evidence if it's there.
The exact degree course doesn't matter much to me. STEM is promising, but most courses will have developed transferable skills in the student - analysis, communication, creativity, critical thinking, writing, presentation and, of course, learning, which is a skill in itself.
The academic trajectory can be evidence too - A level results, which University they were admitted to (requirements vary), degree grade, postgraduate study. Just getting into a university with high entry requirements might be a good sign.
But, we do need to be forgiving. A-level and university students are very young and academia is a strange world that doesn't suit everyone. There are all sorts of reasons why academic grades might not reflect a person's actual abilities. Personally, I found that dry lectures and arbitrary exams went against the grain, but motivation came very naturally in stimulating, real-world project work, so I try to bear that in mind. If someone got into a good university, but left early because it didn't excite them, that's still a positive in my opinion. I know people who did that and I respect it. It might even show an impressive level of maturity and self-awareness.
If someone has not been to university, that's no problem at all. Still, we do need to see some evidence of their high ability and we should be open-minded about the form that evidence takes. It might be implied by a combination of school grades, other qualifications, personal projects or quick progression in any work they've already done.

Experienced candidates

Trajectory is still important, as the need for learning and adapting will never stop. That could be implied by fast-paced growth in knowledge and skills over time.
Evidence of versatility is good to see - varied contributions at work show someone is keen to tackle whatever their project throws at them, even if it's slightly outside their established specialism. In small, modern teams, we need to wear many hats. Plus, every project is different and the industry is always changing.
Growing responsibility over time is very positive - the fact that their colleagues have trusted them to take increasing responsibility is evidence of their past successes. Being an assurance specialist often means being in a leadership position on a team.
Achievements should be described by the real-world value they brought, as this shows that the candidate works with purpose, rather than just seeing work as a technical puzzle. For example, if a candidate writes that they developed a test suite, that's not very insightful for us. It would be better to read that they benefited users and the business, which was a result of quality and efficiency improvements, which were the deliberate result of their work. Look for the impact of product features that they helped to deliver. Look for why the hard problem they solved was worth solving in the first place.
Yes, years of 'experience' can have great value, but we need to interpret what that value is for this particular person. Everyone will have x years of experience at some point, but not everyone is great at every job. It's helpful if candidates write about the value of their experience explicitly, but our inference may play a part too. What wisdom did they gain from a certain project or job? What did they learn from successes and failures? Has their experience made them a deep specialist, or a versatile generalist? What strengths do they offer now, that aren't offered by more junior applicants? In what way is their experience any more valuable than that of their peers?

Advice for applicants - things to avoid:

'Here's a very long list of technical tools I've seen over 25 years.' Yes, but what did you do with them? In your listed experience, sprinkle in some tooling highlights, as evidence that you are an adaptable learner, but remember that what you've achieved with the tools is more important. Anyone can have 'exposure' to tooling. If you are a highly experienced specialist in specific tooling, that's great, highlight that in your core skills, but don't let it get diluted by a list of a dozen random languages you dabbled with in the distant past.
- The exception might be for junior candidates. If you don't have much to put in your CV, but you can say 'look, I've used a very wide range of tech in only 2 years', then that's quite good. You will need to cut it down in future though.
'I participated in planning and retros. I executed tests. I updated Jira.' I'm afraid that all the other applicants did those things too, so it doesn't tell us anything at all. Focus on achievements, impacts and learnings. If you want to show that you understand the value of certain ways of working (e.g. a certain agile methodology), write about why it was good and how you influenced it, don't just tell us that you participated.
'I have a testing qualification.' This might be controversial, but many of us at BJSS were not interested and would completely disregard a testing qualification. We didn't really see any correlation between qualifications and actual competence. If someone is naturally good at the job, it's not because of their qualification. If someone is not naturally good at the job, a short course and a certificate isn't going to be a game-changer. However, there's no harm in someone doing a course if they want to and we obviously wouldn't mark someone down for putting it on their CV. It's just not realistically going to make the difference between being hired or rejected - everything else is much more important. I wouldn't want to be hired by someone who thought otherwise.
Too much length. Admittedly, I struggle with this myself, because I feel I want to tell the whole story accurately and not leave gaps open to guesswork. However, we have to try to be strict with ourselves. You can write about your experience in more detail on LinkedIn and elaborate in the interview. 1 page is good for a junior, 2 pages are good for most people and 3 is probably the sensible upper limit, but only if you have a lot of relevant jobs or projects to list. 5 pages is too many. People just won't read a long CV in the early stages of recruitment and it can even be off-putting when a candidate hasn't made an effort to make the reviewer's life easier.
Too much detail from the distant past. Past education and experience gets less relevant and less impressive over time, so cut it down. It shows good awareness of the industry and your place in it if you cut out things that are now unrelated. Ditch the GCSE results and DofE awards as soon as you're in work, then cut out A-levels at some point in your early career. Keep a university degree on there, but reduce the description about it once your work experience is doing the heavy lifting. Similarly with past work, the descriptions should be more condensed as we look further back through the CV. Once you're on a career path, remove references to casual side-jobs you had as a teenager or a student.
- You can break these rules if you want to highlight something that remains highly relevant to your current work, or was highly influential on your career, or is simply a fascinating talking point that will help you stand out. If you have an A-level in a Finnish, or you used to work as a safari guide, or you're grade 8 on the bagpipes, those could be fun things to leave in, so that interviewers want to meet you and ask for the story.

Page updated

Google Sites

Report abuse