Questions about AI in Software Testing

Yes, but even this process needs to be tested. If our requirements are detailed enough and we have explained the problem well enough, then an AI can perform the tests without any human interaction. However, this has never been the case in the projects we have worked with. We imply a lot of things, and there are a lot of requirements that can be interpreted in two very different ways.

Yes! If we receive a test plan from an AI, we need to see what is missing or needs to be removed. We still need to proofread the results or test to see that the scripts work as intended. Sometimes the work to explain the problem will take us too longtime than we gain from asking an AI.

It’s important to both understand code and also to understand testing on both fundamental levels and advanced levels. AI can only releive us of repetitive work but they still make mistakes, it’s up to us to be the experts that find those mistakes and correct them.

With the recent version of chatGPT you can ask for references or multiple views on a subject. “Give me 3 different ways how to write an automated test case add references”. The problem is that we become lazy when we get a new tool. How often do you look up references from things you find through Google? I am usually too lazy to do this.

Absolutely, it’s able to produce test cases across the entire test process – as long as there is clear requirements and documentation about the product. The AIs in current form can’t explore a software on their own and thereby understand it’s function and write tests.

Yes, this is a big risk. We need to be careful how we use LLMs. We already have this problem today: People copying things from the internet or from other sources without proofreading it. However, with an LLM it is much easier to get information or results that you need fast. Since the results you get are generally correct, people will be more unwilling to proofread the results.

You can use this tool for brainstorming test cases and improving areas like testing practices and processes, among others. It's powerful when you have in-depth product knowledge, like access to source code. It works well for various project types, including automation and documentation. You can collaborate to create test plans and prioritize cases. AI assists in generating content but doesn't replace human decision-making. It's adaptable to your organization's specific needs and structure. 

That is a big question. There's a risk that as testers, we might become complacent and rely too much on AI, neglecting our testing expertise. This risk emerges whenever we introduce automation into any part of the process. 

Moreover, we may fall into the trap of thinking that AI provides a limited set of responses, and we must choose from these. In reality, there are countless other relevant answers for specific situations. 

It's essential to question and challenge AI, encouraging it to think beyond the confines of its predefined box. This is where genuine engineering work and creativity come into play. Creativity plays a crucial role in expanding our horizons and generating diverse scenarios for various questions. 

In summary, AI has the potential to stimulate creativity and serve as a steppingstone to explore various avenues of thought, but it won’t replace the need of deep testing, designing or engineering skills. 

It kind of depends on what you use the test for and what kind of tests you run with it. When it comes to calculating or validating insurance policies or insurance claims, for example, you still have a set of rules like an algorithm or something that you need to follow, even if you were a human inspector.  

The decision making on the ethical part should not be left up to the AI. It's still us as humans who need to handle the ethical aspects. The AI should just be a tool for execution and automation. 

More like a user story or more like a detailed requirement. I would say that you could put in all of the requirements, the epics and the all the levels of requirements that you 

have and it doesn't matter if they're engineering requirements on electronics or PCBS, or if they're all super fluffy, epic user stories. 

You can actually use it to rewrite bad requirements into more structured requirements so that you're homogenizing all of your requirements to have the same format and language. 

And now also there are image recognition software within ChatGPT So you can put in design specifications. 

 If any of you use Figma, there's actually a plugin that you can use in Figma that translates your Figma design to executable frontend code.  

But in basic it's important to know what you're feeding into the chat bot, so I would not suggest taking your entire requirement specification and putting it into ChatGPT, because of the security concerns around that. Unless you're developing a pretty generic application and you're not worried about the requirement specifications leaking. If you're working on very sophisticated products with classified information or something like that, you should be very, very careful of you input to any LLM outside your organization.