Using variables and running tests with driver script causes fails

Question

I am trying (maybe foolishly?) to write a Groovy script to run a series of test cases in a test suite.

The script would/should be able to

1) get all the testnames to run in the test suite,

2) loop through execute each test in order found in suite

A- execute any SQL queries defined in testcase properties and create the dynamic property/value pairing at testcase level

3) loop through steps in each called test run while utilizing any defined variables(properties) referenced for input or assertions

4) produce informative log on each test/step so others know what was test

It is these last two part that have gotten me in a knot--I can not seem to get the test steps to execute and use the referenced variables (either as part of the request or part of the assertion) - for example I may run a SQL - it retrieves an OrderKey - which I write as a Property at the testcase level - then I try to sub in that value using the parameter in the API Request and subbing in ${#TestCase#OrderKey} - it works if I run it manually and it uses the value I manually put in the property but when I run the script to get it to execute this step it fails even though the SQL has run and defined the value in the test case property (OrderKey) successfully - it fails to make the request and throws an error. If I manually execute it again at this point with the retrieved value from the SQL - it works. The same goes for the Assertions - if I reference a property - it fails when run through the script (which I removed the variablized "submission" value and hardcoded it) - but if I click the Request to run manually with same setup - it works. Sooo...what's the deal?

the command I use to run the step is --> (beginning of the loop)

def Steps = it.getTestStepList()
Steps.each //loop thru each step in test case as found by previous command
{
tstep = tcase.getTestStepByName(it.Name);
status = tstep.run(testRunner,context);

and then the rest of the steps follow - does something need to be done to enforce the variablization in the script?

Big question number 2 - can Assertion scripting be done in a regular Groovy script or do you have to go to the Request and bring up the Script Assertion file and code it there? If it can be done directly in a standalone Groovy script - an example would be cool too! Everytime I try I get that the messageExchange object is invalid or unavailable...

Any help is appreciated - but keep in mind - I am a NEWBIE so please don't assume I know anything....THANKS!

tneuschwanger · Answer

hello bqualters&nbsp;
&nbsp;
That seems a little bit like re-inventing the wheel.&nbsp; From your description, it sounds a little complicated for just a little bit of information.&nbsp; Why not just run the suite as normal but include a few well placed log.info lines of groovy code into some groovy test steps for the test cases you would like information from?&nbsp; At the conclusion of the run, the script log will contain all the values from the test case that you would like to look at.&nbsp; If you run your project or suite or testcase from the testrunner.bat command line interface (smartbear has good information for that when googled... https://www.soapui.org/docs/test-automation/running-from-command-line/functional-tests/), it will have a log file created for posterity that anyone who cares about the test run can review.&nbsp; The log would have individual text files for full request/response data that shows what was executed also.
&nbsp;
Below is example of Groovy Script test step that does assertions in code (lines with "assert")...
log.info 'Test Step "' + testRunner.runContext.currentStep.name + '" start...';
log.info "";

//def rawResponse = context.expand( '${HTTP Request GET KeyConfig#RawResponse}' );           // for ReadyAPI
def myRequestStep = testRunner.testCase.getTestStepByName('HTTP Request GET KeyConfig');     
// for SoapUI

def rawResponse = new String(myRequestStep.testRequest.messageExchange.rawResponseData);     // for SoapUI
log.info "rawResponse=&gt;" + rawResponse.toString() + "&lt;";

tokenizedResponse = rawResponse.tokenize();
httpCode = tokenizedResponse[1];
httpMsg = tokenizedResponse[2];
//tokenizedResponse.each {
//   log.info "it=" + it;
//};
def actualResponseCodeStr = "${httpCode} ${httpMsg}";
def expectedResponseCodeStr = '200 OK';
assert expectedResponseCodeStr == actualResponseCodeStr, "Expected Response Code of '" + expectedResponseCodeStr + "', but actual Response Code is '" + actualResponseCodeStr + "'";

def response = context.expand( '${HTTP Request GET KeyConfig#Response}' );
log.info "response=&gt;" + response.toString() + "&lt;";

testRunner.testCase.testSteps['Properties'].setPropertyValue('apiResponse', response.toString());

log.info "";
log.info 'Test Step "' + testRunner.runContext.currentStep.name + '" done...';
&nbsp;
More Groovy code with an assertion and JsonSlurper use...&nbsp;
import groovy.json.JsonSlurper;

log.info 'Test Step "' + testRunner.runContext.currentStep.name + '" start...';
log.info "";

def jsonSlurper = new JsonSlurper();

def apiResponse = context.expand( '${Properties#apiResponse}' );  // log.info 'apiResponse=' + apiResponse;
def jsonResponseObjActual = jsonSlurper.parseText(apiResponse);   // log.info "jsonResponseObjActual=" + jsonResponseObjActual;

log.info "jsonResponseObjActual=" + jsonResponseObjActual;
log.info "element count=" + jsonResponseObjActual.size();

def expectedCount = 3;  // just sanity check since returned data will not contain service level specific data
assert expectedCount == jsonResponseObjActual.size(), "The number of elements returned was expected to be '${expectedCount}' but it was actually '${jsonResponseObjActual.size()}'";

jsonResponseObjActual.each {  name, value -&gt;
   log.info '   ' + name + ' ' + value;
//  expectedValue = databaseExp.("${name}");                             //       log.info  "expectedValue="  + expectedValue;
//  assert value == expectedValue, "The value of '${name}' was expected to be '${expectedValue}' but it was actually '${value}'";     
   
//  expectedValue = jsonResponseObjExpected.("${name}");
//  assert value == expectedValue, "The value of '${name}' was expected to be '${expectedValue}' but it was actually '${value}'";
};
 
log.info "";
log.info 'Test Step "' + testRunner.runContext.currentStep.name + '" done...';
&nbsp;
Regards,
Todd

bqualters · Answer

TNeuschwanger-Thanks for the response!&nbsp;At this point I have figured out how to do what I wanted in terms of being able to loop through all the tests in the suite one at a time, then through each test step in each case, run dynamic data retrieval SQL's and use the values for checkpoints, BUT I am still trying to figure some things out - especially the use of assertions if they fail in the groovy script (not in the embedded groovy script of a step mind you - just in the Groovy Script test that acts as my driver).&nbsp; If the assertion fails the script errors out - that doesn't seem to make sense as I thought it would simply cause the test to fail - so I built my own code to do the check at this point. The assert seems to have made itself useless because of this fail possibility.&nbsp;Also, using the messageExchange object seems to only be available if you are in a Script Assertion step within a test step - I don't know what I need to import in a regular Groovy script to have access to that so I worked my way around it.&nbsp; I get an error when I try to use messageExchange so...not sure what to do other than not use it here.&nbsp; I can use it in the embedded Assert Script but that defeats the purpose of having ONE SCRIPT for all test cases.The point of what I am trying to do is make it so that all the guys on my team, WHO HAVE NEVER DONE API testing or used this tool or written Groovy scripts, can create test cases simply through an excel spreadsheet.&nbsp; I will (eventually) read from an Excel file what the desired test is to create and dynamically create it - assertions and all, dynamically through the driver script - much like I currently do with our functional testing in UFT.&nbsp;&nbsp;I just started with SoapUI - in my free time for the last month or so - and found it cumbersome (the free version mind you as work is too cheap to pay for anything) to build a single test case manually - you have to get the wsdl, make all the header settings, set up properties, do data prep with the AUT to get to initial condition,&nbsp; then make your assertions for each step one by one within the tool, not to mention the ridiculously poor debugging tools and meaningless error messages that tell you the error is on line 12 when it's actually on line 378 etc.&nbsp; Also, am I missing capabilities like putting breakpoints and viewing runtime variable values while the script is in progress?&nbsp; If so, please advise as I am all ears and hoping it gets better.&nbsp; So anyway, I am writing a script that runs in whichever test suite you place it, and utilizes the various property levels to make as much of the testing dynamic with the focus being FUNCTIONAL testing.My big problem right now is figuring out how to make sure the test steps use the property values I put in the api variable (parameter) places - for example, I think I mentioned above, each of my tests is to be written with a SQL statement that finds a valid piece of data that meets needed initial conditions - writes the result to a test case property, and that test case property is referenced in the parameter space for the API call - when the script runs - it fails - don't know why as log doesn't tell me. BUT, after the run, since the data has been retrieved and written to the testcase property being referenced, if I click Run manually to run it again - it works and uses the value the SQL retrieved.&nbsp; So..what's the deal there?&nbsp;I will be the first to admit I am NOT a Java / Java Script / object oriented programming languages person.&nbsp; I am self taught at everything I do and as such - what I've learned MAY BE WRONG!&nbsp; So if you have some advice -I will take it.&nbsp; &nbsp;Also, in regards to the logs that SoapUi provides - they suck.&nbsp; Maybe I'm missing something but they simply tell you whether or not the test step passed or failed - but lack the details as to WHY. For example, I created 5 test cases - ran them at the Test Suite level and generated the report.&nbsp; The only info given is the basics and the response package BUT there was nothing about which assertion failed even for the assertions I created with the built-in tool.&nbsp; If I have 5 or 6 different checks I want to know WHICH one failed IN THE REPORT LOG so I don't have to go open up the test steps individually and see which are green and which are red...what am I missing?&nbsp; Again - I have written my own checkpoints now and don't need the built in ones unless I want to use them so I can turn the step green or red and make it look pretty that way. Seems to me, if you grab the response and convert it to a string you can easily do almost ALL the checks that are "built-in" with very little effort using the Groovy commands - ie contains, count etc.&nbsp; The manual creation part is very cumbersome and time consuming.Let me know what you think - especially on the parts where I have actual questions and not so much the complaints - I complain alot for a non-developer type person - I know that.&nbsp; But hey, with a lack of knowledge comes a willingness to blame others!

tneuschwanger · Answer

hello bqualters&nbsp;
&nbsp;
That is a lot to unpack for a question you need answered...&nbsp; 🙂
&nbsp;
"If the assertion fails the script errors out - that doesn't seem to make sense as I thought it would simply cause the test to fail"
If you would like the testcase to finish even though an assertion fails you can un-ticked the "Abort test if an error occurs" within the TestCase Options feature of every testcase.
&nbsp;
Also, in regards to the logs that SoapUi provides - they suck. Maybe I'm missing something but they simply tell you whether or not the test step passed or failed - but lack the details as to WHY
You get as much out of the log information as you&nbsp; put into them... My testcases consist almost entirely of Groovy test steps...&nbsp;

1) A Groovy test step that calls JDBC for dynamic data to be put into a Properties test step.
2) A Rest/Soap Request test step that uses data from the Properties test step (for Rest, I often use a Groovy test step that has a coded Request instead of using a SoapUI provided Rest test step just so I can add logging to it).
3) A Groovy test step that asserts the content of the Rest or Soap Response from the previous test step.&nbsp;

As you can see from the sample Groovy I submitted before, you can put as much log information as you need to help debug your scripts.&nbsp; As a testcase runs, I can view the "script log" tab at the bottom of SoapUI screen to see all of the log statement content so I know exactly where my issue might be when a failure occurs.&nbsp; I often include a lot of log.info code when developing a test and comment it out or delete it after I see what is happening. &nbsp; If there is a groovy failure, I can usually review the "error log" tab for specific language/programmatic failure reasons.&nbsp; That might be a communication gap here... I use the script log as my canonical report which is different than an actual report that you can generate in SoapUI user interface.
&nbsp;
I was drawn to SoapUI to begin with because of its ability to be flexible in meeting my needs.&nbsp; If it did not have a feature I needed, I could write my own in Groovy.&nbsp; I think I won't be of much help in your endeavor to create a monolithic script to do what you want.&nbsp; I AM a Java / Java Script / object oriented programming languages person and prefer my projects to be modular in nature instead of your desired "purpose of having ONE SCRIPT for all test cases."&nbsp; I can try to answer small specific questions though.
&nbsp;
Regards,
Todd
&nbsp;

bqualters · Answer

Still not sure why this is happening--&gt;I can not seem to get the test steps to execute using the referenced variables (either as part of the request or part of the assertion) - for example, I run a SQL - it retrieves an OrderKey - which I write as a Property at the testcase level - then I sub in that value using the parameter in the API Request of the test step and instead of using the hardcoded value I supply ${#TestCase#OrderKey} in the parameter value column - this works if I trigger the execution of the step manually as it uses the value currently stored in the test case Property called OrderKey&nbsp; BUT--&gt; when I run the script to get it to execute this step it fails even though the SQL has run and defined the value in the test case property (OrderKey) successfully - it fails to make the request and throws an error.&nbsp; If I manually execute it again at this point with the retrieved value from the SQL - it works.&nbsp; The same goes for the Assertions - if I reference a property - it fails when run through the script (which I removed the variablized "submission" value and&nbsp;hardcoded it) - but if I click the Request to run manually with same setup - it works. &nbsp;Sooo...what's the deal?

tneuschwanger · Answer

hello bqualters&nbsp;
&nbsp;
it is hard to visualize what you are doing... could you include some screen pictures of your test case and show the code/dialog where you are setting and getting properties?&nbsp; Are you using jdbc, property transfers, groovy script for the tasks you describe?
The community edition of SoapUI is a pretty mature product, so I don't think it could be a bug or many people would have encountered this issue.&nbsp; We will have to look to see what the test steps are in your test case.
&nbsp;
Regards,
Todd
&nbsp;
&nbsp;

Forum Discussion

Using variables and running tests with driver script causes fails

Recent Discussions

SoapUI 5.9.0 released

OS SoapUI: Can it connect and use Cosmos DB

"Clone interface" between projects not working

Related Content

TestExectute run configuring project variable

Azure DevOps Pipelines - Running “Headless” Tests

Running keywordtest using DDT Driver DriveMethod method