Hi,



Expiration of timer guarding 'execute' operation causes test case verdict to be set to value 'error'. Usually 'error' value of test case verdict is reserved for runtime errors in a test system due to execution of a test suite /overflow, unassigned memory, system limitation, etc./, which causes impossible to perform test scenario correctly.



I am a little bit convinced that expiration of timer guarding 'execute' operation should produce verdict "fail" cause of:

- no test system error happens while test case execution,

- 'execute' timer is yet another guard /supporting timers usually used inside testcase to guard testcase execution in a finite time/,



Receiving a verdict issued due to timeout usually:

- does not lead to asumption that test system have errors,

- requires verification af all guarding times inside test suite according proper IUT behaviour to diagnose if a testcase or an IUT is a reason of failure.



I do not expect more reasons for 'execute' timeout occurence like:

- implementation is correct, test case was not carefully protected inside by timers and system timer value is too short,

- implementation is incorrect, test case was not carefully protected inside by timers and system timer value is correct.



The second is an obvious reason for fail verdict, the first is an error in code design, but possibility of correct testcase execution was fulfilled in both cases.



BR,

Mariusz Kupiec

Hi Mariusz,

I would prefer setting the verdict to inconclusive as opposed to fail, since a guard timer
may be set to a value which is too short to allow the test to occur. If a test case is
interrupted by a timer outside of the test case execution environment, it is not the test
which failed, but the fact that not enough time was allocated it to execute to completion.

I would prefer a value of inconc. :-)

Cheers,

Claude.

---

Original Message

---

From: Mariusz Kupiec [This email address is being protected from spambots. You need JavaScript enabled to view it.]
Sent: 19 December 2002 10:40
To: This email address is being protected from spambots. You need JavaScript enabled to view it.
Subject: Error verdict in 'execute' operation due to timeout



Hi,



Expiration of timer guarding 'execute' operation causes test case verdict to be set to value 'error'. Usually 'error' value of test case verdict is reserved for runtime errors in a test system due to execution of a test suite /overflow, unassigned memory, system limitation, etc./, which causes impossible to perform test scenario correctly.



I am a little bit convinced that expiration of timer guarding 'execute' operation should produce verdict "fail" cause of:

- no test system error happens while test case execution,

- 'execute' timer is yet another guard /supporting timers usually used inside testcase to guard testcase execution in a finite time/,



Receiving a verdict issued due to timeout usually:

- does not lead to asumption that test system have errors,

- requires verification af all guarding times inside test suite according proper IUT behaviour to diagnose if a testcase or an IUT is a reason of failure.



I do not expect more reasons for 'execute' timeout occurence like:

- implementation is correct, test case was not carefully protected inside by timers and system timer value is too short,

- implementation is incorrect, test case was not carefully protected inside by timers and system timer value is correct.



The second is an obvious reason for fail verdict, the first is an error in code design, but possibility of correct testcase execution was fulfilled in both cases.



BR,

Mariusz Kupiec

---

Czatujesz? Zaloz konto pocztowe w domenie @CZATeria.pl
Rezerwacja dla zarejestrowanych nickow tylko do 20 grudnia!
>>> link.interia.pl/f16b0

Thanks Claude,

I got another opinion:

> Concerning your CR below. I have the opinion that ERROR is correct,
> because there is a problem with the test system: it does not
> react at all (or as expected). If you want to have a FAIL, then don't
> use the EXECUTE timeout, use the timer mechanism within the test case
> description. You think that FAIL is correct and
> Claude (see mailing list) thinks that INCONCLUSIVE is correct.

And I agree to ERROR verdict as in standard.
'Execution of test case with not enough time allocated' was misleading for me.
Maybe a small description would be included in 27.5 and 25.2.1 like below:

"If used, value of a system timer guarding 'execute' operation should be calculated carefully, inluding all timer margins and signals propagation, at least for a value, which allow proper testcase execution on a correct implementation for a worst /maximum allowed delays/ case. The main use of this timer is to provide mechanism of continuation of testcases execution in case of testcase design errors leading test system to fall into e.g. infinite loop. In case of an error verdict, 'execute' guard timer /if used/ should be checked first for a safe value."


25.2.1 Error verdict
The error verdict is special in that it is set by the test system to indicate that a test case (i.e. run-time) error has occurred. It shall not be set by the setverdict operation and will not be returned by the getverdict operation. No other verdict value can override an error verdict. This means that an error verdict can only be a result of an execute test case operation due to a test system or test suite erroneous design (overflow, unassigned memory, system limitation, infinite loop, deadlock, etc.) which causes impossible to perform test scenario correctly.


BR,
Mariusz Kupiec

---

Original Message

---

From: Claude Desroches
To: This email address is being protected from spambots. You need JavaScript enabled to view it.
Sent: Thursday, December 19, 2002 11:49 AM
Subject: Re: Error verdict in 'execute' operation due to timeout


Hi Mariusz,

I would prefer setting the verdict to inconclusive as opposed to fail, since a guard timer
may be set to a value which is too short to allow the test to occur. If a test case is
interrupted by a timer outside of the test case execution environment, it is not the test
which failed, but the fact that not enough time was allocated it to execute to completion.

I would prefer a value of inconc. :-)

Cheers,

Claude.