Hi,

I want to raise discussion on 'execute' function protected by timer once again. I fully agree with 'error' verdict as Jens and György have pointed me out. But just after waking up in the morning I got the following conclusions:


What are parts of life of a test siute?
1. Test designer prepares source code, which after completion may be finally tested in a test suite manufacturer environment. Test cases may hang on. Test suite designer may protect none/some/all test cases using 'execute' calls with timer value, which prevents hanging on.
2. Test suite appears as a market product, usually in compiled form /not source/ for a specific platform, e.g. for a Forth machines, which speeds up TS execution. Test suite executor executes it on a manufacturer IUT. Test cases may hang on. None/some/all test cases were protected using 'execute' calls with timer value, which prevents hanging on.


Questions /or request for training material/:
1. Test suite writer is responsible for protocol tests implementation.

Does he should think about runtime errors?
Does he should protect only test cases hanging on in manufacturer test environment?
If not, which test cases or how many test cases should be protected? Does it depend on designer wishes, test case complexity, other reasons?
All these questions are result of observation that only test suite writer may have access to source code of a test suite.

2. For TTCN-2 there was not method of test case time protection. Test cases falling into the infinite loop were simply stopped/removed manually by protocol tester operator. At least I expect that such a possibility was implemented to avoid resetting of tester, interfaces, reloading of system, etc.

Does this method of protection embedded in a TTCN-3 specification is a preferred method /because standardized and supported by specification/ of continuation of test suite execution?
Do other methods may be implemented in protocol testers, e.g. manually setting individual execution time for each test case?
Do these methods should be independent or joined into one?

3. Parametrization of test case timers in a high range may affect availability to determine the 'execute' timer value.

Examples of 'execute' timer in specification use fixed values. Let be a test case in a compiled suite, which uses T_IMPLICIT_EVENT time, which value may be set in a high range, e.g. up to several minutes. Is it possible to protect such a test case with a fixed value of 'execute' timer? How to calculate this value?
Assume there seems to be need for parametrisation of 'execute' timer in case of timers used inside test are in a high range (e.g. T_IMPLICIT_EVENT time). Should 'execute' timer value be passed as a test suite parameter?

4. Assume 'execute' timer method is preferred method of ensuring test suite execution continuation.
Assume we want to give test suite executor access to all 'execute' timers keeping test suite in any compiled form /not source/.
All test cases may hang on, all may/should be protected to ensure unattended test suite execution.

Does it mean all 'execute' timer values should be passed as test usite parameters?
For a number of test case equal to 300 we have additional 300 t.s. parameters and there is also need for '0' value interpretation as 'timer not used'.
There returns previous question, what will happen in case of two independent methods of test case protection, e.g. by t.s. parameters and by individual execution time set in a tester. Should they be joined into one mechanism? Does it require to protect every test case by default by test suite writer?

5. If not all test cases were protected by 'execute' timer and 'test case execution goes an unexpected way' and system can not issue error verdict due to unprotected test case, then test suite conforms to TTCN-3 specification or not? The error verdict 'can be and shall be' set by test system, but setting 'error' depends if execute timer was set for a test case, for which test suite designer is responsible.

6. How useful is a test suite, where some timers were fixed in a very complicated way. This is what I have written earlier in the proposal:

"If used, value of a system timer guarding 'execute' operation should be calculated carefully, inluding all timer margins and signals propagation, at least for a value, which allow proper testcase execution on a correct implementation for a worst /maximum allowed delays/ case."

How this suite can be modified by using protocols timers in test suite parameters if 'execute' timers are fixed?
Is test suite useful for e.g. not only final conformance testing but also for testing incomplete implementations, where e.g. preambles can take longer time than in a final IUT due to other configuration.
Should fixed 'execute' timer values discredit test suite as a highly configurable set of test cases for a given protocol?

---

I hope I asked all questions.
I have a bad feeling that mechanism, which is rather useful for test suite executor is all the time in test suite writer hands.

BR,
Mariusz Kupiec

Hi Mariusz!

Indeed a very long mail but filled with nutritious questions. A lot of the
question are valid and I think either there should be some methods defined
in order to clarify this in an explicit way or test developers shouldn't
bother too much about how to implement the control part of the test suite.

One method could for instance start with separating the definitions part and
the control part to keep the test as abstract as possible. I reckon this is
what one of the TCI tasks are about.

Personally I think this is the way to go, i.e. maintain as much as possible
in the TTCN-2 abstract manner and leave the responsability to the test
developers making adaptations on how to implement the execution of the test
cases, preferrably using TCI. But thats my opinion.

Finally I am looking forward to helping out with making a sound methodology
for how to use TTCN-3 in real-life. Something I miss right now in our TTCN
world which I think could be benefitial for as all.

Enjoy the weekend

/Stefan

---

Original Message

---

From: Mariusz Kupiec
To: This email address is being protected from spambots. You need JavaScript enabled to view it.
Sent: 1/17/2003 8:15 PM
Subject: 'Execute' function and timer protection once more /very long/

Hi,

I want to raise discussion on 'execute' function protected by timer once
again. I fully agree with 'error' verdict as Jens and György have
pointed me out. But just after waking up in the morning I got the
following conclusions:


What are parts of life of a test siute?
1. Test designer prepares source code, which after completion may be
finally tested in a test suite manufacturer environment. Test cases may
hang on. Test suite designer may protect none/some/all test cases using
'execute' calls with timer value, which prevents hanging on.
2. Test suite appears as a market product, usually in compiled form /not
source/ for a specific platform, e.g. for a Forth machines, which speeds
up TS execution. Test suite executor executes it on a manufacturer IUT.
Test cases may hang on. None/some/all test cases were protected using
'execute' calls with timer value, which prevents hanging on.


Questions /or request for training material/:
1. Test suite writer is responsible for protocol tests implementation.

Does he should think about runtime errors?
Does he should protect only test cases hanging on in manufacturer
test environment?
If not, which test cases or how many test cases should be protected?
Does it depend on designer wishes, test case complexity, other reasons?
All these questions are result of observation that only test suite
writer may have access to source code of a test suite.

2. For TTCN-2 there was not method of test case time protection. Test
cases falling into the infinite loop were simply stopped/removed
manually by protocol tester operator. At least I expect that such a
possibility was implemented to avoid resetting of tester, interfaces,
reloading of system, etc.

Does this method of protection embedded in a TTCN-3 specification is
a preferred method /because standardized and supported by specification/
of continuation of test suite execution?
Do other methods may be implemented in protocol testers, e.g.
manually setting individual execution time for each test case?
Do these methods should be independent or joined into one?

3. Parametrization of test case timers in a high range may affect
availability to determine the 'execute' timer value.

Examples of 'execute' timer in specification use fixed values. Let
be a test case in a compiled suite, which uses T_IMPLICIT_EVENT time,
which value may be set in a high range, e.g. up to several minutes. Is
it possible to protect such a test case with a fixed value of 'execute'
timer? How to calculate this value?
Assume there seems to be need for parametrisation of 'execute' timer
in case of timers used inside test are in a high range (e.g.
T_IMPLICIT_EVENT time). Should 'execute' timer value be passed as a test
suite parameter?

4. Assume 'execute' timer method is preferred method of ensuring test
suite execution continuation.
Assume we want to give test suite executor access to all 'execute'
timers keeping test suite in any compiled form /not source/.
All test cases may hang on, all may/should be protected to ensure
unattended test suite execution.

Does it mean all 'execute' timer values should be passed as test
usite parameters?
For a number of test case equal to 300 we have additional 300 t.s.
parameters and there is also need for '0' value interpretation as 'timer
not used'.
There returns previous question, what will happen in case of two
independent methods of test case protection, e.g. by t.s. parameters and
by individual execution time set in a tester. Should they be joined into
one mechanism? Does it require to protect every test case by default by
test suite writer?

5. If not all test cases were protected by 'execute' timer and 'test
case execution goes an unexpected way' and system can not issue error
verdict due to unprotected test case, then test suite conforms to TTCN-3
specification or not? The error verdict 'can be and shall be' set by
test system, but setting 'error' depends if execute timer was set for a
test case, for which test suite designer is responsible.

6. How useful is a test suite, where some timers were fixed in a very
complicated way. This is what I have written earlier in the proposal:

"If used, value of a system timer guarding 'execute' operation should be
calculated carefully, inluding all timer margins and signals
propagation, at least for a value, which allow proper testcase execution
on a correct implementation for a worst /maximum allowed delays/ case."

How this suite can be modified by using protocols timers in test suite
parameters if 'execute' timers are fixed?
Is test suite useful for e.g. not only final conformance testing but
also for testing incomplete implementations, where e.g. preambles can
take longer time than in a final IUT due to other configuration.
Should fixed 'execute' timer values discredit test suite as a highly
configurable set of test cases for a given protocol?

---

I hope I asked all questions.
I have a bad feeling that mechanism, which is rather useful for test
suite executor is all the time in test suite writer hands.

BR,
Mariusz Kupiec

---

Poczta nowych mozliwosci >>> link.interia.pl/f16bc
<link.interia.pl/f16bc>

Hi,

I see your mail mainly as request for training material, for which
due to the preferences of ETSI TC MTS not enough money is available
at the moment. However, I understand ETSI as an organization which
is driven by the needs of its members and therefore your company
and other ETSI members have to ask for additional resources for the
development of such training material.

Some material has been written by György under the umbrella of some
other project, but I do not know the status. Maybe György can add
something?

Even though I understand all your questions, I have the feeling
that the answers heavily depend on the environment in which TTCN-3
is used (i.e., there will be several answers!) and may even go
beyond an ETSI methodology.

Regards
Jens





> Mariusz Kupiec schrieb:
>
> Hi,
>
> I want to raise discussion on 'execute' function protected by timer
> once again. I fully agree with 'error' verdict as Jens and György have
> pointed me out. But just after waking up in the morning I got the
> following conclusions:
>
>
> What are parts of life of a test siute?
> 1. Test designer prepares source code, which after completion may be
> finally tested in a test suite manufacturer environment. Test cases
> may hang on. Test suite designer may protect none/some/all test cases
> using 'execute' calls with timer value, which prevents hanging on.
> 2. Test suite appears as a market product, usually in compiled form
> /not source/ for a specific platform, e.g. for a Forth machines, which
> speeds up TS execution. Test suite executor executes it on a
> manufacturer IUT. Test cases may hang on. None/some/all test cases
> were protected using 'execute' calls with timer value, which prevents
> hanging on.
>
>
> Questions /or request for training material/:
> 1. Test suite writer is responsible for protocol tests implementation.
>
> Does he should think about runtime errors?
> Does he should protect only test cases hanging on in manufacturer
> test environment?
> If not, which test cases or how many test cases should be
> protected? Does it depend on designer wishes, test case complexity,
> other reasons?
> All these questions are result of observation that only test suite
> writer may have access to source code of a test suite.
>
> 2. For TTCN-2 there was not method of test case time protection. Test
> cases falling into the infinite loop were simply stopped/removed
> manually by protocol tester operator. At least I expect that such a
> possibility was implemented to avoid resetting of tester, interfaces,
> reloading of system, etc.
>
> Does this method of protection embedded in a TTCN-3 specification
> is a preferred method /because standardized and supported by
> specification/ of continuation of test suite execution?
> Do other methods may be implemented in protocol testers, e.g.
> manually setting individual execution time for each test case?
> Do these methods should be independent or joined into one?
>
> 3. Parametrization of test case timers in a high range may affect
> availability to determine the 'execute' timer value.
>
> Examples of 'execute' timer in specification use fixed values. Let
> be a test case in a compiled suite, which uses T_IMPLICIT_EVENT time,
> which value may be set in a high range, e.g. up to several minutes. Is
> it possible to protect such a test case with a fixed value of
> 'execute' timer? How to calculate this value?
> Assume there seems to be need for parametrisation of 'execute'
> timer in case of timers used inside test are in a high range (e.g.
> T_IMPLICIT_EVENT time). Should 'execute' timer value be passed as a
> test suite parameter?
>
> 4. Assume 'execute' timer method is preferred method of ensuring test
> suite execution continuation.
> Assume we want to give test suite executor access to all 'execute'
> timers keeping test suite in any compiled form /not source/.
> All test cases may hang on, all may/should be protected to ensure
> unattended test suite execution.
>
> Does it mean all 'execute' timer values should be passed as test
> usite parameters?
> For a number of test case equal to 300 we have additional 300 t.s.
> parameters and there is also need for '0' value interpretation as
> 'timer not used'.
> There returns previous question, what will happen in case of two
> independent methods of test case protection, e.g. by t.s. parameters
> and by individual execution time set in a tester. Should they be
> joined into one mechanism? Does it require to protect every test case
> by default by test suite writer?
>
> 5. If not all test cases were protected by 'execute' timer and 'test
> case execution goes an unexpected way' and system can not issue error
> verdict due to unprotected test case, then test suite conforms to
> TTCN-3 specification or not? The error verdict 'can be and shall be'
> set by test system, but setting 'error' depends if execute timer was
> set for a test case, for which test suite designer is responsible.
>
> 6. How useful is a test suite, where some timers were fixed in a very
> complicated way. This is what I have written earlier in the proposal:
>
> "If used, value of a system timer guarding 'execute' operation should
> be calculated carefully, inluding all timer margins and signals
> propagation, at least for a value, which allow proper testcase
> execution on a correct implementation for a worst /maximum allowed
> delays/ case."
>
> How this suite can be modified by using protocols timers in test suite
> parameters if 'execute' timers are fixed?
> Is test suite useful for e.g. not only final conformance testing but
> also for testing incomplete implementations, where e.g. preambles can
> take longer time than in a final IUT due to other configuration.
> Should fixed 'execute' timer values discredit test suite as a highly
> configurable set of test cases for a given protocol?
>
>

---

> I hope I asked all questions.
> I have a bad feeling that mechanism, which is rather useful for test
> suite executor is all the time in test suite writer hands.
>
> BR,
> Mariusz Kupiec
>

---

>
> Poczta nowych mozliwosci >>> link.interia.pl/f16bc

--

======================================================================
Dr. Jens Grabowski
Institute for Telematics phone: +49 451 500 3723
University of Luebeck fax: +49 451 500 3722
Ratzeburger Allee 160 eMail: This email address is being protected from spambots. You need JavaScript enabled to view it.
D-23538 Luebeck or This email address is being protected from spambots. You need JavaScript enabled to view it.
(Germany) WWW: www.itm.mu-luebeck.de
======================================================================