Small tests should be isolated from each other, but this constraint gets in a way of medium and large tests (especially for web testing). Take a look at what users are doing with Selenium + TestNG.
Also, dependencies and high parallelism are not mutually exclusive, it's unfortunate that this rumor is still around. Here is why:
I was wondering how you are dealing with the issue that Medium or Large tests might want to exercise the system along a "path" (for example, perform step a,b,c then verify), where step b might be dependent on state from step a, and the requirement that they tests should be able to run in parallel? Does the Large tests have a huge "setUp" function to get to the state they need to be in before performing the actual test?
@helino: this is why Selenium users tend to use TestNG: because it supports test dependencies.
Not only does this save a tremendous amount of setup/teardown time (and less use of statics) but it also leads to more precise reports (i.e. "1 failed, 99 skipped" instead of "100 failed").
I like the fact that you can measure the criteria to decide what kind of test you´re talking about. So there´s no room for any fuzzy/blurry interpreations.
To avoid misunderstandings/ miscommunication because of using test-terminology differently, we decided to rely on the ISTQB-glossary as only valid glossary, and for us this works fine.
I also like the idea of independent tests, that don´t rely on each other. We´re trying to do the same at our company, but it´s often very hard to explain to other people.
Can you give an example of using these terms, "network access, database, file system, ...", to test an application? Can you give such example with a procedure with step by step?
In my mind, there is a procedure. For example, you may setup an entry main method, and then in the main method you call method A, B, C, and finally you trigger the network access function, D. After you have such path to trigger D, and then you begin to feed small, medium and large dataset to the main function.
If this is the case, then I think you are talking about an input data generation step during an concrete integration testing over a group of components.
I really like your table and the fact that you guys seem to carefully design the kind of tests you're writing.
Limiting runtime of test suites is a new thought to me but very interesting. After all, test execution also has cost attached in a way that somebody might be "waiting" for the Continuous Integration server to "approve" a check-in.
The issue with all the different names for kinds of tests has not been addressed, right? Using a consistent terminology to classify tests surely helps you guys internally but for that purpose, the more common terms to classify test cases would do, too!?
Module, Integration, End2end types of test cases shows level of isolation (!) of particular test! And it does not corresponds to "size" of test in any way.
"A Small test equates neatly to a unit test, a Large test to an end-to-end or system test and a Medium " - here you comparing "green" and "soft" objects...their parameters are of different perspectives lets say.
As previous commenters mentioned this terminology of tests are clearly stated in ISTQB standarts and it make sense to check it and understand real purpose behind original such naming.
Small tests should be isolated from each other, but this constraint gets in a way of medium and large tests (especially for web testing). Take a look at what users are doing with Selenium + TestNG.
ReplyDeleteAlso, dependencies and high parallelism are not mutually exclusive, it's unfortunate that this rumor is still around. Here is why:
http://beust.com/weblog/2009/11/28/hard-core-multicore-with-testng/
Hi, thanks for sharing your thoughts on this!
ReplyDeleteI was wondering how you are dealing with the issue that Medium or Large tests might want to exercise the system along a "path" (for example, perform step a,b,c then verify), where step b might be dependent on state from step a, and the requirement that they tests should be able to run in parallel? Does the Large tests have a huge "setUp" function to get to the state they need to be in before performing the actual test?
So I don't think you answered your own question.
ReplyDeleteWhat do Googlers (who like to make decisions based on data) call a test that tests your application through its UI?
Is the time limit referring to a single test (test fixture) or an entire suite?
ReplyDelete@helino: this is why Selenium users tend to use TestNG: because it supports test dependencies.
ReplyDeleteNot only does this save a tremendous amount of setup/teardown time (and less use of statics) but it also leads to more precise reports (i.e. "1 failed, 99 skipped" instead of "100 failed").
I like the fact that you can measure the criteria to decide what kind of test you´re talking about. So there´s no room for any fuzzy/blurry interpreations.
ReplyDeleteTo avoid misunderstandings/ miscommunication because of using test-terminology differently, we decided to rely on the ISTQB-glossary as only valid glossary, and for us this works fine.
I also like the idea of independent tests, that don´t rely on each other. We´re trying to do the same at our company, but it´s often very hard to explain to other people.
Can you give an example of using these terms, "network access, database, file system, ...", to test an application? Can you give such example with a procedure with step by step?
ReplyDeleteIn my mind, there is a procedure. For example, you may setup an entry main method, and then in the main method you call method A, B, C, and finally you trigger the network access function, D. After you have such path to trigger D, and then you begin to feed small, medium and large dataset to the main function.
If this is the case, then I think you are talking about an input data generation step during an concrete integration testing over a group of components.
Look forward to hear you back, and best wishes.
I really like your table and the fact that you guys seem to carefully design the kind of tests you're writing.
ReplyDeleteLimiting runtime of test suites is a new thought to me but very interesting. After all, test execution also has cost attached in a way that somebody might be "waiting" for the Continuous Integration server to "approve" a check-in.
The issue with all the different names for kinds of tests has not been addressed, right? Using a consistent terminology to classify tests surely helps you guys internally but for that purpose, the more common terms to classify test cases would do, too!?
Limit of 60 seconds for unit tests sound way too high... how do you manage to write a unit test that can take so long?
ReplyDeletePackaging tests into S, M and L. It again depends.
DeleteWhat if Small tests works less than 60 seconds in release mode and more than 60 seconds in debug mode?
ReplyDeleteHi,
ReplyDeleteModule, Integration, End2end types of test cases shows level of isolation (!) of particular test! And it does not corresponds to "size" of test in any way.
"A Small test equates neatly to a unit test, a Large test to an end-to-end or system test and a Medium " - here you comparing "green" and "soft" objects...their parameters are of different perspectives lets say.
As previous commenters mentioned this terminology of tests are clearly stated in ISTQB standarts and it make sense to check it and understand real purpose behind original such naming.