-one of my email addresses from 1990 is still valid. you would not believe the amount of spam it gets, as those days putting your real address on a usenet posting wasn't considered daft.
-in 1990, testing was niche. Test-centric, test-first dev is the big change. It's taken a while to go mainstream, with MS Visual Studio Architect being the last tool to pretend that testing was for little people -that version is gone now.
One big change I see now is that my test runs create clusters of machines, run the work, tear things down, try and grab logs. Having more of that test data mined, having the logs analysed for me, being able to compare with past runs would be an improvement. Then there's planet-lab, making it possible to test-run across a real distributed infrastructure. That is still not mainstream enough; Everone should be doing that from their IDE. Or Emacs, if that is still live.
Funny, that was just the opposite thought that I was having when considering hiring; technological experience more that about 10 years old is completely irrelevant. Think about what OSs and technologies have come about, USB, bluetooth, wifi on the hardware side alone were non-existent or in their infancy at the turn of the century. Would someone who has never Googled be a goos choice for testing Google maps? Would someone who has never touched a cell phone be good at testing an iPhone app?
Perhaps the difference is that I generally work on small team where a tester has to take a wider focus on the package and the product with the interface (UI or API) testing is a small part of the picture.
Comparing the general public's general use of technology to how a slim segment of our industry does day to day work is a false dichotomy. How does it look when you compare programming with testing over those 20 years? Has programming changed fundamentally while testing languished?
So I'm not a tester and I was still in high school in 1980. But did we're there unit testing frameworks? Mocking Frameworks? We're systems decoupled and testable with Inversion of Control and IOC containers?
Was there Acceptance frameworks for the mainframe screens like we have today for web applications and Windows applications (white or .net)?
Was their BDD frameworks that allowed us to describe tests in a way that business analysis could write and comprehend?
I think there has been a lot of innovation in the testing space, especially if you are building applications in a TDD style, which probably did exist in some form.
An Interesting thought... Do you think that Software Testing can be closely integraded wtih Development, with teams being taught to do Software Dev/Test independantly so that Software Testing would be extinct by 2030? Just a thought...
There's no question that software has evolved at much faster pace than testing methodologies have been able to keep up with.
Multi-tier apps running distributed with virtual global teams supporting apps from different locations world-wide.
If you're running Java apps on Tomcat, JBoss, or WebLogic please have a look at some innovation in this space call ReplayDIRECTOR. It's a record/replay solution for software.
http://replaysolutions.com
We built it and it's helping a lot of folks improve quality.
Great dreams James... way to go.. dream big & we'll get there one day. Would we be testers if what you are saying comes true? we'd be developers writing applications/platforms which generate test code :) assuming that's what you mean to say with "Imagine an app downloading its test suites"
It'd be good to define a scope to this dream :).. an app varies in size and complexity manyfolds.. an app could be developed and written with 0 external dependencies by 1 person to having 100s of underlying components and dependencies developed by thousands of people..
There are some tools even in today's world which with some tweaks could give u just that (but the challenge becomes harder as the scope of your problem domain gets bigger and complexed).. "download and execute tests on itself" pls lemme know if you are thinking something different than this or hope i'm missing something obvious.
picking up a simple web app for this example.. you could potentially extend this and apply it to a SOA Service as well..
To execute tests against any web app you'll need to know 1. The use case scenarios that define the application's use 2. The controls used to build the page 3. Automation to interact with the controls on the page 4. Automated tests that implement the use case scenarios 5. Execution framework to execute tests, log test results & logs etc..
you could probably achieve by some of the following #1 generate automation that will interact with your web app under test #2 - you can define a custom language for documenting the use cases which then can be translated by a compiler to generate the corresponding automated test cases that leveraged the generated Automation in #1 to interact with the system under test #3 its not hard to automagically define your test environment and contextual information that maybe needed for the tests to execute successfully depending on the configuration of your application #4 then its not hard for an execution framework to pick up the generated test cases and execute them.. #5 the challenge will still remain in analyzing the results to make sure the test case didn't fail because of a valid bug, but that's at least analysis, skipping a lot of it ahead.. and with some superior developer design and applying some logging/monitoring best practices, we could even reduce that and get to the core of the issue quickly, don't think we are far from getting automation to be able to identify atleast 50% of the issues atleast in the not so complexed applications.
hope i'm making sense to you guys.. hope I got my assumption on your dream right.. lemme know if i'm missing something. there's available tools that do most of that.. and some others that can be tweaked to do that..
btw, just thought that it was a bit of an exaggeration to claim the last 20yrs to be the most changing ones in Human history :) Human history has a lot more to it than just technological advances we've made in the last few years... While I do love technology, i think it has its place in human life and history and there's more impactful things that humans have encountered since time has begun... no offense.. i just thought it was too big of a claim to ignore :) sorry, no offense meant.
I have the same thinking, about why is not Software Testing is changing in a fundamental way. People are writing the same kind of scripts for different types of applications. I am currently working to Implement the following :
1. I am Using Covering Arrays for Test Generation, thanks to the people at NIST for the Brilliant ACTS tool: http://csrc.nist.gov/groups/SNS/acts/index.html Want to Use NuSMV as Model Checker and test Oracle, but Since the System Under Test is Large, I need to write a compiler for generating NuSMV code from Application Source Code.
2. trying to learn semantic web to write Semantic Wrappers to Automatically Instrument Code for Autonomous Characteristics of Self healing, to replace the need of Regression Testing.
3. Standard method of Automating defect to remove manual retesting effort.
4. Genetic Algorithm Based Automation Suite with backtracking Capabilities. for Application testing and Distributed Fault Localization.
5. Trying to Learn Constraint Programming to Solve the Combinatorial Search of Defects in the Application. Since requirements can be naturally modelled as Constraints for the application.
James - couldn't agree with you more. the art & science of testing has not changed in the 15 years i've been doing it, except to get more efficient. I don't know that we have materially affected the effectiveness of our testing all that much.
In fact, your assertion applies to all of software development. I recently left MSFT for a local insurance company, and am amazed at how much the mainframe team's development practices here resemble so-called modern PC/Web development practices -- and they've been doing it that way for 30+ years.
It's great to throw down the gauntlet and challenge testers (we've got to start somewhere), but let's also take a look at the state of software development as a whole.
-one of my email addresses from 1990 is still valid. you would not believe the amount of spam it gets, as those days putting your real address on a usenet posting wasn't considered daft.
ReplyDelete-in 1990, testing was niche. Test-centric, test-first dev is the big change. It's taken a while to go mainstream, with MS Visual Studio Architect being the last tool to pretend that testing was for little people -that version is gone now.
One big change I see now is that my test runs create clusters of machines, run the work, tear things down, try and grab logs. Having more of that test data mined, having the logs analysed for me, being able to compare with past runs would be an improvement. Then there's planet-lab, making it possible to test-run across a real distributed infrastructure. That is still not mainstream enough; Everone should be doing that from their IDE. Or Emacs, if that is still live.
Funny, that was just the opposite thought that I was having when considering hiring; technological experience more that about 10 years old is completely irrelevant. Think about what OSs and technologies have come about, USB, bluetooth, wifi on the hardware side alone were non-existent or in their infancy at the turn of the century. Would someone who has never Googled be a goos choice for testing Google maps? Would someone who has never touched a cell phone be good at testing an iPhone app?
ReplyDeletePerhaps the difference is that I generally work on small team where a tester has to take a wider focus on the package and the product with the interface (UI or API) testing is a small part of the picture.
Comparing the general public's general use of technology to how a slim segment of our industry does day to day work is a false dichotomy. How does it look when you compare programming with testing over those 20 years? Has programming changed fundamentally while testing languished?
ReplyDeleteSo I'm not a tester and I was still in high school in 1980. But did we're there unit testing frameworks? Mocking Frameworks? We're systems decoupled and testable with Inversion of Control and IOC containers?
ReplyDeleteWas there Acceptance frameworks for the mainframe screens like we have today for web applications and Windows applications (white or .net)?
Was their BDD frameworks that allowed us to describe tests in a way that business analysis could write and comprehend?
I think there has been a lot of innovation in the testing space, especially if you are building applications in a TDD style, which probably did exist in some form.
An Interesting thought... Do you think that Software Testing can be closely integraded wtih Development, with teams being taught to do Software Dev/Test independantly so that Software Testing would be extinct by 2030? Just a thought...
ReplyDelete- Do visit me @ http://fakesoftwaretester.blogspot.com
There's no question that software has evolved at much faster pace than testing methodologies have been able to keep up with.
ReplyDeleteMulti-tier apps running distributed with virtual global teams supporting apps from different locations world-wide.
If you're running Java apps on Tomcat, JBoss, or WebLogic please have a look at some innovation in this space call ReplayDIRECTOR. It's a record/replay solution for software.
http://replaysolutions.com
We built it and it's helping a lot of folks improve quality.
Great dreams James... way to go.. dream big & we'll get there one day. Would we be testers if what you are saying comes true? we'd be developers writing applications/platforms which generate test code :) assuming that's what you mean to say with "Imagine an app downloading its test suites"
ReplyDeleteIt'd be good to define a scope to this dream :).. an app varies in size and complexity manyfolds.. an app could be developed and written with 0 external dependencies by 1 person to having 100s of underlying components and dependencies developed by thousands of people..
There are some tools even in today's world which with some tweaks could give u just that (but the challenge becomes harder as the scope of your problem domain gets bigger and complexed).. "download and execute tests on itself" pls lemme know if you are thinking something different than this or hope i'm missing something obvious.
picking up a simple web app for this example.. you could potentially extend this and apply it to a SOA Service as well..
To execute tests against any web app you'll need to know
1. The use case scenarios that define the application's use
2. The controls used to build the page
3. Automation to interact with the controls on the page
4. Automated tests that implement the use case scenarios
5. Execution framework to execute tests, log test results & logs etc..
you could probably achieve by some of the following
#1 generate automation that will interact with your web app under test
#2 - you can define a custom language for documenting the use cases which then can be translated by a compiler to generate the corresponding automated test cases that leveraged the generated Automation in #1 to interact with the system under test
#3 its not hard to automagically define your test environment and contextual information that maybe needed for the tests to execute successfully depending on the configuration of your application
#4 then its not hard for an execution framework to pick up the generated test cases and execute them..
#5 the challenge will still remain in analyzing the results to make sure the test case didn't fail because of a valid bug, but that's at least analysis, skipping a lot of it ahead.. and with some superior developer design and applying some logging/monitoring best practices, we could even reduce that and get to the core of the issue quickly, don't think we are far from getting automation to be able to identify atleast 50% of the issues atleast in the not so complexed applications.
hope i'm making sense to you guys.. hope I got my assumption on your dream right.. lemme know if i'm missing something. there's available tools that do most of that.. and some others that can be tweaked to do that..
btw, just thought that it was a bit of an exaggeration to claim the last 20yrs to be the most changing ones in Human history :)
Human history has a lot more to it than just technological advances we've made in the last few years... While I do love technology, i think it has its place in human life and history and there's more impactful things that humans have encountered since time has begun... no offense.. i just thought it was too big of a claim to ignore :) sorry, no offense meant.
@ fake software: do u think your model is a good one? have you ever tried continuous integration with instant testing feedback? give it a try!;)
ReplyDeleteI have the same thinking, about why is not Software Testing is changing in a fundamental way. People are writing the same kind of scripts for different types of applications. I am currently working to Implement the following :
ReplyDelete1. I am Using Covering Arrays for Test Generation, thanks to the people at NIST for the Brilliant ACTS tool:
http://csrc.nist.gov/groups/SNS/acts/index.html
Want to Use NuSMV as Model Checker and test Oracle, but Since the System Under Test is Large, I need to write a compiler for generating NuSMV code from Application Source Code.
2. trying to learn semantic web to write Semantic Wrappers to Automatically Instrument Code for Autonomous Characteristics of Self healing, to replace the need of Regression Testing.
3. Standard method of Automating defect to remove manual retesting effort.
4. Genetic Algorithm Based Automation Suite with backtracking Capabilities. for Application testing and Distributed Fault Localization.
5. Trying to Learn Constraint Programming to Solve the Combinatorial Search of Defects in the Application. Since requirements can be naturally modelled as Constraints for the application.
James - couldn't agree with you more. the art & science of testing has not changed in the 15 years i've been doing it, except to get more efficient. I don't know that we have materially affected the effectiveness of our testing all that much.
ReplyDeleteIn fact, your assertion applies to all of software development. I recently left MSFT for a local insurance company, and am amazed at how much the mainframe team's development practices here resemble so-called modern PC/Web development practices -- and they've been doing it that way for 30+ years.
It's great to throw down the gauntlet and challenge testers (we've got to start somewhere), but let's also take a look at the state of software development as a whole.