Hi all... I've been trying out tumblr.com for a month or so now for my blogging-ish outlet and have been really digging it. For my 7 followers out there, I think I've made the decision to post solely over there--check it at http://musicismyeverest.tumblr.com. I've actually moved all my posts from here to there too!
13 February 2010
16 January 2010
Seven Steps to Test Automation Success
In starting a new test automation group at work, I started doing some industry research to get some do/don't ideas. I really, really dislike the idea of GUI x,y coordinate automation; other GUI object-aware automation tools seem to be pricey and still seem like tons of maintenance and really distanced from testing the real meat of most apps. Most info that I've found on the topic has either dealt with things at the GUI level (probably since that's "easiest" to implement, despite being difficult to maintain), or at the unit level--while these focuses certainly have their merits, I was looking for info on the level in between those things; Google calls this the "medium" level.
I found this article by Brett Pettichord called "Seven Steps to Test Automation Success", where he talks about the difficulties in automating tests, and I really think he hits a lot of nails on the head. He argues that in developing automate tests, you should really treat your work like that of software development, then discusses his seven steps:
- Improve the Testing Process
- Define Requirements
- Prove the Concept
- Champion Product Testability
- Design for Sustainability
- Plan for Deployment
- Face the Challenges of Success
All in all, there's tons of meat in this article, so take it in sections or get a cup of coffee and get comfy, but either way I think this is going to be a reference of mine for quite some time now.
Ruby Net::HTTP Cheat Sheet
I think this will prove to be a good resource--especially since I don't use this library that often. Hope you find it useful...
Net::HTTP Cheat Sheet
04 November 2009
Clean IRCing on OSX
I've been IRCing at work quite a bit lately, which has been pretty great. Every now and then I find it exciting to return back to old tech and start using it again--it's usually simple and complicated all at the same time, but for some reason I find that stimulating.
In any case... I used Colloquy for a while, which is a great modern IRC client, but after dealing with some annoying bugs, downloading the source, rebuilding, and not getting any love, I decided to search for something new. And I found irssi. irssi is totally old school, but totally new school. I love it. There are a ton of getting started guides out there, but it was nice to find a Mac-specific one: A Guide to Efficiently Using Irssi and Screen | quadpoint.org; definitely worth checking out.
My irssi'ing got even better when combining with Blacktree's (makers of Quicksilver) visor. Visor is just a simple HUD-type SIMBL plugin for Terminal.app that allows for key-combo hiding/unhiding. This is great for moving my IRC client (Terminal.app + irssi) on/off screen quickly, keeping my desktop clean so I can focus on my work.
Just thought I'd share with y'all...
19 May 2009
GTAC 2008 Keynote Address: The Future of Testing
Quite a thought provoking speech. I was fortunate to see this in person! Take an hour and check it out--it's totally worth it.
23 February 2009
Hitler's nightly build fails
Oh man this is good. We should hire (no, not "heil") Hitler.... I love the part where the lady consoles the other lady saying, "Don't worry, he didn't mean that about Scrum." Genius.
26 September 2008
Politics should be like Software Development
25 July 2008
Vista auto-reboot-for-updates #4
12 June 2008
Vista auto-reboot-for-updates #3
11 June 2008
RANT: writing 1000 test cases is not that bad
04 June 2008
Words (for all developers) To Live By
29 May 2008
Vista & Its Own Mind #2
28 May 2008
QC's: Maturity v. Stability
ISO 9126 presents the concepts of "quality characteristics", which is proving to be both helpful and confusing here at work. We've had our longest group process meetings on this topic--just trying to figure out and agree on what each one of these means. One of the big problems is figuring out what some of these are not. We use many of these words loosely in this world of testing/development, thus setting ourselves up for confusion when talking about this stuff.
Technical Malapropisms
Now, no one likes a vocal grammar Nazi around, but sometimes it really is important to talk about the distinctions between different things you talk about on a day to day basis.
One of the many bits of confusion came when talking about "Stability". Stability falls (as a sub-characteristic) under the main characteristic of "Maintainability", which can make sense if you take a minute to ponder. When we started trying to come up with examples of this, however, most of us came up with examples of "Maturity", which falls under "Reliability." It seems that in non-ISO conversations, stability is talked about in a way that encompasses both of these sub-characteristics. I did some brief google-ing to see if we were the only ones with this "problem" and found an abstract on Ecological Modelling: An inverse relationship between stability and maturity in models of aquatic ecosystems. This article on software maturity models suggests that stability is a factory that makes up maturity. ISO says otherwise:
Maturity:
The capability of the software product to avoid failure as a result of faults in the software.
Stability:
The capability of the software product to avoid unexpected effects from modifications of the software.
Maturity deals with "failure as a result of faults"; Stability deals with "unexpected effects from modifications."
Failures v. Unexpected Effects
Maturity deals with failures; Stability deals with unexpected effects. What's the difference? ...in terms of test cases passing I say, nothing. Both are the absence of passing; it doesn't matter what you call it. ...unless this means to say that a product is unstable if changes in module A unexpectedly effect the behavior in module B. One could then interpret "failure" as the inability for module A to meet specification for module A when testing module A; "unexpected effects" are then the inability for module A to meet specification for module A when testing module B.
Faults v. Modifications
Maturity deals with faults in the software; Stability deals with modifications of the software. What's the difference? Key differences become possibly more confusing when you take a look at the definition and notes of the main characteristics for each sub-characteristic. Note 1 for "Reliability" (parent of Maturity) says:
"Wear or aging does not occur in software. Limitations in reliability are due to faults in requirements, design, and implementation. Failures due to these faults depend on the way the software product is used and the program options selected rather than on elapsed time."Interesting. A fault can be in the requirements. A fault can be in the design. A fault can be in the implementation. Lack of maturity, then, is failures in one of these fault areas. The 2nd sentence in the definition of "Maintainability" (parent of Stability) says:
"Modifications may include corrections, improvements or adaptation of the software to changes in environment, and in requirements and functional specifications."A modification requires that something exist already; a new "version" of that thing comes about after modifications are made. Stability, then, is strictly tied to how the application behaves in comparison to how it behaved in the last version. Lack of stability is evident, then, when the same test activities are conducted in two different versions of the application, both in the same environment, and failures occur in unexpected areas of the application.
Conclusions
So one might be led to conclude that when testing version 2 of some app, you really could run test A and essentially be testing for both stability and maturity. But like many of these other ISO characteristics, the line is quite fine. The only way I can see it comes back to relate to one thing: what's effected. If you test A after changes were made in A, then you're testing maturity of A. If you test [B-Z] after making changes in A, you're testing stability of the app.
21 May 2008
14 May 2008
Teach me to test
05 May 2008
A test case is a scientific procedure
21 April 2008
Testing is an Engineering discipline
Engineering is the discipline and profession of applying scientific knowledge and utilizing natural laws and physical resources in order to design and implement materials, structures, machines, devices, systems, and processes that realize a desired objective and meet specified criteria. More precisely, engineering can be defined as “the creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.”Engineering - Wikipedia, the free encyclopedia ...and notice the word "discipline". Development Engineers can (and should be) hounded for bad formatting and not commenting their code--along those same lines of being disciplined, let the rest of us know what you're trying to accomplish when writing your tests! Be explicit about what you're trying to test for, show steps for how to test for that, and be concise about what to expect. Just as a screwy, uncommented piece of code that someone else wrote can drive a Development Engineer nuts, a screwy test can have the same effect on your fellow Test Engineers. Please, learn to be disciplined in your work--you'll gain respect from your co-workers and probably realize that you missed a few things along the way. It's not hard--it just takes discipline!
16 April 2008
US FDA/CDRH: General Principles of Software Validation; Final Guidance for Industry and FDA Staff
Software verification and validation are difficult because a developer cannot test forever, and it is hard to know how much evidence is enough. In large measure, software validation is a matter of developing a "level of confidence" that the device meets all requirements and user expectations for the software automated functions and features of the device. Measures such as defects found in specifications documents, estimates of defects remaining, testing coverage, and other techniques are all used to develop an acceptable level of confidence before shipping the product. The level of confidence, and therefore the level of software validation, verification, and testing effort needed, will vary depending upon the safety risk (hazard) posed by the automated functions of the device.There's also a great section called "Software is Different From Hardware", which points out some great subtle-but-huge differences between the two. Section 5. Activities and Tasks has some good practical info on planning and test tasks--both Black Box and White Box (although not explicitly so). US FDA/CDRH: General Principles of Software Validation; Final Guidance for Industry and FDA Staff