Vista auto-reboot-for-updates #3

Yet again, I'm in the middle of my morning emailing, and all of a sudden, Vista decides to reboot without warning.  How the hell could M$ ever decide this was a good idea?  I really can't think of anything more frustrating than losing work.  And oh ya... it's been "Configuring updates" for the past 10 minutes now... awesome.  *sigh*

RANT: writing 1000 test cases is not that bad

I've heard a number of times over the past few months here at work things like: "...but if we do it like that, that means I'll be writing test cases forever!"  Testers: do you think developers say that when they're assigned to engineer some feature?  If they do, they probably won't be keeping their job long.  When assigned to engineer a feature, a developer goes through some process to figure out what it is they need to do/not do, then write the code that makes the thing do that, one line of code at a time, until the thing does what they think it should do.  What's so craaaazy about testing those lines of code in their different permutations??  Sure, there's merit in testing efficiently, but that's beside the point.  The point is, you do what it takes to get the job done--if that means writing 1000 new test cases in the next month, then that's what it means.

Words (for all developers) To Live By

In coming up with a presentation for work on characteristics of quality, I've done a lot of expanding my brain.  Lots of defining, clarifying, quantifying, and a fair amount of reading.  This morning I took another looks at Apple's ADC guide for Human Interface Guidelines--particularly the part on Characteristics of Great Software.  I'm pretty sure they'd updated it since the last time I'd been there, as I noticed a link on that page to another page called "Know Your Audience".  I followed. First nugget: "It is useful to create scenarios that describe a typical day of a person who uses the type of software product you are designing." Second nugget: "Develop your product with people and their capabilities—not computers and their capabilities—in mind." Third nugget: "It is not your needs or your usage patterns that you are designing for, but those of your (potential) customers." Read it. Enough said.

Vista & Its Own Mind #2

Yet again, Vista presents the pleasant surprise of exiting all my work and rebooting, just to apply updates that it never told me about. I'm appalled for the second time now. Update: And Outlook didn't even save the emails I had written but not yet sent... So freaking annoying.

QC's: Maturity v. Stability

ISO 9126 presents the concepts of "quality characteristics", which is proving to be both helpful and confusing here at work. We've had our longest group process meetings on this topic--just trying to figure out and agree on what each one of these means. One of the big problems is figuring out what some of these are not. We use many of these words loosely in this world of testing/development, thus setting ourselves up for confusion when talking about this stuff.

Technical Malapropisms
Now, no one likes a vocal grammar Nazi around, but sometimes it really is important to talk about the distinctions between different things you talk about on a day to day basis.

One of the many bits of confusion came when talking about "Stability". Stability falls (as a sub-characteristic) under the main characteristic of "Maintainability", which can make sense if you take a minute to ponder. When we started trying to come up with examples of this, however, most of us came up with examples of "Maturity", which falls under "Reliability." It seems that in non-ISO conversations, stability is talked about in a way that encompasses both of these sub-characteristics. I did some brief google-ing to see if we were the only ones with this "problem" and found an abstract on Ecological Modelling: An inverse relationship between stability and maturity in models of aquatic ecosystems. This article on software maturity models suggests that stability is a factory that makes up maturity. ISO says otherwise:

Maturity:
The capability of the software product to avoid failure as a result of faults in the software.

Stability:
The capability of the software product to avoid unexpected effects from modifications of the software.

Maturity deals with "failure as a result of faults"; Stability deals with "unexpected effects from modifications."

Failures v. Unexpected Effects
Maturity deals with failures; Stability deals with unexpected effects. What's the difference? ...in terms of test cases passing I say, nothing. Both are the absence of passing; it doesn't matter what you call it. ...unless this means to say that a product is unstable if changes in module A unexpectedly effect the behavior in module B. One could then interpret "failure" as the inability for module A to meet specification for module A when testing module A; "unexpected effects" are then the inability for module A to meet specification for module A when testing module B.

Faults v. Modifications
Maturity deals with faults in the software; Stability deals with modifications of the software. What's the difference? Key differences become possibly more confusing when you take a look at the definition and notes of the main characteristics for each sub-characteristic. Note 1 for "Reliability" (parent of Maturity) says:

"Wear or aging does not occur in software. Limitations in reliability are due to faults in requirements, design, and implementation. Failures due to these faults depend on the way the software product is used and the program options selected rather than on elapsed time."

Interesting. A fault can be in the requirements. A fault can be in the design. A fault can be in the implementation. Lack of maturity, then, is failures in one of these fault areas. The 2nd sentence in the definition of "Maintainability" (parent of Stability) says:

"Modifications may include corrections, improvements or adaptation of the software to changes in environment, and in requirements and functional specifications."

A modification requires that something exist already; a new "version" of that thing comes about after modifications are made. Stability, then, is strictly tied to how the application behaves in comparison to how it behaved in the last version. Lack of stability is evident, then, when the same test activities are conducted in two different versions of the application, both in the same environment, and failures occur in unexpected areas of the application.

Conclusions
So one might be led to conclude that when testing version 2 of some app, you really could run test A and essentially be testing for both stability and maturity. But like many of these other ISO characteristics, the line is quite fine. The only way I can see it comes back to relate to one thing: what's effected. If you test A after changes were made in A, then you're testing maturity of A. If you test [B-Z] after making changes in A, you're testing stability of the app.

Apple's Characteristics of Great Software

A friend & coworker of mine sent me a link to Apple's Human Interface Guidelines, with the intent to give some insight in to testing the UI applications that I test for my job (thanks Mike). After stepping through a few pages in Part I, it clicked for me that a number of the "Characteristics of Great Software" that Apple lists are the same as some of ISO 9126's quality characteristics. After attending a CVBI meeting this morning where the topic of conversation was "building quality in to software products", I'm really encouraged to see Apple not only list these characteristics that it thinks are important, but back up why they think they're important. One of the things that was discussed implicitly in this morning's meeting was that in order to end up with better quality products, developers (and testers) need to be taught at some point what quality is and how to get there. Apple uses this doc to clearly communicate some ways to raise the quality, not just on Apple applications, but on all applications. Seeing this crossover between the quality characteristics that ISO setup and the ones that Apple setup really gives extra weight to these characteristics. Without having read too much in to the Apple doc yet, it looks like for the characteristics that Apple lists:

• 2 out of 7 (Ease of Use, Attractive Appearance) are related to ISO's Usability
• 1 out of 7 (Reliability) are related to ISO's Reliability
• 1 out of 7 (High Performance) are related to ISO's Efficiency
• 1 out of 7 (Interoperability) are related to ISO's Functionality
• 1 out of 7 (Adaptability) are related to ISO's Portability
• Mobility seems to be a hybrid of ISO's Usability, Functionality, and Efficiency

...interesting that in ISO terms, Apple is most heavily weighted on Usability...

FIFA.com - Football - Test Criteria

It's always refreshing to switch contexts for a subject you're immersed into.  I ran across FIFA's criteria for testing their balls, which is just good to take a look at.  Here's an excerpt from the write-up:

FIFA Quality Concept

The FIFA Quality Concept for Footballs is a test programme for Outdoor, Futsal and Beach Soccer footballs. Manufacturers have the possibility to enter into a licensing agreement for the use of the prestigious 'FIFA APPROVED' and 'FIFA INSPECTED' Quality Marks on footballs which have passed a rigorous testing procedure. As an alternative there is the possibility to use the wording 'IMS International Matchball Standard'. Footballs bearing this designation have passed the same quality requirements as 'FIFA INSPECTED' footballs. The use of this designation is however not subject to a licence fee and any association with FIFA is prohibited.

There are two levels of criteria for the three designations. Footballs applying for the category 'FIFA INSPECTED' or the technically equivalent 'IMS - International Matchball Standard' must pass the following six rigorous laboratory tests:

1. Weight
2. Circumference
3. Sphericity
4. Loss of Air Pressure
5. Water Absorption(replaced with,Balance' test for testing of Futsal balls )
6. Rebound Footballs applying for the higher 'FIFA APPROVED' mark must pass the six tests at an even more demanding level and must undergo an additional test:
7. Shape and Size Retention (Shooting Test)

FIFA.com - Football - Test Criteria

Truth and quality I

The Free On-line Dictionary of Computing defines quality as:

The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs.  Not to be mistaken for "degree of excellence" or "fitness for use" which meet only part of the definition.

Put more simply, quality can be defined as how well something measures up to a standard. But what if the standard sucks? What if the quality of the standard is low? Example: the standard for an app says it only has to run 1 out of 5 times when you double-click its icon. If you test for that and the app meets the standard, you can say it's of good quality, right? ...right? It reminds me of testing apps whose functionality was written without specifications.  When I test the app and find something that doesn't make sense or "doesn't work", Development can say that it's not a bug--it's working as intended.  ...which is totally relative, but totally true.  It's just the intention that was flawed.  ...and the only way to coerce a change is to make some great and sneaky argument, or bring in some sort of exterior, already-defined standard that all of the sudden makes the given functionality look like crap. In the example above, the developer had set his own standard, to which I thought wouldn't be up to standards of the end user.  In the example of the 1/5 standard above, if you're like me, your brain makes a judgment call on the standard itself--probably without you realizing it.  In essence and in context to this post, you're impelled to hold the standard to some implied standard--a standard that's sort of like a code of ethics that pervades a culture.  You know that cutting in line at the Post Office is just a no-no, not because there are any signs there that say so--you just know.  Same idea.  In the case of the developer working without a spec, my code of ethics was just different than his. So in order to have Dev and QA teams be efficient, a general practice is to have both departments agree on what's acceptable and what's not. They define the standard as they see fit for their organization and customers. But the trick is: how do we know when the standards that we've set are good standards?  Where does the standard's standard come from? How do you know right off the bat that the requirement that the app only has to run 1/5th of the time really sucks? Some correlation can be found when looking at the study of Truth.  People have been studying concepts here for a couple thousand years, as opposed to the drop in the bucket we've spent on studying software.  And there are probably just as many theories on SW development and testing practices as there are on Truth.  So without going too in-depth to the topic, I believe there's some interesting discoveries to be made when considering the various theories of Truth.