Politics should be like Software Development

I'm watching the presidential debates.  Each candidate makes claims that the other said "this", then the other retorts with, "No, I said 'that'."  Back and forth.  So how do you know who's telling the truth, who's lying, who's twisting the truth, and who's misunderstood who? How great would it be to have a list of things that each candidate stated at any given time, then when whoever gets elected, we can all see if they've actually done what they said they'd do.  It'd be kinda like SW requirements for some project--you define them, your teams implement them in the software, then other teams make sure they were implemented as stated, then you all get together and talk about (and tell the rest of the company) the good things you did and the bad things you did. Who in the public actually keeps our president accountable?  I sure couldn't tell you what Dubbya claimed he did and if he did it.  It just doesn't seem like there's much accountability for the things these guys say they're going to do. Also, it seems like these guys don't know if the other is talking about strategy, tactics, or post-mortem topics.  One attacks the other's strategies, then the first justifies with their tactics, then the first attacks back talking about the results of everything.  Is it that unclear?  It just seems like candidates thrive (or are told they'll thrive) on attacking the other, and in order to retort "politically", they work around the issues and talk about something else. It's just annoying at how much of a TV show these campaigns have become--candidates can say whatever they want to say as long as it entertains the public.  They work on our emotions, using planned out catch phrases and directed topics.  But once the winner gets in office, who's there to keep them accountable?

Vista auto-reboot-for-updates #4

*sigh*  I guess I'm used to this now, so it's not so surprising, although just as annoying.  I was just setting up a meeting in Outlook when all of the sudden Outlook just started going in and out of focus, then *poof* everything closed and it was rebooting.  Once again, thanks POS Vista for not considering that I was actually working on something before you decided to reboot--reboot without ever having asked me if that's what I wanted.  I actually like how OS X assumes a lot for me, but never does it assume that I want to throw my work to the wind. I really just don't get how this is good behavior.  Judging by the frequency of my blogs on the topic, this happens about once every 3 - 6 weeks since April.  Amazing.  My opinion of MS continues to plummet...

Vista auto-reboot-for-updates #3

Yet again, I'm in the middle of my morning emailing, and all of a sudden, Vista decides to reboot without warning.  How the hell could M$ ever decide this was a good idea?  I really can't think of anything more frustrating than losing work.  And oh ya... it's been "Configuring updates" for the past 10 minutes now... awesome.  *sigh*

RANT: writing 1000 test cases is not that bad

I've heard a number of times over the past few months here at work things like: "...but if we do it like that, that means I'll be writing test cases forever!"  Testers: do you think developers say that when they're assigned to engineer some feature?  If they do, they probably won't be keeping their job long.  When assigned to engineer a feature, a developer goes through some process to figure out what it is they need to do/not do, then write the code that makes the thing do that, one line of code at a time, until the thing does what they think it should do.  What's so craaaazy about testing those lines of code in their different permutations??  Sure, there's merit in testing efficiently, but that's beside the point.  The point is, you do what it takes to get the job done--if that means writing 1000 new test cases in the next month, then that's what it means.

Words (for all developers) To Live By

In coming up with a presentation for work on characteristics of quality, I've done a lot of expanding my brain.  Lots of defining, clarifying, quantifying, and a fair amount of reading.  This morning I took another looks at Apple's ADC guide for Human Interface Guidelines--particularly the part on Characteristics of Great Software.  I'm pretty sure they'd updated it since the last time I'd been there, as I noticed a link on that page to another page called "Know Your Audience".  I followed. First nugget: "It is useful to create scenarios that describe a typical day of a person who uses the type of software product you are designing." Second nugget: "Develop your product with people and their capabilities—not computers and their capabilities—in mind." Third nugget: "It is not your needs or your usage patterns that you are designing for, but those of your (potential) customers." Read it. Enough said.

Vista & Its Own Mind #2

Yet again, Vista presents the pleasant surprise of exiting all my work and rebooting, just to apply updates that it never told me about. I'm appalled for the second time now. Update: And Outlook didn't even save the emails I had written but not yet sent... So freaking annoying.

QC's: Maturity v. Stability

ISO 9126 presents the concepts of "quality characteristics", which is proving to be both helpful and confusing here at work. We've had our longest group process meetings on this topic--just trying to figure out and agree on what each one of these means. One of the big problems is figuring out what some of these are not. We use many of these words loosely in this world of testing/development, thus setting ourselves up for confusion when talking about this stuff.

Technical Malapropisms
Now, no one likes a vocal grammar Nazi around, but sometimes it really is important to talk about the distinctions between different things you talk about on a day to day basis.

One of the many bits of confusion came when talking about "Stability". Stability falls (as a sub-characteristic) under the main characteristic of "Maintainability", which can make sense if you take a minute to ponder. When we started trying to come up with examples of this, however, most of us came up with examples of "Maturity", which falls under "Reliability." It seems that in non-ISO conversations, stability is talked about in a way that encompasses both of these sub-characteristics. I did some brief google-ing to see if we were the only ones with this "problem" and found an abstract on Ecological Modelling: An inverse relationship between stability and maturity in models of aquatic ecosystems. This article on software maturity models suggests that stability is a factory that makes up maturity. ISO says otherwise:

Maturity:
The capability of the software product to avoid failure as a result of faults in the software.

Stability:
The capability of the software product to avoid unexpected effects from modifications of the software.

Maturity deals with "failure as a result of faults"; Stability deals with "unexpected effects from modifications."

Failures v. Unexpected Effects
Maturity deals with failures; Stability deals with unexpected effects. What's the difference? ...in terms of test cases passing I say, nothing. Both are the absence of passing; it doesn't matter what you call it. ...unless this means to say that a product is unstable if changes in module A unexpectedly effect the behavior in module B. One could then interpret "failure" as the inability for module A to meet specification for module A when testing module A; "unexpected effects" are then the inability for module A to meet specification for module A when testing module B.

Faults v. Modifications
Maturity deals with faults in the software; Stability deals with modifications of the software. What's the difference? Key differences become possibly more confusing when you take a look at the definition and notes of the main characteristics for each sub-characteristic. Note 1 for "Reliability" (parent of Maturity) says:

"Wear or aging does not occur in software. Limitations in reliability are due to faults in requirements, design, and implementation. Failures due to these faults depend on the way the software product is used and the program options selected rather than on elapsed time."

Interesting. A fault can be in the requirements. A fault can be in the design. A fault can be in the implementation. Lack of maturity, then, is failures in one of these fault areas. The 2nd sentence in the definition of "Maintainability" (parent of Stability) says:

"Modifications may include corrections, improvements or adaptation of the software to changes in environment, and in requirements and functional specifications."

A modification requires that something exist already; a new "version" of that thing comes about after modifications are made. Stability, then, is strictly tied to how the application behaves in comparison to how it behaved in the last version. Lack of stability is evident, then, when the same test activities are conducted in two different versions of the application, both in the same environment, and failures occur in unexpected areas of the application.

Conclusions
So one might be led to conclude that when testing version 2 of some app, you really could run test A and essentially be testing for both stability and maturity. But like many of these other ISO characteristics, the line is quite fine. The only way I can see it comes back to relate to one thing: what's effected. If you test A after changes were made in A, then you're testing maturity of A. If you test [B-Z] after making changes in A, you're testing stability of the app.

Sneaky

Teach me to test

In seeking to appease my recent desires to get back in to writing code, I flirted with the idea of purchasing a book on Xcode and Objective-C so I could get crackin' on some Mac dev.  With my other recent desire of cutting back on spending, I decided to check out cocoalab.com's free eBook on the topic.  While it covers the uber-basics of getting in to development, it also covers the uber-basics of Xcode, which is what I really need.  It also does this on Leopard--something no books on the market can tout yet (so I've read). So I blew through the first 6 chapters before having to attend my roomie's bday party, and am excited to get back to it ASAP. It just occurred to me though, that while the book talks about debugging in Xcode, it barely talks about testing (well, so far).  And then it occurred to me: most development books that I've ever read don't really make many suggestions about testing at all--much less about how or when to test the code you wrote.  I realize that Test-driven Development is really a suggested technique, but it seems to me that if developers at least followed these concepts, they would find more success.  Thus, if books taught how to test your code in equal doses at they taught how to write code, we might see a reversal in the economy.  :-)

A test case is a scientific procedure

...so treat it as such!  After reading through hundreds of test cases at work in the past couple of weeks, I'm getting extremely frustrated seeing test cases that don't state their point, nor how to know if what happens when running the test was good or bad (pass/fail).  I've seen countless functional test case summaries that state how to do something (instead of what the point of the test is); ...countless test descriptions that state only the object under test (instead of how acting on that object will result in some expected outcome); ...and countless "expected results" sections with sentences that aren't even complete sentences, let alone describe how to determine if the test passed or failed. We all did experiments in junior high, right?  In its simplest form, a test case is just like that--make your hypothesis, plan the steps out you need to "run" in order to try to prove your hypothesis, get your gear together, run through your steps, then get a poster board and some spatter paint, write up your results (plus charts & graphs) and stick them to the board (make sure to not sniff the glue or you'll get in trouble!  jk...). Without even knowing a product, I should at least be able to understand what the point of a test case is when reading it.  Next, the test case must have a goal of achieving something explicit, so that when I run through the steps I won't have to make any new decisions about injecting any new data from outside of my initial plan (the initial plan is a direct result of my hypothesis, so injecting new data may not coincide with what I'm trying to prove in the first place).  Lastly, I should never see multiple possible expected outcomes as the result for a test--if there are 2 possible paths through a certain component, each path needs its own test case. Be explicit with one (and only one) point to the test case.  Please.

Testing is an Engineering discipline

In fact, as I understand it, in many companies Testing is the department that imposes discipline on Development--which shouldn't be.  SW Development Engineers have a set of steps that they go through when developing an application/component/whatever, which is different than the steps that SW Test Engineers go through--but each have their purposes.  It is not Test's job to ensure that SW Development Engineers have gone through their steps properly; it is however, Test's job to ensure that the product of Development's efforts is something suitable for what it was intended for. Next, SW Test Engineers--please please pleeeeeeaaassseee take some accountability for your job title!!  Check out wikipedia's entry for Engineering:

Engineering is the discipline and profession of applying scientific knowledge and utilizing natural laws and physical resources in order to design and implement materials, structures, machines, devices, systems, and processes that realize a desired objective and meet specified criteria. More precisely, engineering can be defined as “the creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.”

Engineering - Wikipedia, the free encyclopedia ...and notice the word "discipline".  Development Engineers can (and should be) hounded for bad formatting and not commenting their code--along those same lines of being disciplined, let the rest of us know what you're trying to accomplish when writing your tests!  Be explicit about what you're trying to test for, show steps for how to test for that, and be concise about what to expect.  Just as a screwy, uncommented piece of code that someone else wrote can drive a Development Engineer nuts, a screwy test can have the same effect on your fellow Test Engineers.  Please, learn to be disciplined in your work--you'll gain respect from your co-workers and probably realize that you missed a few things along the way.  It's not hard--it just takes discipline!

US FDA/CDRH: General Principles of Software Validation; Final Guidance for Industry and FDA Staff

I ran across this link while looking for a specific page in my del.icio.us test links, and while the products that this document is geared towards are a little different than what I deal with on a daily basis, the concepts are great.  I particularly liked this section--especially the first sentence here--in the Verification & Validation section:

Software verification and validation are difficult because a developer cannot test forever, and it is hard to know how much evidence is enough. In large measure, software validation is a matter of developing a "level of confidence" that the device meets all requirements and user expectations for the software automated functions and features of the device. Measures such as defects found in specifications documents, estimates of defects remaining, testing coverage, and other techniques are all used to develop an acceptable level of confidence before shipping the product. The level of confidence, and therefore the level of software validation, verification, and testing effort needed, will vary depending upon the safety risk (hazard) posed by the automated functions of the device.

There's also a great section called "Software is Different From Hardware", which points out some great subtle-but-huge differences between the two. Section 5. Activities and Tasks has some good practical info on planning and test tasks--both Black Box and White Box (although not explicitly so). US FDA/CDRH: General Principles of Software Validation; Final Guidance for Industry and FDA Staff

Vista annoyance

I love how Vista just decides to reboot and apply some updates while I'm in the middle of writing an email.  No warnings, nothing.  Just shuts down as if I said "Shutdown and kill all my apps in the process"...  Nicely done M$.  I love it when my OS gives me the finger.

Apple's Characteristics of Great Software

A friend & coworker of mine sent me a link to Apple's Human Interface Guidelines, with the intent to give some insight in to testing the UI applications that I test for my job (thanks Mike). After stepping through a few pages in Part I, it clicked for me that a number of the "Characteristics of Great Software" that Apple lists are the same as some of ISO 9126's quality characteristics. After attending a CVBI meeting this morning where the topic of conversation was "building quality in to software products", I'm really encouraged to see Apple not only list these characteristics that it thinks are important, but back up why they think they're important. One of the things that was discussed implicitly in this morning's meeting was that in order to end up with better quality products, developers (and testers) need to be taught at some point what quality is and how to get there. Apple uses this doc to clearly communicate some ways to raise the quality, not just on Apple applications, but on all applications. Seeing this crossover between the quality characteristics that ISO setup and the ones that Apple setup really gives extra weight to these characteristics. Without having read too much in to the Apple doc yet, it looks like for the characteristics that Apple lists:

• 2 out of 7 (Ease of Use, Attractive Appearance) are related to ISO's Usability
• 1 out of 7 (Reliability) are related to ISO's Reliability
• 1 out of 7 (High Performance) are related to ISO's Efficiency
• 1 out of 7 (Interoperability) are related to ISO's Functionality
• 1 out of 7 (Adaptability) are related to ISO's Portability
• Mobility seems to be a hybrid of ISO's Usability, Functionality, and Efficiency

...interesting that in ISO terms, Apple is most heavily weighted on Usability...

Quote of the day

"Never mistake motion for action."     -Ernest Hemingway

The BS7925-2 Standard for Software Component Testing

This doc is now 11 years old, but has some really good basic ideas for proceeding through a release cycle.  While on first glance the doc looks to be ginormous, it's not really that bad; the meat of it is really covered in the first 30%. Sections of interest:

• 2.1.1.8 mentions the order in which test activities should be done--it includes Component Test Specification (sandwiched between Planning and Execution)
• 2.3 "Component test specification".  In brief, tests are written here using the techniques that were determined doing Planning (read: tests weren't written during Planning, but the techniques were chosen then)
• 3 "Test Case Design Techniques".  Concisely describes popular and useful techniques for creating test cases for a component.
• 4 "Test Measurement Techniques".  These aren't criteria, but rather methods for helping to figure out progress--and maybe setting criteria based on this info.  They show how to do this for each test case technique type in section 3 (most are pretty obvious, but it's still nice to see on paper).

Grab some coffee and take a gander, here.  Oh ya, and in case you're familiar with this group, this doc was produced by the BCS SIGIST.  (seriously, why?  I guess the real question is why "British Computer Society Specialist Interest Group in Software Testing"?  Reminds me of Monty Python's Life of Brian...)

softwaretestingsucks.com

I'm not exactly sure how this got started, but there are actually a few decent articles/pages of info in here on the basics of testing:

• What is Quality?
• Life Cycle testing
• Testing Types
• Testing Techniques
• Testing Tools
• Certification Programs
• Testing Jokes

Beware--in that last one, the jokes are uber-cheesey, and mostly only remotely entertaining to the Test Geek... Check it out here.

Testing Removable Media

Guys on the UK's The Gadget Show cook with propane, freeze with dry ice, submerge in acidic liquids, drive over, and mortar-fire against a wall a number of types of removable media.  Good stuff.

Negative test cases

I came across this today while doing some Test Analysis research.  It's nothing super profound, but a good checklist for writing negative test cases--if you're in to that sort of thing. [via PragmaticSW]

Thinking Out Loud: Requirements vs. Criteria

What's the difference?  I can't find any good resources for comparison on the web... I'm thinking:

• Requirements are a list of things that must get done for a project, regardless of their outcome.
• Criteria is the assessment of the results of the activities done to meet those requirements.

You set up requirements so you know what you have to do.  Ex.:

MyApp must run/startup on Vista.

...This lets you know that you (at least) have to write tests that cause MyApp to try to run/start on Vista, then run those tests. You set criteria so you know the level of "quality" to which that requirement was met.  In order to determine the level of quality, you look at the results of running the tests.  Ex.:

Criteria = MyApp must run/startup 100% of the time, in at least 1000 attempts, on each edition of Vista.  MyApp ran/started-up on 5 different Vista Editions, a total 1200 times each, and started 99.999% of the time.

...Criteria not met. Hmm... I think I like it....

Mountain Wingsuit

I want to know who tests this equipment...

FIFA.com - Football - Test Criteria

It's always refreshing to switch contexts for a subject you're immersed into.  I ran across FIFA's criteria for testing their balls, which is just good to take a look at.  Here's an excerpt from the write-up:

FIFA Quality Concept

The FIFA Quality Concept for Footballs is a test programme for Outdoor, Futsal and Beach Soccer footballs. Manufacturers have the possibility to enter into a licensing agreement for the use of the prestigious 'FIFA APPROVED' and 'FIFA INSPECTED' Quality Marks on footballs which have passed a rigorous testing procedure. As an alternative there is the possibility to use the wording 'IMS International Matchball Standard'. Footballs bearing this designation have passed the same quality requirements as 'FIFA INSPECTED' footballs. The use of this designation is however not subject to a licence fee and any association with FIFA is prohibited.

There are two levels of criteria for the three designations. Footballs applying for the category 'FIFA INSPECTED' or the technically equivalent 'IMS - International Matchball Standard' must pass the following six rigorous laboratory tests:

1. Weight
2. Circumference
3. Sphericity
4. Loss of Air Pressure
5. Water Absorption(replaced with,Balance' test for testing of Futsal balls )
6. Rebound Footballs applying for the higher 'FIFA APPROVED' mark must pass the six tests at an even more demanding level and must undergo an additional test:
7. Shape and Size Retention (Shooting Test)

FIFA.com - Football - Test Criteria

Music for test documentation

Truth and quality I

The Free On-line Dictionary of Computing defines quality as:

The totality of features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs.  Not to be mistaken for "degree of excellence" or "fitness for use" which meet only part of the definition.

Put more simply, quality can be defined as how well something measures up to a standard. But what if the standard sucks? What if the quality of the standard is low? Example: the standard for an app says it only has to run 1 out of 5 times when you double-click its icon. If you test for that and the app meets the standard, you can say it's of good quality, right? ...right? It reminds me of testing apps whose functionality was written without specifications.  When I test the app and find something that doesn't make sense or "doesn't work", Development can say that it's not a bug--it's working as intended.  ...which is totally relative, but totally true.  It's just the intention that was flawed.  ...and the only way to coerce a change is to make some great and sneaky argument, or bring in some sort of exterior, already-defined standard that all of the sudden makes the given functionality look like crap. In the example above, the developer had set his own standard, to which I thought wouldn't be up to standards of the end user.  In the example of the 1/5 standard above, if you're like me, your brain makes a judgment call on the standard itself--probably without you realizing it.  In essence and in context to this post, you're impelled to hold the standard to some implied standard--a standard that's sort of like a code of ethics that pervades a culture.  You know that cutting in line at the Post Office is just a no-no, not because there are any signs there that say so--you just know.  Same idea.  In the case of the developer working without a spec, my code of ethics was just different than his. So in order to have Dev and QA teams be efficient, a general practice is to have both departments agree on what's acceptable and what's not. They define the standard as they see fit for their organization and customers. But the trick is: how do we know when the standards that we've set are good standards?  Where does the standard's standard come from? How do you know right off the bat that the requirement that the app only has to run 1/5th of the time really sucks? Some correlation can be found when looking at the study of Truth.  People have been studying concepts here for a couple thousand years, as opposed to the drop in the bucket we've spent on studying software.  And there are probably just as many theories on SW development and testing practices as there are on Truth.  So without going too in-depth to the topic, I believe there's some interesting discoveries to be made when considering the various theories of Truth.

Let's get mathy

Pairwise testing is a technique that was introduced to about 6 months ago and have really looked forward to getting the chance to use since then.  I re- ran across a list of tools to help generate lists of test cases to run according to the possible input data (variants).  When the time is right, I wanna try some of these out.

Metrics

(source)

What's a "feature"? Really...

I've been searching all over the interweb for someone software-y to define "feature" and can find a bazillion uses of the word, but no definitions. Strange. Lots of assumptions made in this field...

ISO 9001 vs CMM

An basic high-level comparison between the two industry quality standards: What do ISO 9001 and CMM mean to your organization? (courtesy techrepublic.com)

ISO 9126 and Reliability

I've reading through the ISO 9126 Standard doc (in case you don't have ISO numbers memorized, it's Software engineering - Product quality) and amongst other interesting things thus far, I encountered a section titled "Quality model for external an internal quality" which outlines areas of an application that should be tested; test types, in essence. Of those test types, one was Reliability.  It defined Reliability as:

The capability of the software product to maintain a specified level of performance when used under specified conditions.

...but what really grabbed me was the first note under that definition:

NOTE 1  Wear or ageing does not occur in software.  Limitations in reliability are due to faults in requirements, design, and implementation.  Failures due to these faults depend on the way the software product is used and the program options selected rather than on elapsed time.

This blatantly points the finger to requirements, design, and implementation for when dealing with reliability. I think to some, this might seem to be a "duh" statement, but that whole note is quite enabling as a software test engineer.

Philosophia I

From wikipedia.com:

Philosophy is the discipline concerned with questions of how one should live (ethics); what sorts of things exist and what are their essential natures (metaphysics); what counts as genuine knowledge (epistemology); and what are the correct principles of reasoning (logic).^[1][2] The word is of Greek origin: φιλοσοφία (philosophía), meaning love of wisdom.^[3]

I recently had a conversation with a friend of mine who's teaching a JC class on Logic--on such subject. We found ourselves on said topic, however, by discussing a real-life situation that seemed to violate logic of foundational core moral standards. Things about the situation were utterly perplexing; the path from the reality of yesteryear seemed that it could not lead to today's reality. Yet, as I seem to hear so often lately: "It is what it is." Funny though... I couldn't help but notice that the logic that we tried to apply to my friend's situation was quite similar to that of the logic we try to apply to engineering a piece of software:

We take this timeframe (SDLC) and try to mesh different sorts of tasks in to virtual compartments of the timeframe--sort of like Play Dough in to cookie cutters--to help us manage intricacies that our brains can't really handle all at once. The SDLC is really just a tool that some smart people came up with to help us get stuff done.

Logic is really just some framework that helps us explain certain things (!=all things) away in reality. Also derived by smart people.

Sometimes the Play Dough just doesn't all fit.