Comments on: Software Testing Categorization

By: misko

misko — Fri, 14 Aug 2009 17:26:19 +0000

@Arnold,

we said we were working on it, but it never got finished. The person who was working on it left the project.

– Misko

By: Arnold Strasser

Arnold Strasser — Fri, 14 Aug 2009 10:05:32 +0000

You mentioned that testability-explorer provides c++ parsing … I can’t find any information on this topic on the official testability-homepage … are you sure that this is correct?

By: Scott David Daniels

Scott David Daniels — Tue, 21 Jul 2009 16:16:38 +0000

When I was developing automated tests for the microcode on a large departmental printer (almost all functional, some verging on unit), it was a rare test that took under 5 seconds. Things had to come up to temperature and speed, and lots of mechanical action happened on every test. The longest (non-endurance) tests took more like twenty minutes.

By: Sarthak

Sarthak — Thu, 16 Jul 2009 20:45:08 +0000

@Misko,

Shouldnt the rule depend on the size of the application. I agree though that there cant be so many functional tests.

Here is probably one criterion to separate that :

–> Only test the wiring using functional test. Only if the wiring is done in a different way for some use case, then write a separate functional test for that. Otherwise, it should be safe to assume that the wiring works. Basically, cover all the wiring cases using different functional tests.

Does that sound like a feasible solution?

By: misko

misko — Thu, 16 Jul 2009 16:03:11 +0000

@Josh,

These are not theatrics. These tests are fundamentally different. I write them in different way and I write them for different reasons and they find different kinds of bugs.

– Misko

By: Josh

Josh — Thu, 16 Jul 2009 16:00:04 +0000

How much patience you have is subjective, if you modify variables you are performing I/O so I guess he really meant disk I/O.

Why go through all the theatrics of try to categorize tests into “unit”, “functional”,”end-to-end” when it obviously depends on the app and the developers opinion of “fast” vs “slow”. What if you have 4 tiers of test progressively getting slower, what’s the 4th word that needs to be made up for the new tier?

how about just “automated tests” then everyone can just categorize them into tiers based on their specific development lifecycle.

By: misko

misko — Wed, 15 Jul 2009 17:01:23 +0000

@ Sarthak,
your Medium test should have external services mocked out, otherwise you are testing too much and it becomes slow and flaky and it is a large test. I almost feel like there should be a rule that you should not have more than 20 large tests.

By: misko

misko — Wed, 15 Jul 2009 16:59:06 +0000

@Peter,

On a small project you can get away with everything running in 2 secs. On large project you can not. You need to break it to S/M/L and than run L/M separately, as they tend to be slower. I always strive to make things fast and I run as many tests as I can fit in the two second rule.

By: Sarthak

Sarthak — Wed, 15 Jul 2009 16:48:36 +0000

I think there should still be a way for test categorization and running only tests from a particular category.

For functional tests, there might be huge systems and it may take around 5-7 seconds per unit test (we have a system which makes external service calls which calls another service, etc.). For unit tests we mock this behavior, but for functional tests, we still rely on the external service.

So if we have such test cases, I feel we should still be able to only run tests of a particular category. TestNG provides such a feature (never used it, but have read about it), whereas the junit framework doesnt have such features. However, a custom JunitRunner can solve this problem.

The best practice would still be around 2 secs max. That would be cool …

By: Peter J.

Peter J. — Wed, 15 Jul 2009 16:34:05 +0000

Oh, your two-second rule applies to the entire test suite and not just the unit tests? Wow… we run up against a local 30-second limit all the time, and that’s without any “large” tests.