¿ªÔÆÌåÓý

ctrl + shift + ? for shortcuts
© 2025 Groups.io
  1. Testdrivendevelopment
  2. Messages
Single Messages Topics Expanded Polls

Re: [TDD] Long running test suite

 

On Fri, Jun 10, 2016 at 6:00 PM, Kim Gr?sman kim.grasman@... [testdrivendevelopment] <testdrivendevelopment@...> wrote:
?


> I test for threading nasties by structuring things so I can explicitly trigger _every_
> possible event sequence order and verify it works.

I've seen several test suites where people force execution order on multithreaded SUTs (using test-only synchronization points), effectively creating a programmatic single-step debugger. It's not pretty.

I have a colleague who tried to go that route, I agree, not pretty.

But it sounds like you're talking more about decoupling testable logic from the threads that drive it, and only test the logic? I like that better.

My approach does assume that you have identified all shared state and appropriately serialized access to it. So it is now merely a matter of verify that your event handlers can handle any sequence of events the outside world may throw at them.

This is made a _lot_ easier by only communicating between threads via message passing. (A matter that requires discipline).

Did I ever mention I hate multithreading? I know I am not smart enough to do it reliably, and all evidence I have seen leads me to have severe doubts about the rest of humanity.

Yes, it is trivial to get a trivial threading program to work, it's pretty easy to get a non-trivial one working 90% of the time.

But as you add 9's there seems to always be one more bug no matter how smart the programmers.
?
Carter's Corollary to the Dunning-Kruger effect : Those people who are indeed smart and are accurately aware of how smart they are, still aren't smart enough to do multi-threading. They produce pretty much the same number of bugs, the bugs are just further down the long tail of probability.

> I have some subsystem tests, which are _not_ part of the unit test suite which I
> leave running for as long as I want to hunt for cases I may have missed.....?

The Google LLVM crowd are doing lots of interesting stuff for C++ here.

Assuming you have a high-level test suite driving the whole program, they provide compiler-supported dynamic analysis to expose threading bugs at runtime, e.g.:





So the combination of low-level 100% stable unit tests, and driving the (instrumented) application at a high-leve, maybe with fuzzed input to expose flaky behavior is really compelling.

I've devised a "build for checkin" and a "background build" split. It would be nice if we could run every test fast enough to give instant feed back... but as we add products.... we can't.

So the "build for checkin" guarantees that all developers on currently active projects will be able to checkout and continue working.

The background build wakes up a farm of servers and runs the tests under valgrind and address sanitizer and for every product.

I haven't got as far as adding fuzzers yet.
?

It's fun to see that C++ has such elaborate infrastructure, I don't know if the same kind of machinery is readily available for other languages/environments. Then again, maybe those languages/environments don't encourage those kinds of bugs :-)

Most of what valgrind / address sanitizer does isn't needed in a saner language like ruby (or D).


And given a chance I would hop our developement to D to keep largish classes of bugs from being writen in the first place.


--
John Carter? ? ? ? ? ? ? ? ? ? ? ? ? ?? Phone : (64)(3) 358 6639
Tait Electronics? ? ? ? ? ? ? ? ? ? ? ? Fax?? : (64)(3) 359 4632
PO Box 1645 Christchurch? ? ? ? ? ? ? ? Email : john.carter@...
New Zealand


This Communication is Confidential. We only send and receive email on the
basis of the terms set out at

Join [email protected] to automatically receive all group messages.