?

Hi, folks. My recent messages are either not being delivered or are slow to be delivered, so I can't tell whether anyone is hearing me. :)

Please do these two things:

1. Make me an owner of this group.

2. Invite [email protected] to this group.

These will help us transfer the group to .

Thanks!
--

J. B. (Joe) Rainsberger :: :: ::

On Fri, Jun 10, 2016 at 6:00 PM, Kim Gr?sman kim.grasman@... [testdrivendevelopment] <testdrivendevelopment@...> wrote:

?

> I test for threading nasties by structuring things so I can explicitly trigger _every_

> possible event sequence order and verify it works.

I've seen several test suites where people force execution order on multithreaded SUTs (using test-only synchronization points), effectively creating a programmatic single-step debugger. It's not pretty.

I have a colleague who tried to go that route, I agree, not pretty.

But it sounds like you're talking more about decoupling testable logic from the threads that drive it, and only test the logic? I like that better.

My approach does assume that you have identified all shared state and appropriately serialized access to it. So it is now merely a matter of verify that your event handlers can handle any sequence of events the outside world may throw at them.

This is made a _lot_ easier by only communicating between threads via message passing. (A matter that requires discipline).

Did I ever mention I hate multithreading? I know I am not smart enough to do it reliably, and all evidence I have seen leads me to have severe doubts about the rest of humanity.

Yes, it is trivial to get a trivial threading program to work, it's pretty easy to get a non-trivial one working 90% of the time.

But as you add 9's there seems to always be one more bug no matter how smart the programmers.

?

Carter's Corollary to the Dunning-Kruger effect : Those people who are indeed smart and are accurately aware of how smart they are, still aren't smart enough to do multi-threading. They produce pretty much the same number of bugs, the bugs are just further down the long tail of probability.

> I have some subsystem tests, which are _not_ part of the unit test suite which I

> leave running for as long as I want to hunt for cases I may have missed.....?

The Google LLVM crowd are doing lots of interesting stuff for C++ here.

Assuming you have a high-level test suite driving the whole program, they provide compiler-supported dynamic analysis to expose threading bugs at runtime, e.g.:

So the combination of low-level 100% stable unit tests, and driving the (instrumented) application at a high-leve, maybe with fuzzed input to expose flaky behavior is really compelling.

I've devised a "build for checkin" and a "background build" split. It would be nice if we could run every test fast enough to give instant feed back... but as we add products.... we can't.

So the "build for checkin" guarantees that all developers on currently active projects will be able to checkout and continue working.

The background build wakes up a farm of servers and runs the tests under valgrind and address sanitizer and for every product.

I haven't got as far as adding fuzzers yet.

?

It's fun to see that C++ has such elaborate infrastructure, I don't know if the same kind of machinery is readily available for other languages/environments. Then again, maybe those languages/environments don't encourage those kinds of bugs :-)

Most of what valgrind / address sanitizer does isn't needed in a saner language like ruby (or D).

And given a chance I would hop our developement to D to keep largish classes of bugs from being writen in the first place.

--

John Carter? ? ? ? ? ? ? ? ? ? ? ? ? ?? Phone : (64)(3) 358 6639
Tait Electronics? ? ? ? ? ? ? ? ? ? ? ? Fax?? : (64)(3) 359 4632
PO Box 1645 Christchurch? ? ? ? ? ? ? ? Email : john.carter@...
New Zealand

---

This Communication is Confidential. We only send and receive email on the
basis of the terms set out at

---

> In fact, it ought to be possible to surrender all control of ordering and parallelism to the runner.

Part of me says, "Any code that depends on the exact performance of the scheduler, the load the number of CPU's, .... is flaky, I want my tests to break and tell me so."

"Any suit of tests that depend on order of execution to work has destroyed the value of defect localization (ie. Which test breaks tells you which part of the code is broken)"

So part of me says you are right.

However watching the behaviour of the herd of cats I work with....flaky sporadic failing tests are a human process level disaster.

Tests that fail for any other reason other than the last checkin,? results in dilution of responsibility and ever growing flakiness.

ie. Unit tests suit must be 100% deterministic and reproducible, and we must use other techniques to hunt threading / ordering issues.

Processes are very nice in the sense that they completely clean up at the OS level. I use valgrind to enforce that....

1) Every Unit Under Test initialises everything it should.

2) Doesn't leak any resources (all _must_ be collected by test process exit point)

I test for threading nasties by structuring things so I can explicitly trigger _every_ possible event sequence order and verify it works.

I have some subsystem tests, which are _not_ part of the unit test suite which I leave running for as long as I want to hunt for cases I may have missed.....

If I find one I encode it as an explicit ordering in a unit test.

On Tue, Jun 7, 2016 at 2:27 PM, Charlie Poole charliepoole@... [testdrivendevelopment] <testdrivendevelopment@...> wrote:

?

Hi John,

I like your list of ideas, but I take a small exception with your comment about running tests on multiple threads.

I agree that it sometimes introduces non-deterministic results, but that's generally due to faults in the tests. Ideally, it should be possible to run any set of unit tests (i.e. microtests) in random order on parallel threads. In fact, it ought to be possible to surrender all control of ordering and parallelism to the runner. I grant that this is a hard goal to achieve.

Charlie

On Mon, Jun 6, 2016 at 6:38 PM, John Carter john.carter@... [testdrivendevelopment] <testdrivendevelopment@...> wrote:

?

Things I do to handle this....

* Benchmark. Every build spits out an order list of how long each test took. Attack the long ones first.

* Parallelism via Processes not threads. Multithreaded unit testing is a recipe for non-determinism and flakiness.

* No sleep rule. Find every test with a "sleep" in it and fix it. Sleeps, delays, timeouts are test smells of the first order.

* Unreal time. Wrap all time services and create a test interface that allows you to "roll time forward" and trigger all events in order.

* Some tests want to be able to run for days to get better state space coverage. Make that a "developer only" option. Run a tiny / fast subset for unit test checkin coverage and let developer run exhaustive test himself when needed.

On Mon, May 23, 2016 at 3:25 PM, Joselito D Moreno joenmoreno@... [testdrivendevelopment] <testdrivendevelopment@...> wrote:

?

Hello,

To those who have experienced test suites that run long, say ~25 minutes, what are some techniques you have used to mitigate it?? We still would like to be able to check in often and run the test suites prior to checking our code in to our repository but these long running tests makes us stretch what we mean with "often".

Joen

__._,

--

John Carter? ? ? ? ? ? ? ? ? ? ? ? ? ?? Phone : (64)(3) 358 6639
Tait Electronics? ? ? ? ? ? ? ? ? ? ? ? Fax?? : (64)(3) 359 4632
PO Box 1645 Christchurch? ? ? ? ? ? ? ? Email : john.carter@...
New Zealand

---

This Communication is Confidential. We only send and receive email on the
basis of the terms set out at

---

--

John Carter? ? ? ? ? ? ? ? ? ? ? ? ? ?? Phone : (64)(3) 358 6639
Tait Electronics? ? ? ? ? ? ? ? ? ? ? ? Fax?? : (64)(3) 359 4632
PO Box 1645 Christchurch? ? ? ? ? ? ? ? Email : john.carter@...
New Zealand

---

This Communication is Confidential. We only send and receive email on the
basis of the terms set out at

---

?

Hi John,

I like your list of ideas, but I take a small exception with your comment about running tests on multiple threads.

I agree that it sometimes introduces non-deterministic results, but that's generally due to faults in the tests. Ideally, it should be possible to run any set of unit tests (i.e. microtests) in random order on parallel threads. In fact, it ought to be possible to surrender all control of ordering and parallelism to the runner. I grant that this is a hard goal to achieve.

Charlie

On Mon, Jun 6, 2016 at 6:38 PM, John Carter john.carter@... [testdrivendevelopment] <testdrivendevelopment@...> wrote:

?

Things I do to handle this....

* Benchmark. Every build spits out an order list of how long each test took. Attack the long ones first.

* Parallelism via Processes not threads. Multithreaded unit testing is a recipe for non-determinism and flakiness.

* No sleep rule. Find every test with a "sleep" in it and fix it. Sleeps, delays, timeouts are test smells of the first order.

* Unreal time. Wrap all time services and create a test interface that allows you to "roll time forward" and trigger all events in order.

* Some tests want to be able to run for days to get better state space coverage. Make that a "developer only" option. Run a tiny / fast subset for unit test checkin coverage and let developer run exhaustive test himself when needed.

On Mon, May 23, 2016 at 3:25 PM, Joselito D Moreno joenmoreno@... [testdrivendevelopment] <testdrivendevelopment@...> wrote:

?

Hello,

To those who have experienced test suites that run long, say ~25 minutes, what are some techniques you have used to mitigate it?? We still would like to be able to check in often and run the test suites prior to checking our code in to our repository but these long running tests makes us stretch what we mean with "often".

Joen

__._,

--

John Carter? ? ? ? ? ? ? ? ? ? ? ? ? ?? Phone : (64)(3) 358 6639
Tait Electronics? ? ? ? ? ? ? ? ? ? ? ? Fax?? : (64)(3) 359 4632
PO Box 1645 Christchurch? ? ? ? ? ? ? ? Email : john.carter@...
New Zealand

---

This Communication is Confidential. We only send and receive email on the
basis of the terms set out at

---

?

Things I do to handle this....

* Benchmark. Every build spits out an order list of how long each test took. Attack the long ones first.

* Parallelism via Processes not threads. Multithreaded unit testing is a recipe for non-determinism and flakiness.

* No sleep rule. Find every test with a "sleep" in it and fix it. Sleeps, delays, timeouts are test smells of the first order.

* Unreal time. Wrap all time services and create a test interface that allows you to "roll time forward" and trigger all events in order.

* Some tests want to be able to run for days to get better state space coverage. Make that a "developer only" option. Run a tiny / fast subset for unit test checkin coverage and let developer run exhaustive test himself when needed.

On Mon, May 23, 2016 at 3:25 PM, Joselito D Moreno joenmoreno@... [testdrivendevelopment] <testdrivendevelopment@...> wrote:

?

Hello,

To those who have experienced test suites that run long, say ~25 minutes, what are some techniques you have used to mitigate it?? We still would like to be able to check in often and run the test suites prior to checking our code in to our repository but these long running tests makes us stretch what we mean with "often".

Joen

__._,

--

John Carter? ? ? ? ? ? ? ? ? ? ? ? ? ?? Phone : (64)(3) 358 6639
Tait Electronics? ? ? ? ? ? ? ? ? ? ? ? Fax?? : (64)(3) 359 4632
PO Box 1645 Christchurch? ? ? ? ? ? ? ? Email : john.carter@...
New Zealand

---

This Communication is Confidential. We only send and receive email on the
basis of the terms set out at

---

?

Hello,

To those who have experienced test suites that run long, say ~25 minutes, what are some techniques you have used to mitigate it?? We still would like to be able to check in often and run the test suites prior to checking our code in to our repository but these long running tests makes us stretch what we mean with "often".

Joen

__._,

?

Hello,

To those who have experienced test suites that run long, say ~25 minutes, what are some techniques you have used to mitigate it?? We still would like to be able to check in often and run the test suites prior to checking our code in to our repository but these long running tests makes us stretch what we mean with "often".

Joen