> Building a quickcheck style tester in AS3 drove me nuts at one point before finding out that the conversions between strings and floating point numbers has a bunch of weird issues. There are some obvious ones, but then also things like certain numbers convert differently if you add a trailing zero (so 0.7362856270 is turned into a different number to 0.736285627 and 0.73628562700). So my little "decode(encode(x)) == x" test example broke!
I have in fact had to deal with exactly this problem in Hypothesis internals. The example saving code didn't work correctly with floats in the first edition, because it was serializing them as JSON, which loses some information encoded in the actual float. In the end I serialize floats (which are actually doubles) by converting them to a 64-bit integer with the same bitwise representation first.
The bug aryx is talking about though actually illustrates an amusing feature of Hypothesis, which is that the code in it is just weird enough that it tends to do unexpected things that trigger bugs which have nothing to do with the properties being tested. :-) In this case it was putting unicode objects onto sys.path, which is 100% allowed but causes problems for certain code running on python 2.7 on windows that previously appeared to work.
> I have in fact had to deal with exactly this problem in Hypothesis internals.
Hah, wonderful. It's a great example of how this type of testing can really dig out odd bugs. Once you've tried property based testing, I think you never really trust things quite the same :)
> The bug aryx is talking about though actually illustrates an amusing feature of Hypothesis, which is that the code in it is just weird enough that it tends to do unexpected things that trigger bugs which have nothing to do with the properties being tested
Nice :)
Thanks for releasing this library, it's great to see more work being done in this area. There are usually a few random testing libraries floating about but adding things like minimisation (and I'm still reading through the templating & other new stuff) really makes it stand out.
The API testing example in particular actually comes at a perfect time for me, so it'll definitely be getting some use.
> Once you've tried property based testing, I think you never really trust things quite the same :)
Oh god, tell me about it. So far I've hit 4 pypy bugs, 3 cpython bugs, two pytz bugs (one caused by a cpython bug), and I've learned far more about the edge cases of the language than I ever wanted to know.
Lets just not talk about the number of bugs I've found in Hypothesis itself in the course of testing bits of the system I was sure worked perfectly.
I have in fact had to deal with exactly this problem in Hypothesis internals. The example saving code didn't work correctly with floats in the first edition, because it was serializing them as JSON, which loses some information encoded in the actual float. In the end I serialize floats (which are actually doubles) by converting them to a 64-bit integer with the same bitwise representation first.
The bug aryx is talking about though actually illustrates an amusing feature of Hypothesis, which is that the code in it is just weird enough that it tends to do unexpected things that trigger bugs which have nothing to do with the properties being tested. :-) In this case it was putting unicode objects onto sys.path, which is 100% allowed but causes problems for certain code running on python 2.7 on windows that previously appeared to work.