Friday, 15 March 2019

Cucumber is evil

Cucumber. What to say about it.

Cons:

  • Thin end of the wedge/bait and switch:
    • It's nominally free and very easy to start using, however there is no central 'best practice' for maintainability to address the sorts of issues that arise from serious 500+ multi-targeted use.
    • With a tool this flexible, centrally collected best practice is vital, but you have to pay for training that isn't necessarily going to address the various scaling issues, and
    • By the time you reach that point, you've invested so heavily in it, it's hard to switch.
  • Encourages use of Ruby
    • Nobody actually uses the 'duck-typing' feature - everyone really wants a more strictly typed language with type inference like Kotlin because nobody wants run-time failures.
  • No enforced separation of Given When Then step definitions.
    • You have to structure your language to high level concepts and only to work with the GIVEN/WHEN/THEN prefixes, but the baked-in example steps are unhelpfully low-level and work with any prefix.
    • You should use:
      • GIVEN noun-phrase: user "Sam" with black handbag
      • WHEN sentence: "Sam" goes into new airport security tunnel
      • THEN check-sentence: check "Sam" emerges from tunnel
      • AND check "Sam" still has handbag
  • No parallel execution support. Sure you can use 'parallel_test' or 'parallel_calabash', but...
    • that leaves you at the mercy of any of your threads/phones/browsers crashing. There should be a central feed queue of tests, supplying test execution threads and monitoring their health - re-scheduling tests if the thread died, or whatever else.
    • and even with parallel_cucumber offering a central queue, you still have an ugly nightmare of layers.
    • you have to collate reports yourself. Which is a nightmare with the ugly HTML source code that wasn't designed with parallel anything in mind.
  • Internal formatters do not follow the same naming convention as user-supplied ones: -f pretty invokes class Pretty, whereas -f myformatter is an error, not invoking class MyFormatter. This is not clearly documented.
  • Error handling is deeply inconsistent. Goes ten times more when you throw Calabash iOS or Android into the mess.
  • No support for multiple concerns: modern systems test on several levels - basic functionality, user interface consistency, activity tracking, response time, etc. There is no provision for a test to fail in response time or sending activity markers, but to pass in basic functionality: it's an all or nothing test.

    Pros:
    • Non-programmers can use it, so long as the language is clear enough and the system under test simple enough, and you have programmers to write sufficiently unambiguous steps, and the number of steps doesn't get completely out of hand, and you have some sort of auto-completion to make it easy to navigate which of the hundreds of step names you want.
    • Actually, there aren't really any pros.