implement event protocol (#879)

Author: charlierudolph

CHANGELOG.md

@@ -1,12 +1,15 @@
 ### [master (unreleased)](https://github.com/cucumber/cucumber-js/compare/v2.3.1...master)
 
 #### BREAKING CHANGES
-* `pretty` formatter has been removed. All errors are now reported in a `pretty` format instead. Use the `progress-bar` or `progress` formats instead.
+* `pretty` formatter has been removed. All errors are now reported in a `pretty` format instead. The `progress` formatter is now the default.
+* Major changes to custom formatters API due to rewrite in support of event protocol. Please see the updated documentation.
+* Remove `registerHandler` and `registerListener`. Use `BeforeAll` / `AfterAll` for setup  code. Use the event protocol formatter if used for reporting. Please open an issue if you have another use case.
 
 #### New Features
 
 * Add `--i18n-languages` and `--i18n-keywords <ISO 639-1>` CLI options
-* Add `BeforeAll` / `AfterAll` hooks to replace uses of `registerHandler`
+* Add `BeforeAll` / `AfterAll` hooks for suite level setup / teardown
+* Add event protocol formatter
 
 #### Bug Fixes
 

CONTRIBUTING.md

@@ -40,13 +40,12 @@ The browser example is only updated when releasing a new version.
     │
     ├── models                # data structures
     │
-    ├── runtime               # run features / scenarios / steps, trigger events
+    ├── runtime               # run test cases, emits the event protocol
     │
     └── support_code_library  # load hooks / step definitions
 ```
 
-The runtime triggers events similar to an [EventEmitter](https://nodejs.org/api/events.html#events_class_eventemitter)
-but waits for the listener to finish (the same style used in hook and step definitions).
+The runtime emits events with an [EventEmitter](https://nodejs.org/api/events.html#events_class_eventemitter)
 
 ### Coding style
 

docs/cli.md

@@ -12,8 +12,8 @@ The executable is also aliased as `cucumber-js` and `cucumberjs`.
 The latter is causing the operating system to invoke JScript instead of Node.js,
 because of the file extension.
 
-**Note on global installs:** Cucumber does not work when installed globally because cucumber 
-needs to be required in your support files and globally installed modules cannot be required. 
+**Note on global installs:** Cucumber does not work when installed globally because cucumber
+needs to be required in your support files and globally installed modules cannot be required.
 
 ## Running specific features
 
@@ -47,12 +47,12 @@ This option may be used multiple times in order to output different formats to d
 If multiple formats are specified with the same output, only the last is used.
 
 Built-in formatters
+* event-protocol - prints the event protocl. See [current docs](https://docs.cucumber.io/event-protocol/) and the [proposed updates](https://github.com/cucumber/cucumber/pull/172) which were implemented.
 * json - prints the feature as JSON
-* pretty - prints the feature as is (default)
-* progress - prints one character per scenario
+* progress - prints one character per scenario (default)
 * progress-bar - prints a progress bar and outputs errors/warnings along the way
 * rerun - prints the paths of any non passing scenarios ([example](/features/rerun_formatter.feature))
-  * suggested use: add the rerun formatter to your default profile and the output file to your `.gitignore`. 
+  * suggested use: add the rerun formatter to your default profile and the output file to your `.gitignore`.
   * After a failed run, remove any arguments used for selecting feature files and add the rerun file in order to rerun just failed scenarios. The rerun file must start with `@` sign in order for cucumber to parse it as a rerun file instead of a feature file.
   * Use with `--fail-fast` to rerun the failure and the remaining features.
 * snippets - prints just the code snippets for undefined steps

docs/custom_formatters.md

@@ -1,14 +1,16 @@
 # Custom Formatters
 
-Custom formatters should be a javascript class. The constructor will be an options object  which is the options defined with `--format-options` and the following:
+Custom formatters should be a javascript class. The constructor will be an options object which is the options defined with `--format-options` and the following:
 
 * `colorFns`: a series of helper functions for outputting colors. See [here](/src/formatter/get_color_fns.js). Respects `colorsEnabled` option
 * `cwd`: the current working directory
+* `eventBroadcaster`: an event emitter that emits the event protocol (which is still being defined). See [current docs](https://docs.cucumber.io/event-protocol/) and the [proposed updates](https://github.com/cucumber/cucumber/pull/172) which were implemented.
+* `eventDataCollector`: an instance of [EventDataCollector](/src/formatter/helpers/event_data_collector.js) which handles the complexity of grouping the data for related events
 * `log`: function which will write the passed string to the the designated stream (stdout or the to file the formatter output is being redirected to).
-* `snippetBuilder`: an object with a `build` method that should be called with a [step](/src/models/step.js) to get the snippet for an undefined step
+* `snippetBuilder`: an object with a `build` method that should be called with `{keywordType, pickleStep}`. The `pickleStep` can be retrieved with the `eventDataCollector` while the `keywordType` is complex to compute (see the [SnippetsFormatter](/src/formatter/snippets_formatter.js) for an example).
 * `stream`: the underlying stream the formatter is writing to. `log` is a shortcut for writing to it.
 
-Custom formatters should then define instance methods in order to output something during a particular event. For example it should define a `handleBeforeFeatures` function in order to output something as cucumber is starting.
+The constructor of custom formatters should add listeners to the `eventBroadcaster`.
 
 See a couple examples [here](/features/custom_formatter.feature) and the built in formatters [here](/src/formatter)
 

docs/support_files/api_reference.md

@@ -112,17 +112,6 @@ Alias of `defineStep`.
 
 ---
 
-#### `registerHandler(event[, options], fn)`
-
-* `event`: One of the supported event names [listed here](./event_handlers.md).
-* `options`: An object with the following keys:
-  - `timeout`: A step-specific timeout, to override the default timeout.
-* `fn`: A function, defined as follows:
-  - The first argument is the object as defined [here](./event_handlers.md).
-  - When using the asynchronous callback interface, have one final argument for the callback function.
-
----
-
 #### `setDefaultTimeout(milliseconds)`
 
 Set the default timeout for asynchronous steps. Defaults to `5000` milliseconds.

docs/support_files/event_handlers.md

@@ -29,8 +29,8 @@ Feature: Attachments
       """
     When I run cucumber.js
     Then the "Before" hook has the attachment
-      | DATA     | MIME TYPE |
-      | iVBORw== | image/png |
+      | DATA     | MEDIA ENCODING | MEDIA TYPE |
+      | iVBORw== | base64         | image/png  |
 
   Scenario: Attach a stream (callback)
     Given a file named "features/support/hooks.js" with:
@@ -50,8 +50,8 @@ Feature: Attachments
       """
     When I run cucumber.js
     Then the "Before" hook has the attachment
-      | DATA     | MIME TYPE |
-      | iVBORw== | image/png |
+      | DATA     | MEDIA ENCODING | MEDIA TYPE |
+      | iVBORw== | base64         | image/png  |
 
     Scenario: Attach a stream (promise)
       Given a file named "features/support/hooks.js" with:
@@ -72,8 +72,8 @@ Feature: Attachments
         """
       When I run cucumber.js
       Then the "Before" hook has the attachment
-        | DATA     | MIME TYPE |
-        | iVBORw== | image/png |
+        | DATA     | MEDIA ENCODING | MEDIA TYPE |
+        | iVBORw== | base64         | image/png  |
 
   Scenario: Attach from a before hook
     Given a file named "features/support/hooks.js" with:
@@ -88,7 +88,7 @@ Feature: Attachments
       """
     When I run cucumber.js
     Then the "Before" hook has the attachment
-      | DATA | MIME TYPE  |
+      | DATA | MEDIA TYPE |
       | text | text/plain |
 
   Scenario: Attach from an after hook
@@ -104,7 +104,7 @@ Feature: Attachments
       """
     When I run cucumber.js
     Then the "After" hook has the attachment
-      | DATA | MIME TYPE  |
+      | DATA | MEDIA TYPE |
       | text | text/plain |
 
   Scenario: Attach from a step definition
@@ -120,5 +120,5 @@ Feature: Attachments
       """
     When I run cucumber.js
     Then the step "a step" has the attachment
-      | DATA | MIME TYPE  |
+      | DATA | MEDIA TYPE |
       | text | text/plain |

features/attachments.feature

@@ -14,21 +14,35 @@ Feature: custom formatter
       import {Formatter} from 'cucumber'
 
       class SimpleFormatter extends Formatter {
-        handleBeforeScenario(scenario) {
-          this.log(scenario.feature.name + ' / ' + scenario.name + '\n');
+        constructor(options) {
+          super(options)
+          options.eventBroadcaster
+            .on('test-case-started', ::this.logTestCaseName)
+            .on('test-step-finished', ::this.logTestStep)
+            .on('test-case-finished', ::this.logSeparator)
+            .on('test-run-finished', ::this.logTestRunResult)
         }
 
-        handleStepResult(stepResult) {
-          const {status, step} = stepResult;
-          this.log('  ' + step.keyword + step.name + ' - ' + status + '\n');
+        logTestCaseName({sourceLocation}) {
+          const {gherkinDocument, pickle} = this.eventDataCollector.getTestCaseData(sourceLocation)
+          this.log(gherkinDocument.feature.name + ' / ' + pickle.name + '\n');
         }
 
-        handleAfterScenario() {
+        logTestStep({testCase, index, result}) {
+          const {gherkinKeyword, pickleStep, testStep} = this.eventDataCollector.getTestStepData({testCase, index})
+          if (pickleStep) {
+            this.log('  ' + gherkinKeyword + pickleStep.text + ' - ' + result.status + '\n');
+          } else {
+            this.log('  Hook - ' + result.status + '\n');
+          }
+        }
+
+        logSeparator() {
           this.log('\n');
         }
 
-        handleFeaturesResult(featuresResult) {
-          this.log(featuresResult.success ? 'SUCCESS' : 'FAILURE');
+        logTestRunResult({result}) {
+          this.log(result.success ? 'SUCCESS' : 'FAILURE');
         }
       }
 
@@ -56,16 +70,29 @@ Feature: custom formatter
       import {SummaryFormatter} from 'cucumber'
 
       class SimpleFormatter extends SummaryFormatter {
-        handleBeforeScenario(scenario) {
-          this.log(scenario.feature.name + ' / ' + scenario.name + '\n');
+        constructor(options) {
+          super(options)
+          options.eventBroadcaster
+            .on('test-case-started', ::this.logTestCaseName)
+            .on('test-step-finished', ::this.logTestStep)
+            .on('test-case-finished', ::this.logSeparator)
+        }
+
+        logTestCaseName({sourceLocation}) {
+          const {gherkinDocument, pickle} = this.eventDataCollector.getTestCaseData(sourceLocation)
+          this.log(gherkinDocument.feature.name + ' / ' + pickle.name + '\n');
         }
 
-        handleStepResult(stepResult) {
-          const {status, step} = stepResult;
-          this.log('  ' + step.keyword + step.name + ' - ' + status + '\n');
+        logTestStep({testCase, index, result}) {
+          const {gherkinKeyword, pickleStep, testStep} = this.eventDataCollector.getTestStepData({testCase, index})
+          if (pickleStep) {
+            this.log('  ' + gherkinKeyword + pickleStep.text + ' - ' + result.status + '\n');
+          } else {
+            this.log('  Hook - ' + result.status + '\n');
+          }
         }
 
-        handleAfterScenario() {
+        logSeparator() {
           this.log('\n');
         }
       }

features/custom_formatter.feature

@@ -0,0 +1,61 @@
+Feature: Event Protocol Formatter
+
+  Scenario: gherkin error
+    Given a file named "features/a.feature" with:
+      """
+      Feature: a feature
+        Scenario: a scenario
+          Given a step
+          Examples:
+            | a | b |
+      """
+    When I run cucumber.js with `--tags @a -f event-protocol`
+    Then the output matches the fixture "event_protocol_formatter/gherkin-error.ndjson"
+    And it fails
+
+  Scenario: rejected pickle
+    Given a file named "features/a.feature" with:
+      """
+      Feature: a feature
+        Scenario: a scenario
+          Given a step
+      """
+    When I run cucumber.js with `--tags @a -f event-protocol`
+    Then the output matches the fixture "event_protocol_formatter/rejected-pickle.ndjson"
+
+  Scenario: passed
+    Given a file named "features/a.feature" with:
+      """
+      Feature: a feature
+        Scenario: a scenario
+          Given a step
+      """
+    Given a file named "features/step_definitions/steps.js" with:
+      """
+      import {defineSupportCode} from 'cucumber'
+
+      defineSupportCode(({Given}) => {
+        Given(/^a step$/, function() {})
+      })
+      """
+    When I run cucumber.js with `-f event-protocol`
+    Then the output matches the fixture "event_protocol_formatter/passed.ndjson"
+
+  Scenario: failed
+    Given a file named "features/a.feature" with:
+      """
+      Feature: a feature
+        Scenario: a scenario
+          Given a step
+      """
+    Given a file named "features/step_definitions/steps.js" with:
+      """
+      import {defineSupportCode} from 'cucumber'
+
+      defineSupportCode(({Given}) => {
+        Given(/^a step$/, function(callback) { callback(new Error('my error')) })
+      })
+      """
+    When I run cucumber.js with `-f event-protocol`
+    Then the output matches the fixture "event_protocol_formatter/failed.ndjson"
+    And it fails

features/event_protocol_formatter.feature

@@ -22,5 +22,5 @@ Feature: Fail fast
       })
       """
     When I run cucumber.js with `--fail-fast`
-    Then it runs the scenario "Failing"
+    Then the step "a passing step" has status "skipped"
     And it fails

features/fail_fast.feature

@@ -0,0 +1,11 @@
+{"type":"source","uri":"features/a.feature","data":"Feature: a feature\n  Scenario: a scenario\n    Given a step","media":{"encoding":"utf-8","type":"text/vnd.cucumber.gherkin+plain"}}
+{"type":"gherkin-document","uri":"features/a.feature","document":{"type":"GherkinDocument","feature":{"type":"Feature","tags":[],"location":{"line":1,"column":1},"language":"en","keyword":"Feature","name":"a feature","children":[{"type":"Scenario","tags":[],"location":{"line":2,"column":3},"keyword":"Scenario","name":"a scenario","steps":[{"type":"Step","location":{"line":3,"column":5},"keyword":"Given ","text":"a step"}]}]},"comments":[]}}
+{"type":"pickle","uri":"features/a.feature","pickle":{"tags":[],"name":"a scenario","language":"en","locations":[{"line":2,"column":3}],"steps":[{"text":"a step","arguments":[],"locations":[{"line":3,"column":11}]}]}}
+{"type":"pickle-accepted","pickle":{"tags":[],"name":"a scenario","language":"en","locations":[{"line":2,"column":3}],"steps":[{"text":"a step","arguments":[],"locations":[{"line":3,"column":11}]}]},"uri":"features/a.feature"}
+{"type":"test-run-started"}
+{"type":"test-case-prepared","steps":[{"sourceLocation":{"uri":"features/a.feature","line":3},"actionLocation":{"uri":"features/step_definitions/steps.js","line":4}}],"sourceLocation":{"uri":"features/a.feature","line":2}}
+{"type":"test-case-started","sourceLocation":{"uri":"features/a.feature","line":2}}
+{"type":"test-step-started","index":0,"testCase":{"sourceLocation":{"uri":"features/a.feature","line":2}}}
+{"type":"test-step-finished","index":0,"result":{"duration":0,"exception":"Error: my error\n    at World.<anonymous> (features/step_definitions/steps.js:4:51)","status":"failed"},"testCase":{"sourceLocation":{"uri":"features/a.feature","line":2}}}
+{"type":"test-case-finished","result":{"duration":0,"status":"failed"},"sourceLocation":{"uri":"features/a.feature","line":2}}
+{"type":"test-run-finished","result":{"duration":0,"success":false}}

features/fixtures/event_protocol_formatter/failed.ndjson

@@ -0,0 +1,2 @@
+{"type":"source","uri":"features/a.feature","data":"Feature: a feature\n  Scenario: a scenario\n    Given a step\n    Examples:\n      | a | b |","media":{"encoding":"utf-8","type":"text/vnd.cucumber.gherkin+plain"}}
+{"type":"attachment","source":{"uri":"features/a.feature","start":{"line":4,"column":5}},"data":"(4:5): expected: #EOF, #TableRow, #DocStringSeparator, #StepLine, #TagLine, #ScenarioLine, #ScenarioOutlineLine, #Comment, #Empty, got 'Examples:'","media":{"encoding":"utf-8","type":"text/vnd.cucumber.stacktrace+plain"}}

features/fixtures/event_protocol_formatter/gherkin-error.ndjson

@@ -0,0 +1,11 @@
+{"type":"source","uri":"features/a.feature","data":"Feature: a feature\n  Scenario: a scenario\n    Given a step","media":{"encoding":"utf-8","type":"text/vnd.cucumber.gherkin+plain"}}
+{"type":"gherkin-document","uri":"features/a.feature","document":{"type":"GherkinDocument","feature":{"type":"Feature","tags":[],"location":{"line":1,"column":1},"language":"en","keyword":"Feature","name":"a feature","children":[{"type":"Scenario","tags":[],"location":{"line":2,"column":3},"keyword":"Scenario","name":"a scenario","steps":[{"type":"Step","location":{"line":3,"column":5},"keyword":"Given ","text":"a step"}]}]},"comments":[]}}
+{"type":"pickle","uri":"features/a.feature","pickle":{"tags":[],"name":"a scenario","language":"en","locations":[{"line":2,"column":3}],"steps":[{"text":"a step","arguments":[],"locations":[{"line":3,"column":11}]}]}}
+{"type":"pickle-accepted","pickle":{"tags":[],"name":"a scenario","language":"en","locations":[{"line":2,"column":3}],"steps":[{"text":"a step","arguments":[],"locations":[{"line":3,"column":11}]}]},"uri":"features/a.feature"}
+{"type":"test-run-started"}
+{"type":"test-case-prepared","steps":[{"sourceLocation":{"uri":"features/a.feature","line":3},"actionLocation":{"uri":"features/step_definitions/steps.js","line":4}}],"sourceLocation":{"uri":"features/a.feature","line":2}}
+{"type":"test-case-started","sourceLocation":{"uri":"features/a.feature","line":2}}
+{"type":"test-step-started","index":0,"testCase":{"sourceLocation":{"uri":"features/a.feature","line":2}}}
+{"type":"test-step-finished","index":0,"result":{"duration":0,"status":"passed"},"testCase":{"sourceLocation":{"uri":"features/a.feature","line":2}}}
+{"type":"test-case-finished","result":{"duration":0,"status":"passed"},"sourceLocation":{"uri":"features/a.feature","line":2}}
+{"type":"test-run-finished","result":{"duration":0,"success":true}}