Before/After Feature Hook Types?

[clrudolphi]

🤔 What's the problem you're trying to solve?

I see in the recently merged and released #102 that an enumeration has been added that outlines the types of Hooks (Before/After TestRun, BeforeAfter TestCase, and Before/After TestStep).

The .NET implementations (Reqnroll and SpecFlow before it) also support hooks that surround Features.

✨ What's your proposed solution?

Would it be acceptable to add two additional HookType enumeration values to the list (BeforeFeature and AfterFeature)?

⛏ Have you considered any alternatives or workarounds?

No response

📚 Any additional context?

No response

[luke-hill]

I personally wouldn't support this, as a feature is just a set of test cases. You shouldn't ideally be doing something around a feature. Plus the concept of tagging is in place for this - should you want to run something before only some subset of your tests.

If something is unique to specflow/reqnroll for this. It probably should live as something custom for it

[mpkorstanje]

Would it be acceptable to add two additional HookType enumeration values to the list (BeforeFeature and AfterFeature)?

@clrudolphi would that not also require a message to indicate a feature started?

If something is unique to specflow/reqnroll for this. It probably should live as something custom for it

@luke-hill that's a tricky one.

I do think that feature hooks enable some really bad practices. But at the same time, for the ecosystem as a whole I'd rather see that we have a common messages format. The community would really benefit from the network effect it creates.

[clrudolphi]

would that not also require a message to indicate a feature started?

Very likely, yes. There would be the need to associate which Feature the hook is related to (in a stream with multiple features execution intermixed).

[mpkorstanje]

Hey, we had a talk about this in the committers meeting. Cucumber currently doesn't support feature hooks and we've generally considered it a bad practice. But that's neither here nor there for inclusion in messages. A common reporting format gets better when used by many.

And now that the situation for both Cucumber and Reqnroll have stabilized, we are starting the process of working out how and how much Cucumber and Reqnroll will work together and if that includes for example harmonizing the execution model between implementations.

So for now we'll leave this parked. And I would suggest talking to @gasparnagy to get this on his agenda.

[gasparnagy]

@mpkorstanje @luke-hill Definitely we can defer the discussion about the concrete topic, but currently this is a blocking issue for us and some (temporary) guidance would be appreciated. Let me explain the situation.

We are about to have the first version of Reqnroll producing cucumber messages in a few weeks. Reqnroll does have feature hooks (regardless of how we like it) and people use it, so there will be some execution data produced by them. (There is even a before/after scenario block hook that runs before/after all givens/whens/thens btw, but let's stick to the feature hooks for the sake of simplicity.)

With the current messages structure we considered two options:

1. Represent the execution of the before/after feature hooks as before/after test run hooks in the messages structure. This way the execution information is preserved but maybe some downstream usage will be confused to receive multiple before/after test run hooks and also it will be hard to distinguish which one was which.

2. We completely exclude the execution of these hooks from the messages, but in this case there will be no way to diagnose issues related to those hook executions.

Which one we should go for now?

Isn't there a generic "extensibility" envelope type that could be used to preserve some special data that is not (or not yet) supported by the official messages structure? That could be a good entry step for the discussion of canonization (or not) of such extensions. Because otherwise it's more like a chicken-egg problem: we would like to see more usage data before extending the messages structure, but we cannot see this usage data because the structure does not support it.

Or shall we somehow make own envelope types that are specific to Reqnroll? (But the Envelope class is sealed, so this does not seem to be possible either.)

Update: just realized that the Attachment envelope type could be used for this, not very nice, but would work.

[davidjgoss]

We've previously (I can't find examples though) resisted suggestions about having messages be generically extensible.

You could indeed use an Attachment that targets the testRunStartedId to contain some extra metadata, but that's a bit icky.

I wanted to understand this a bit more:

We completely exclude the execution of these hooks from the messages, but in this case there will be no way to diagnose issues related to those hook executions.

My assumption so far has been that inclusion of these feature hooks in messages would be nice, but since no downstream consumers (i.e. formatters) would use them yet anyway, there wasn't a rush. Maybe I misunderstood then - do aspects of Reqnroll depend on the messages e.g. console reporters and such?

[gasparnagy]

My assumption so far has been that inclusion of these feature hooks in messages would be nice, but since no downstream consumers (i.e. formatters) would use them yet anyway, there wasn't a rush.

The html renderer is just one usage example. If we have the data we can imagine to have better usage. I know bigger enterprises who just wait to receive the raw data so that they can build their custom reporting based on that (maybe based on the html formatter, but not necessarily).

We don't need to rush with feature hooks, but I see this is more of a strategic question: there will be always ideas that potentially extend execution data. Existing downstream usages will not do anything to them but experimental ones can be created to test the idea. It the idea is proven to be generally useful, we can integrate it to the standard messages and the existing downstream users will eventually handle it. Like the browser specific css settings that eventually become part of css.

But to flow through this, we need to have a way to include the data related to the idea to the messages so that the experimental usage can be tested. Chicken-egg.