Report DUT-related errors in fixtures as fail instead of error?

[Slarag]

Pytest has the really great approach that unhandeled exceptions or a call to pytest.fail in a fixture reports the test as error instead of failed.

We're testing devices (DUT, device under test) with a pytest-based framework. And before each test, we use fixtures to ensure that the device is in a certain state. If somehow the DUT fails to enter the correct state due to a DUT-related problem (which will raise a specific exception in the fixture) the test analysts want to see the according test marked as failed and not as error as the root-cause is DUT-related and not environment-related.

So far, I could not find an easy solution for this. Can I do this via a hook function? If yes, which hook function would be the correct one?

I've tried pytest_runtest_makereport as there I have acces to the call object after the test has run (or failed), but there I cannot alter call.result and calling pytest.fail inside this hook will raise an internal error and caues pytest to exit.

Another approach was to just catch the exception in a try/except block, append some error infomation to the request.node object and continue fixture setup. Then we would evaluate that information in the pytest_runtest_call hook and call pytest.fail from there. This works, but the big downside here is that other fixtures that might depend on the failed fixture are still being setup. And as the device is already in a bad state, those fixture might have undefined behavior. In this case I would need an option to avoid setting up other fixtures (without needing to add checks in each and every other fixture).