Possibility to give forall a cleanup hook?

[turion]

Often, I encounter the following bug in my tests:

property "test database", [], ctx do
  forall row_to_be_inserted = %schema{} <- gen_row() do
    Ecto.insert(row_to_be_inserted, ctx.repo)
    match([%schema{}], Ecto.all(schema))
  end
end

The bug is that upon multiple iterations of forall, the database is polluted with test data and the property fails because of side effects introduced in earlier iterations.

One possibility is to clean up manually like this:

property "test database", [], ctx do
  forall row_to_be_inserted = %schema{} <- gen_row() do
    Ecto.delete_all(schema)
    Ecto.insert(row_to_be_inserted, ctx.repo)
    match([%schema{}], Ecto.all(schema))
  end
end

But this is tedious when doing it in many similar tests, especially if the database setup changes. It would be nice if there was a hook similar to ExUnit's on_exit which is called at the end or at the beginning of every forall in every property, and can be configured per module. This hook could then do the cleanup in one central place.

[LostKobrakai]

For ecto checking out the sandbox manually seems to work:

use ExUnit.Case

# In the property
forall cmds <- commands(__MODULE__) do
      :ok = Ecto.Adapters.SQL.Sandbox.checkout(MyApp.Repo)
      {history, state, result} = run_commands(__MODULE__, cmds)
      :ok = Ecto.Adapters.SQL.Sandbox.checkin(MyApp.Repo)

      result == :ok
end

[alfert]

That's a nice example how to make an intrinsic stateful situation quasi stateless. And that approach immediately translates into similar approaches outside of stateful testing (cmds <- commands(__MODULE__) suggests that).

[evnu]

Note that checkin/1 only happens if run_commands/2 does not crash. I don't know the specifics within PropEr w.r.t. run_commands/2 (there could be a catch-all to never crash the function), but this also applies to non-statem properties. A clean-up hook would handle this transparently. A user can usually also handle this in a property using this pattern:

forall x <- nat() do
  try do
    true = x < 0 # this crashes
  after
    cleanup() # this should be called, no matter if we crashed or not.
  end
end