RFC for testing package design

[joshsmith]

We should discuss what this should look like.

Here are some notes from beam-community/stripity-stripe#124

The test package can distribute a lightweight replica of the Stripe service. That way, testing can be done something like the following:

test "retrieves all plans where the storage limit is greater than 30" do
  Stripe.Test.create(:plan, 10)
  Stripe.Test.create(:plan, 10, metadata: [storage_limit: 10])
  Stripe.Test.create(:plan, 10, metadata: [storage_limit: 35])
  Stripe.Test.create(:plan, 10, metadata: [storage_limit: 30])

  # This is business logic that will call the StripityStripe module
  plans = MyApp.Module.get_plans(:gt, 30) 

  assert Enum.count(plans) == 10
end

The data can be stored on a per-test basis in ETS, allowing users to build up the necessary state and tear it down without needing to know the intricacies of the system.

For example:

defmodule Stripe.CustomerTest do
  # this may be able to be true
  use ExUnit.Case, async: false
  
  # Most of this setup process can be abstracted into the library, but putting
  # it here so that others understand
  setup do
    # Starts a separate process
    #   - That process loads a Plug responder that binds to an available port
    #   - It also seeds an ETS table that it claims ownership of
    #   - The ETS table and Plug responder are bound to this specific test,
    #      so they are isolated to this test and any state manipulation on them
    #      will not affect other tests
    #   - Returns a struct of some form like %Stripe.Test.Server{port: 4515, pid: #PID<0.35.0>}
    {:ok, stripe_server} = Stripe.Test.Server.start()
    
    Application.put_env(:stripity_stripe, :api_base_url, "http://localhost:#{stripe_server.port}")
    
    on_exit fn ->
       #  Cleans up the ETS table and stops the plug responder and owning process
       Stripe.Test.Server.stop(stripe_server)
    end
    
    {:ok, stripe: stripe_server}
  end

  describe "Stripe.Customer.retrieve/1" do
    test "retrieves a customer by ID", %{stripe: stripe} do
      # Uses the Stripe.Test.Customer module to create a new customer in the
      # ETS backed server (essentially, this is a factory method)
      %{id: id} = Stripe.Test.Customer.create(stripe)

      # Makes an actual API request from the high level API through the low level
      # API and through the OS network stack
      {:ok, customer} = Stripe.Customer.retrieve(id)
      
      assert customer.id == id
    end
  end
end

[pdilyard]

How would this work if you were calling the Stripe modules from outside your actually test cases?

Would you need to put something like this in your code?:

if Mix.env == :test do
  Stripe.Test.Customer.create(args)
else
  Stripe.Customer.create(args)
end

Maybe this could be done more simply through some configuration option?

[pdilyard]

Nevermind 😄 I like the approach used in code-corps-api:

# config.exs
config :my_app,
  stripe_api: Stripe

# config.test.exs
config :my_app,
  stripe_api: Stripe.Test

# my_service.ex
@stripe Application.get_env(:my_app, :stripe_api)
...
def charge do
  @stripe.Charges.create(...)
end

[johannesE]

Can we use a @behaviour-based approach for that? See also https://medium.com/@lasseebert/mocks-in-elixir-7204f8cc9d0f

[joshsmith]

@johannesE thanks for the suggestion! Apologies if I'm being dense, but how exactly would @behaviour be used in this case?

[DavidAntaramian]

I thought I provided an updated spec from this one that used GenServer instead of ETS due to the ETS table limit.

In regards to @behaviour, that isn't useful here. The @behaviour methodology (which provides compile-time guarantees for the test double method) mocks the formal integration layer to reduce test surface area to the unit under test.

This testing library is for integration tests, not unit tests. Due to the way that the Stripe API is designed, integration tests can't be performed easily against the live service; furthermore, it's not possible for people to perform them without a Stripe API key and is difficult to perform during CI tests. So while you can't get a full integration test using it, it provides a near alternative.

[jayjun]

I believe Stripe's OpenAPI spec can be used to generate a realistic mock. See beam-community/stripity-stripe#227.