For the last year, the Integrations Team has been busy working rebuilding our legacy integrations from scratch in Ruby on Rails. We launched with an integration to send Tithe.ly Giving data to QuickBooks Online (QBO), followed by Tithe.ly ChMS, and Raiser’s Edge. We are about to release integrations to FellowshipOne, Shelby Arena, Breeze ChMS, Minitry Platform as well as planned integrations for a handful of others.

The (Good) Problem

It quickly became clear that we were duplicating code as we were creating a separate Rails application for each integration, interacting with the same backends, except for the second party (2ndP), e.g. QBO or Tithe.ly ChMS. To reduce duplication, we needed pull in the common functionality into a single repository and reference everywhere.

Some options were to

1. Create a git submodule and just pull in those files we needed.
2. Create a ruby gem that would provide the properly namespaced records.
3. Create a Rails engine to encapsulate the common application functionality.

We chose option #3 because we knew we wanted to house duplicate model, views, routes, helpers, and specs. We could drop this into a Rails application and have all the functionality available to the app by default.

Challenges

This however, was a bit more challenging for a myriad of reasons.

1. Engine development isn’t a straight forward process.
2. Some of the common functionality required knowledge about the 2ndP data and models.
3. The application need to override some of the default functionality in the engine.
4. The engine’s isolate_namespace was good in practice but prevented the app from seeing all engine functionality.
5. Use of ThoughBot’s FactoryBot created difficult dependencies
6. Setting up Rails Previews and overriding email functionality required some fiddling.

We were able overcome each of these obstacles and I’ll share how we did it.

Engine Development Process

Since a rails engine isn’t a full Rails::Application, it requires a dummy app that provides additional mechanisms for testing, predominately a database connection schema. We decided to use one of our main app repo as a submodule in the spec/dummy folder. This allowed the engine and interdependent app to be updated together. Here’s how it looks.

Propping Up the Dummy

The Rails Guide for Engines is a pretty good tutorial to get an engine setup. I won’t duplicate that here to highlight where we deviated from the norm.

As previously stated, we have an integration application repo as a submodule in the spec/dummy as we use rspec for testing. We then have a commit that contains all the changes that we need to transform a standard Rails Application to a dummy application. Changes required for this are contained Appendix A: Dummy App Changes.

Making Changes

This is where it gets interesting. Since the engine and the dummy app are interdependent, we need to update them in sync and keep it straight in both repositories. Here is the process we landed on.

# Initial setup
cd spec
git submodule add -b dummy git@github.com:tithely/rails-app.git ./spec/dummy

# Refresh dummy
git submodule sync
git submodule update --init --recursive

# Create a new feature branch for the engine
git checkout -b IN-<issue>/<feature-name> # we use JIRA to handle tickets

# Checkout a temporary dummy app with latest master changes
cd spec/dummy
git checkout -b master
git pull
git checkout -b IN-<issue>/dummy
git cherry-pick <dummy-commit> # usually the last commit on a previous dummy branch

###########################################################################
# Make necessary changes to gem and spec/dummy required for your feature. #
###########################################################################

# Be sure to increment your engine's version and `bundle install` following Semantic Versioning https://semver.org/
vi lib/integrations/version.rb
bundle install
rake install #install the engine gem locally

# If you've moved classes to a different namespace, you'll likely need to clean up the bootsnap directory.
rm -rf spec/dummy/tmp/cache/bootsnap-*

# Add dummy app changes that to be committed to master (without dummy-commit).
# Notice, we don't commit these modifications to the dummy branch. It'll get those when we rebase below.
cd spec/dummy
git checkout master
git checkout IN-<issue>/<feature-name>
vi Gemfile # update to use new engine gem version
bundle install
bundle exec rspec # ensure test pass one last time
git add -A
git commit -a
git push origin IN-<issue>/<feature-name>

# Rebase changes to dummy branch
git checkout IN-<issue>/dummy
git rebase IN-<issue>/<feature-name> # resolve conflicts to use incoming changes
bundle exec rspec # because I really like the green!
git push --set-upstream -f IN-<issue>/dummy

# Commit changes to the engine gem
cd ..
git add -A
git commit
git push origin IN-<issue>/<branch-name>

You should now have

1. an engine gem that points to an app’s dummy branch
2. an app that has new dummy branch and a new feature branch incrementing the engine version
3. successful tests in the gem and the app on all branches

Firing the Engines

We’ve got a working engine! Hooray! Now we can release it and get it deployed. Since this is a private gem, we don’t want just anyone to have access to it. We’ve started to use GitHub Packages to house our gems. We build them with GitHub Actions automatically when we create a new release.

This requires a custom gem source in our Gemfile as well as a valid GitHub personal access token. Something a bit like this:

source "https://rubygems.pkg.github.com/tithely" do
  gem "integrations", "3.8.3""
end

Solutions

As promised, there were some sticky wickets in getting this engine fully launched. I’ll detail the solutions we came up with for each of the challenges we encountered.

Exposing 2ndP Models to the Engine

We have a series of jobs that are executed in the background to send transactions to the second party application. This is essentially the same for each application except for the last part where we push to the second party application. Rails initializers makes it pretty simple to expose the classes needed by the calling code in the engine.

For instance, we have an initializer in each integration that looks something like this with the same keys.

Rails.application.config.integrations = {
  second_party_deposit_loader_class: SecondParty::Loader::Deposit,
  second_party_transaction_loader_class: SecondParty::Loader::Transaction,
}

We make reference to those classes where we need to with

Rails.application.config.integrations[:second_party_deposit_loader_class]

Overriding Engine Default Behavior

We had cases where we need to slightly alter the engine’s default behavior, usually to add additional fields to models or functionality. This was easily done by reopening the engine class and defining our new methods. However, in development, with Rails autoloading, it required we load the engine class before reopening the class.

Ideally, we would have loved to use the require_dependency as recommended by the rails engine guide but we found this to not work in development after the initial request. Using load is a bit of a hammer but it works and it performant in environments with class caching enabled.

A reopened class with added functionality looks something like this

load Integrations::Engine.root.join("app", "models", "integrations", "transaction.rb").to_s

module Integrations
    class Transaction
        def new_method
            puts "Do stuff!"
        end
    end
end

Make sure you have a test on all new functionality to ensure it still works with an engine update. See Appendix B: Engine’s Rails Helper for our full spec/rails_helper.rb used in the engine.

Handling Isolated Namespaces

We liked the idea of isolate_namespace but prevented some functionality we initially thought would be available to us, namely routes and url and path helpers. Our final iteration retained isolate_namespace and required updates to our views.

Routes to Root

Our engine routes looks like

Integrations::Engine.routes.draw do
    resource :sessions
end

and the application routes has the engine mounted at the root. This provides pages like /transactions and /deposits tied directly to the domain.

Rails.application.routes.draw do
    resource :dashboard
    mount Integrations::Engine => "/"
end

We liked this because we knew we wouldn’t have a similar named controller in our apps that would conflict. If we did need to add functionality, we’d likely make it available to all our integrations.

URL Helpers

Unfortunately, the Path and URL Helpers referenced in the engine (e.g. the main layout shared by all integrations), requires referencing the routes by namespace.

For example, the routes provided by the engine are prefixed by the name of the engine like integrations.transactions_path. The routes provided by the app as main_app.dashboard_path

Working with Engine Factories

FactoryBot makes it pretty easy to append a set of factory files for loading. This needed to happen after all the dependencies are loaded, including the engine gem, so it it required a custom engine initializer.

module Integrations
    class Engine < ::Rails::Engine
        initializer "integrations.factories", after: "factory_bot.set_factory_paths" do
            FactoryBot.definition_file_paths << File.expand_path("../../../spec/factories", __FILE__) if defined?(FactoryBot)
        end
    end
end

Using Engine Mailer Previews

Moving common email functionality into the engine was one of the trickiest problems we encountered. Rails Previews are only for development purposes but it gives a great way to visualize what an email would see

For reference, here is one of our preview classes. Notice the use of FactoryBot used for setup.

# Preview all emails at http://localhost:3000/rails/mailers
class SyncPreview < ActionMailer::Preview
  def completed_with_transactions
    @sync_log = FactoryBot.create(:sync_log, sync_type: Integrations::Settings.sync_types[:transaction])
    transactions = [
      FactoryBot.create(:transaction, sync_log: sync_log, gross: 150, fee: 15, net: 135),
      FactoryBot.create(:transaction, sync_log: sync_log, gross: 100, fee: 5, net: 95),
    ]
    Integrations::Mailer::Sync.completed(sync_log: sync_log)
  end
end

The first thing we wanted was a way to rollback database changes made by the preview. We again relied a custom engine initializer insert functionality into to Rails::MailersController#preview to rollback the changes.

module Integrations
    class Engine < ::Rails::Engine
        initializer "integrations.mailers_controller.preview.transaction" do |app|
            # Allow us to create records when doing previews and rolling back those changes so the development database does not get polluted.
            # See https://stackoverflow.com/a/60154367/326979
            app.config.after_initialize do
                class Rails::MailersController
                    alias_method :preview_orig, :preview
                    def preview
                        ActiveRecord::Base.transaction do
                            preview_orig
                            raise ActiveRecord::Rollback
                        end
                    end
                end
            end
        end
    end
end

Second, we needed to tell the application where to look for the preview files. This was easily done by adding the following to the dummy app’s config/environments/development.rb to point to where the preview classes are found.

config.action_mailer.preview_path = Integrations::Engine.root.join("spec", "models", "integrations", "mailer", "preview")

Conclusion

I’m sure our Rails Engine gem has saved us hundreds of developer hours by sharing the same models, controllers, and views across many different applications. We’ve been able to create new integrations in a matter of weeks, not months. At this point, much of the integration code is served from the engine and not the main app.

It takes some orientating when first getting it established but using a Rails Engine for common code has worked well for us.

Thanks!

A hearty thank you to the other members of the Integrations Team (Karl Meisterheim and Max VelDink) for help in developing our engine and also reviewing and commenting on this treatise about Rails Engine development. It’s a pleasure to work with you fine gentlemen.

Appendix A: Dummy Commit

The following are modifications in a commit of a standard Rails 6 application that we continue to rebase to the top of our dummy branches to make it play nice with the engine gem. For reference, we call our engine gem integrations.

# File: Gemfile.rb

# Reference the engine root
+ gem "integrations", path: "../..", require: false

# Remove the reference by version
- gem "integrations", "3.8.3"

# File: Gemfile.lock

# This changes are a result of the above Gemfile changes with a `bundle install`, no need to change this by hand but I include it for completeness.
+PATH
+  remote: ../..
+  specs:
+    integrations (3.8.3)
+      http (~> 4.4.1)
+      mjml-rails (~> 4.6.1)
+      rails (~> 6.1.1)
+      scenic (~> 1.5.4)
+      scenic-mysql_adapter (~> 1.0.1)
+      sentry-delayed_job (~> 4.2.1)
+      sentry-rails (~> 4.2.2)
+      sentry-ruby (~> 4.2.2)
+      tithely-authentication (= 0.3.0)
+      tithely-hub (= 0.2.0)
+      tithely-transaction (= 0.9.0)
+

-    integrations (3.8.3)
-      http (~> 4.4.1)
-      mjml-rails (~> 4.6.1)
-      rails (~> 6.1.1)
-      scenic (~> 1.5.4)
-      scenic-mysql_adapter (~> 1.0.1)
-      sentry-delayed_job (~> 4.2.1)
-      sentry-rails (~> 4.2.2)
-      sentry-ruby (~> 4.2.2)
-      tithely-authentication (= 0.3.0)
-      tithely-hub (= 0.2.0)
-      tithely-transaction (= 0.9.0)

-  integrations (= 3.8.3)!
+  integrations!

# File: config/application.rb

# Add this directly after the `Bundler.require(*Rails.groups)` call and before the `MyApp::Application` class definition.
+require "integrations"

# File: config/boot.rb

# Reference the engine Gemfile
-ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../Gemfile', __dir__)
+# Set up gems listed in the Gemfile.
+ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../../Gemfile', __dir__)

# Load
-require 'bundler/setup' # Set up gems listed in the Gemfile.
-require 'bootsnap/setup' # Speed up boot time by caching expensive operations.

+require 'bundler/setup' if File.exist?(ENV['BUNDLE_GEMFILE'])
+$LOAD_PATH.unshift File.expand_path('../../../lib', __dir__)

# File: rails/config/environments/development.rb

# Provides mailer previews from the engine into the app, if needed
+  config.action_mailer.preview_path = File.expand_path("../../../../spec/models/integrations/mailer/preview", __dir__)

Appendix B: Engine's Rails Helper

# frozen_string_literal: true
# Contents of the engine's spec/rails_helper.rb

ENV["RAILS_ENV"] ||= "test"

require File.expand_path("dummy/config/environment", __dir__)

# Prevent database truncation if the environment is production
abort("The Rails environment is running in production mode!") if Rails.env.production?

# Checks for pending migrations and applies them before tests are run.
# If you are not using ActiveRecord, you can remove these lines.
begin
  ActiveRecord::Migration.maintain_test_schema!
rescue ActiveRecord::PendingMigrationError => e
  puts e.to_s.strip
  exit 1
end

require "spec_helper"

# Rails specific rspec configurations
require "rspec/rails"
require "factory_bot_rails"

Dir.glob(Integrations::Engine.root.join("spec", "support", "helpers", "**", "*.rb"), &method(:require))

RSpec.configure do |config|
  config.fixture_path = "#{::Rails.root}/spec/fixtures"
  config.include FactoryBot::Syntax::Methods
  config.use_transactional_fixtures = true

  # The different available types are documented in the features, such as in
  # https://relishapp.com/rspec/rspec-rails/docs
  config.infer_spec_type_from_file_location!

  # Filter lines from Rails gems in backtraces.
  config.filter_rails_from_backtrace!

  # include custom helper modules
  config.include Support::Helpers::Authentication
end