Managing change with Rollout Flags

The content of this article has been adapted from internal Scrive documentation, initially authored by Tom Feron.

Rollout Flags

Rollout flags are a software component used to control the progressive activation of new code paths throughout a system, at runtime. They are designed in a way that allows for partial activation over the population of users (or user groups) in the system.

The main advantage is avoiding the risk of breaking the system and the users' workflows when a new release exposes a new code path to all users.

What Rollout Flags are not

Given the nature of rollout flags, it would be easy to misuse them for feature (de)activation; that's not what they are meant for. For this use case we have Feature flags.

Whilst they work very similarly on a technical level, by enabling code paths according to the activation status of a flag, rollout flags are not intended to persist in time. This is a major difference, because the users of a system may have a feature flag enabled due to an agreement with the service provider (by upgrading to a paid plan, for instance).

Rollout flags must have an expiration date, as we intend to end up with the new code either fully deployed, or not. The in-between that rollout flags allow is helpful to monitor the system's reaction to the new code paths enabled in select parts, but the progressive deployment of a code path should indeed aim for total deployment — be it in the code's current form or a later one — as we can refine the code based on our system's observability.

But to every rule there are exceptions, and in some cases rollout flags can stay around indefinitely. The most common example would be when the removal of the rollout flag depends on a customer fixing their integration with the system.

Some heuristics

Here are a few situations that are appropriate for the use of rollout flags:

• Is the feature (or code path) likely to break something in production?
  • Rewrite/optimisation of complex code and SQL queries. They may have been tested, but test data and real-world data tend to be fairly different.
• You need to alter an endpoint behaviour.
  • You have to introduce new validations/checks that were missing, but are critical for the proper function of your software.
    There are sometimes a few customers that manage to misuse this and you would break their integrations, so you want to be able to disable the new checks for the few affected customers, while they work on the fix. The rollout flag in this case may actually be quite long-lived.
• You're introducing an experimental feature.
  • This one is not a clear-cut, as it depends if the feature will be controlled by a Feature flag later.
    If so, you may save yourself some work and use feature flag immediately, but for features that will be eventually rolled out to everyone, the rollout flag is a good choice.
• You're updating or replacing an external tool.
  • These kinds of changes always pose some risk of not working with data provided by some users (e.g. when they use some specific format or feature).
    It's good to have the option to revert to the old tool for those users quickly, without needing to redeploy your software.

Dealing with complexity over time

Rollout flags do increase complexity of the code, but it can be manageable if flags are used in a fairly isolated way. More than half of our rollout flags have only a single check in the code, most of them have less than 5 checks, because they have to be used in different components.

When adding a new rollout flag, do consider if other rollout flags can be removed.

Here is an example implementation of rollout flags.

Flags Metadata

We are going to start with a list of the flags:

data RolloutFlag
    = EnableEmailValildation
    | EnableWritingHandValidation
    | EnableBiscuitToken
    deriving stock (Eq, Ord, Show)

And we will define a record to store metadata about the flag:

data RolloutFlagInfo = RolloutFlagInfo
    { title :: Text
      -- ^ User friendly name to be displayed.
    , owner :: Text
      -- ^ Full name of developer who owns this flag.
    , defaultValue :: Bool
      -- ^ Enable/disable rollout flag by default.
      --   In most cases, the flag should start as disabled
    , description :: Text
      -- ^ Explanation of what it is supposed to do.
    }
    deriving stock (Eq, Ord, Show)

You can also add other fields like the link to a ticket in your tracking system.

Connecting Flags and their Information

Then, we establish the mapping between RolloutFlags and their corresponding RolloutFlagInfo:

rolloutFlagInfo :: RolloutFlag -> RolloutFlagInfo
rolloutFlagInfo = \case
  EnableEmailValildation -> RolloutFlagInfo
    { title        = "Enable email validation"
    , owner        = "Agnieszka Konstantinopoulos"
    , defaultValue = False
    , description  = "Email must contain '@'"
    }

As mentioned above, the default value for this rollout flag is False.
it is much better to enable it manually at a given time, especially if you need
someone from your SRE or Customer Support team to be warned beforehand.

Then, create a function that takes a RolloutFlag and gives you its value:

getRolloutFlag :: RolloutFlag -> m Bool

(Usually, m would be IO, MonadIO, or your effect system's own monad)

Here is a sample use-case:

Let us take a code path that starts with someFunctionInTheCode. We wish to replace its behaviour for something different that may break some things in production.
What we can do is extract the old implementation into a function that would be called makeNewEmail, and implement the new code path as a function called parseEmail:

someFunctionInTheCode :: Text -> m (Maybe Email)
someFunctionInTheCode x = do
  rolloutFlag <- getRolloutFlag EnableEmailValildation
  case rolloutFlag of
    True -> parseEmail x
    False -> pure (Just (makeNewEmail x)) 
    ^ -- We used to unconditionally create new email addresses

parseEmail :: Text -> m (Maybe Text)
parseEmail x =
  -- New implementation

makeNewEmail :: Text -> Email
makeNewEmail x =
  -- Old implementation

The function someFunctionInTheCode handles the dispatching of the control flow according to the value of the flag.

Operating Rollout Flags

To preserve consistency, it is recommended that the rollout flag settings be stored in your main database, so that these settings are preserved across deployments and application reboots. To avoid needing people touch the database by hand, an interface should be created in your back-office (for administrators):

I hope this introduction can give you some ideas on how to implement rollout flags in your application!