UIAlertController alerts form the backbone of a lot of the interactions between our users and our apps. Alerts are often shown at those critical points in our apps where we are asking to confirm an action or allow access to a resource. While there have been some changes to alerts over the years, very little has changed about their appearance. This lack of customisation presents significant difficulties for app designers 😱. Having an UIAlertController alert pop up at the most critical points of our apps with its semi-transparent rectangular layout and standard fonts, breaking the app's theme is enough to make any app designer spill their flat white coffee down their checkered shirt and throw their Caran d'Ache coloured pencils across the table with an angst that few could ever comprehend, never mind experience. At this point, as you gaze into their tear-filled eyes, you offer a few comforting words:

"We can always write our own custom alerts instead"

Slowly as your words start to cut through their anguish, they start to nod, and with strength returning to their voice, they say:

"Sounds good man, I'll get started on those right away"

And with that, you turn away and start thinking about how you can present those custom alerts.

This article will look at how to build a custom alert presenter that will try and follow most of the conventions used by UIAlertController alerts. Along the way, we will even overcome a particularly tricky and challenging to reproduce navigation bug 💪.

This post will gradually build up to a working example however if you are too excited and want to jump ahead then head on over to the completed example and take a look at AlertPresenter and AlertWindow to see how things end up.

Thinking about being standard

A standard UIAlertController alert has 4 parts:

1. Foreground view.
2. Background view.
3. Presentation animation.
4. Dismissal animation.

The first thing to decide is:

"What is part of the custom alert, and what is part of the mechanism to show an alert?"

Let's look at how UIAlertController handles the 4 parts of an alert:

1. Foreground view - UIAlertController allows some customisation.
2. Background view - UIAlertController handles this for us.
3. Presentation animation - UIAlertController handles this for us.
4. Dismissal animation - UIAlertController handles this for us.

The only customisable part of an UIAlertController alert is the foreground view. This lack of control with the other parts may at first feel limiting, but by preventing customisation of 3-of-the-4 parts, iOS forces us to focus the most critical part - the message. The message is contained in the foreground view.

Just like UIAlertController, the alert presentation layer we will build below will only allow the foreground view will be customisable. This limitation will ensure that presenting and dismissing alerts will happen consistently across the app. Instead of the foreground view being a UIView instance, it will be a UIViewController instance to provide greater control to our custom alerts. This UIViewController instance will be added to the view hierarchy as a child view-controller. This functionality will come together in the following class structure:

• AlertPresenter is the entry point for presenting and dismissing alerts.
• AlertWindow, as we will see shortly, each alert is presented in its own UIWindow instance. Used to overcome that tricky navigation issue we spoke about above.
• HoldingViewController, the windows root view-controller that is responsible for presenting the AlertContainerViewController that will hold the alert view-controller as a child view-controller.
• AlertContainerViewController, the parent view-controller that the alert view-controller is embedded in as a child view-controller.
• CustomAlertPresentAnimationController is responsible for presenting the AlertContainerViewController instance with the same animation as a standard UIAlertController.
• CustomAlertDismissAnimationController is responsible for dismissing the AlertContainerViewController instance with the same animation as a standard UIAlertController.

HoldingViewController, AlertContainerViewController, CustomAlertPresentAnimationController and CustomAlertDismissAnimationController are private and only known to AlertWindow.

Don't worry if that doesn't all make sense yet, we will look into each class in greater depth below.

Let's start with AlertPresenter:

class AlertPresenter {
    static let shared = AlertPresenter()

    // MARK: - Present

    func presentAlert(_ viewController: UIViewController) {
        os_log(.info, "Alert being presented")

        //TODO: Present
    }
}

AlertPresenter is a singleton, so the same instance will be used to present (and dismiss) all alerts. As AlertPresenter isn't a UIViewController subclass, it's not possible to directly present the alert. Instead, we are going to use a dedicated UIWindow instance to present alerts from. Using a dedicated UIWindow instance should avoid the situation where multiple simultaneous navigation events (presentation/dismissal) occur at the same time, resulting in one of those events being cancelled and the following error is generated:

The navigation stack in one window is independent of the navigation stack of any other windows. An alert in the process of being presented on window A will not cause a navigation collision with a view-controller being pushed on window B 🥳.

Before delving into how to use a dedicated window to present alerts, let's get to know windows better.

If you are comfortable with how UIWindow works, feel free to skip ahead.

Getting to know windows 💭

UIWindow is a subclass of UIView that acts as the container for an app's visible content - it is the top of the view hierarchy. All views that are displayed to the user need to be added to a window. An app can have multiple windows, but only windows that are visible can have their content displayed to the user - by default windows are not visible. Multiple windows can be visible at once. Each window's UI stack is independent of other window's UI stacks. Where a window is displayed in regards to other visible windows is controlled by setting that window's windowLevel property. The higher the windowLevel the nearer to the user that window is. UIWindow has three default levels:

1. .normal
2. .statusBar
3. .alert

With .alert > .statusBar > .normal. If a more fine-grain level of control is needed it's possible to use a custom level:

window.windowLevel = .normal + 25

As of iOS 13: .alert has a raw value of 2000, .statusBar has a raw value of 1000 and .normal has a raw value of 0.

Where two or more windows have the same level, their ordering is determined by the order they were made visible in - the last window made visible is nearest one to the user.

It's unusual to directly add subviews to a window instead each window should have a root view-controller who's view is used as the window's initial subview.

As well as displaying content, UIWindow is also responsible for forwarding any events (touch, motion, remote-control or press) to interested parties in it's responder chain. While all touch events are forwarded to the responder chain of the window that the event occurred on, events that are outside of the app's UI such as motion events or keyboard entry, are forwarded to the key window. Only one window can be key at any one given time (which window is key can change throughout the lifetime of the app).

An iOS app needs at least one window, a reference to this window can be found in the app delegate:

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {

    var window: UIWindow?

    //Omitted methods
}

If you're using storyboards to layout your interface, then most of the work of setting up this window is happening under the hood. When the app is launched, a UIWindow instance is created that fills the screen. This window is then assigned to the window property (declared in the UIApplicationDelegate protocol), configured with the view-controller declared as the storyboard entry point from the project's main storyboard and made key and visible.

If you are not using storyboards you can better see this setup:

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {

    var window: UIWindow?

    // MARK - AppLifecycle

    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
        window = UIWindow()
        window?.rootViewController = YourViewController()
        window?.makeKeyAndVisible()

        return true
    }
}

Using a dedicated window

As this new window will only display alerts, let's subclass UIWindow for this single purpose:

class AlertWindow: UIWindow {

    // MARK: - Init

    init(withViewController viewController: UIViewController) {
        super.init(frame: UIScreen.main.bounds)

        // 1
        rootViewController = viewController

        // 2
        windowLevel = .alert
    }

    @available(*, unavailable)
    required init?(coder aDecoder: NSCoder) {
        fatalError("Unavailable")
    }

    // MARK: - Present

    // 3
    func present() {
        makeKeyAndVisible()
    }
}

In AlertWindow we:

1. Set the alert as the window's rootViewController.
2. Set the window level value to .alert. The .alert level will put this window above the app's main window, ensuring that it can be seen.
3. present() is a bit of syntactic sugar around making the window key and visible, it will act as a mirror to the soon-to-be-seen dismiss method.

An AlertWindow instance is only meant to show one alert and then be disposed of.

Let's use AlertWindowto present alerts:

class AlertPresenter {
    // 1
    private var alertWindows = Set()

    //Omitted other properties

    // MARK: - Present

    // 2
    func presentAlert(_ viewController: UIViewController) {
        os_log(.info, "Alert being presented")

        let alertWindow = AlertWindow(withViewController: viewController)
        alertWindow.present()

        alertWindows.insert(alertWindow)
    }
}

With the above changes we:

1. Store all windows that are being presented in a Set. Storing the window inside the set will keep that window alive by increasing its retain count (as making a window key-and-visible doesn't increase the retain count).
2. Create an AlertWindow instance using the alert to be presented and then instruct that window to present itself.

If you were to create an instance of AlertWindow and make it visible, you would notice that the alert is presented without an animation - it just appears. A window's root view-controller cannot be animated on-screen so an intermediate view-controller is needed which can be the window's root view-controller and then the alert can be presented from that view-controller:

class HoldingViewController: UIViewController {
    private let viewController: UIViewController

    // MARK: - Init

    init(withViewController viewController: UIViewController) {
        self.viewController = viewController
        super.init(nibName: nil, bundle: nil)
    }

    required init?(coder aDecoder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }

    // MARK: - ViewLifecycle

    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)

        present(viewController, animated: true, completion: nil)
    }
}

HoldingViewController only has one responsibility - presenting an alert once viewDidAppear(_:) has been called.

Trying to present an alert earlier will cause a navigation collision due to the HoldingViewController instance having not finished its own "animation" on screen. While (at the time of writing - iOS 13) this doesn't seem to affect the actual presentation of the alert, waiting for HoldingViewController to be presented before attempting another presentation ensures that the error isn't produced.

Lets go back to AlertWindow and make use of HoldingViewController:

class AlertWindow: UIWindow {
    private let holdingViewController: HoldingViewController

    // MARK: - Init

    init(withViewController viewController: UIViewController) {
        holdingViewController = HoldingViewController(withViewController: viewController)
        super.init(frame: UIScreen.main.bounds)

        rootViewController = holdingViewController

        windowLevel = .alert
    }


    // MARK: - Present

    func present() {
        makeKeyAndVisible()
    }
}

If you run the project with the above changes and hook in your own alert, you would notice two issues:

1. Sizing - the alert occupies the full-screen.
2. Presentation - the alert is animated from bottom to top.

Sizing alerts

An alert should be shown at the smallest size possible - we can't do this if that alert is the modal. Instead, we need to embed that alert into another view controller. HoldingViewController can't be used for this as it is being used as the window's root view-controller so can't be animated on-screen. We need to introduce a new view-controller that will act as a container so that that container can be animated on-screen from the HoldingViewController instance:

class AlertContainerViewController: UIViewController {
    let childViewController: UIViewController

    // MARK: - Init

    // 1
    init(withChildViewController childViewController: UIViewController) {
        self.childViewController = childViewController
        super.init(nibName: nil, bundle: nil)
    }

    required init?(coder aDecoder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }

    // MARK: - ViewLifecycle

    override func viewDidLoad() {
        super.viewDidLoad()

        // 2
        addChild(childViewController)
        view.addSubview(childViewController.view)
        childViewController.didMove(toParent: self)

        // 3
        childViewController.view.translatesAutoresizingMaskIntoConstraints = false
        NSLayoutConstraint.activate([
            childViewController.view.centerXAnchor.constraint(equalTo: view.centerXAnchor),
            childViewController.view.centerYAnchor.constraint(equalTo: view.centerYAnchor),
            childViewController.view.widthAnchor.constraint(lessThanOrEqualTo: view.widthAnchor, multiplier: 1),
            childViewController.view.heightAnchor.constraint(lessThanOrEqualTo: view.heightAnchor, multiplier: 1)
        ])
    }
}

In AlertContainerViewController we:

1. Store the alert passed in via the init'er as a property.
2. Embed the alert as a child view-controller.
3. Centre the alert and constrain it to (at maximum) the width of the view's width and height.

I don't view an alert that is too large for the container as being the presentation layers problem to solve - if an alert is too large for a single screen, I'd question if it really is an alert.

Let's update HoldingViewController to use AlertContainerViewController:

class HoldingViewController: UIViewController {
    let containerViewController: AlertContainerViewController

    // MARK: - Init

    init(withViewController viewController: UIViewController) {
        // 1
        containerViewController = AlertContainerViewController(withChildViewController: UIViewController)
        super.init(nibName: nil, bundle: nil)
    }

    //Omitted other methods

    // MARK: - ViewLifecycle

    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)

        // 2
        present(containerViewController, animated: true, completion: nil)
    }
}

With the changes to HoldingViewController we:

1. Create an AlertContainerViewController instance.
2. Modally present that AlertContainerViewController instance rather than the alert.

Now if you run the above code with your UIViewController subclass, you would notice that your alert is compressed to be as small as it can be and centred in the screen.

Time to address the second point - Presentation.

Presenting alerts

Let's change the presentation animation to make it feel more like the UIAlertController.

As we move from one view-controller to another, we are used to seeing different types of animation, the most common being:

• Push: the new view slides in from the side on top of the current view.
• Pop: the current view slides out to the side to reveal another view underneath.
• Present: the new view slides up from the bottom on top of the current view.
• Dismiss: the current view slides down to reveal another vie underneath.

These view transactions are provided free by iOS. However it is possible to specify our own view transactions by using the Transitions API. The Transitions API is a suite of protocols that determine how custom transitions should behave, there are quite a few protocols in the suite however only two of them are of interest to us:

1. UIViewControllerTransitioningDelegate - is a set of methods that vend objects used to manage a fixed-length or interactive transition between view controllers. Every view-controller has a transitioningDelegate which has a type of UIViewControllerTransitioningDelegate. When a transaction is about to happen, iOS asks the transitioning-delegate for an animator to use. If the transitioningDelegate is nil or the necessary UIViewControllerTransitioningDelegate method hasn't been implemented, then iOS will fall back to using the default animation for that type of transition.
2. UIViewControllerAnimatedTransitioning - is a set of methods for implementing the animations for a custom view controller transition. A class that conforms to UIViewControllerAnimatedTransitioning is known as the animator. An animator controls the duration of the transition and allows for manipulating the from-view-controller's view and to-view-controller's view on the transition canvas.

I briefly covered the Transitions API above - if you want to read more, I'd recommend this post.

Let's create an animator to handle presenting the alert:

class CustomAlertPresentAnimationController: NSObject, UIViewControllerAnimatedTransitioning // 1 {

    // MARK: - UIViewControllerAnimatedTransitioning

    // 2
    func transitionDuration(using transitionContext: UIViewControllerContextTransitioning?) -> TimeInterval {
        return 0.2
    }

    // 3
    func animateTransition(using transitionContext: UIViewControllerContextTransitioning) {
        guard let toViewController = transitionContext.viewController(forKey: .to),
            let snapshot = toViewController.view.snapshotView(afterScreenUpdates: true)
            else {
                return
        }

        let containerView = transitionContext.containerView
        let finalFrame = transitionContext.finalFrame(for: toViewController)

        snapshot.frame = finalFrame
        snapshot.transform = CGAffineTransform(scaleX: 1.1, y: 1.1)
        snapshot.alpha = 0.0

        containerView.addSubview(toViewController.view)
        containerView.addSubview(snapshot)
        toViewController.view.isHidden = true

        let duration = transitionDuration(using: transitionContext)

        UIView.animate(withDuration: duration, animations: {
            snapshot.alpha = 1.0
            snapshot.transform = CGAffineTransform(scaleX: 1.0, y: 1.0)
        }) { _ in
            toViewController.view.isHidden = false
            snapshot.removeFromSuperview()
            transitionContext.completeTransition(!transitionContext.transitionWasCancelled)
        }
    }
}

CustomAlertPresentAnimationController can at first glance look a little intimidating so let's break it down:

1. CustomAlertPresentAnimationController conforms to UIViewControllerAnimatedTransitioning and needs to be a subclass of NSObject as UIViewControllerAnimatedTransitioning extends NSObjectProtocol.
2. In transitionDuration(using:) the duration of the animation is specified as 0.2 seconds - we've copied the timing of an UIAlertController alert here.
3. UIAlertController instances when animated onto the screen, gradually fade in with a slight bounce effect before settling to its actual size. In animateTransition(using:) we use the UIViewControllerContextTransitioning instance (transitionContext) to get the offscreen view of the to-view-controller (i.e. the AlertContainerViewController instance holding our alert). We take a snapshot (screenshot) of the AlertContainerViewController instances view, add that snapshot to the animator's container view (think of this as a temporary transaction view that is present during the animation) and animate the snapshot on the screen to mimic a UIAlertController animation. Taking a snapshot means we avoid having to deal with any constraint issues that may arise from manipulating the actual AlertContainerViewController instances view. Once the animation is finished, we remove the snapshot from the view hierarchy and reveal the AlertContainerViewController instance's view occupying the same position.

As HoldingViewController is the view-controller that presents the AlertContainerViewController instance, it needs to conform to UIViewControllerTransitioningDelegate:

class HoldingViewController: UIViewController, UIViewControllerTransitioningDelegate {
    //Omitted properties

    init(withViewController viewController: UIViewController) {
        //Omitted start of method

        // 1
        containerViewController.modalPresentationStyle = .custom

        // 2
        containerViewController.transitioningDelegate = self
    }

    //Omitted other methods

    // MARK: - UIViewControllerTransitioningDelegate

    // 3
    func animationController(forPresented presented: UIViewController, presenting: UIViewController, source: UIViewController) -> UIViewControllerAnimatedTransitioning? {
        return CustomAlertPresentAnimationController()
    }
}

With the changes to HoldingViewController we:

1. Set the modalPresentationStyle to custom as we will be providing the modal presentation animation.
2. Set this HoldingViewController instance as the transitioningDelegate of the AlertContainerViewController instance.
3. Return an instance of CustomAlertPresentAnimationController when presenting an AlertContainerViewController instance.

If you add in the above code changes to your project and run it, you would now see your alert being animated onto the screen in the same way as an UIAlertController alert.

So we just need to add in a semi-transparent background, and our alert presentation will be complete:

class AlertContainerViewController: UIViewController {
    //Omitted properties and other methods

    override func viewDidLoad() {
        super.viewDidLoad()

        let backgroundView = UIView()
        backgroundView.backgroundColor = UIColor.darkGray.withAlphaComponent(0.75)
        backgroundView.translatesAutoresizingMaskIntoConstraints = false

        view.addSubview(backgroundView)

        //Omitted child view-controller setup

        NSLayoutConstraint.activate([
            //Omitted child view-controller layout setup

            backgroundView.leadingAnchor.constraint(equalTo: view.leadingAnchor),
            backgroundView.topAnchor.constraint(equalTo: view.topAnchor),
            backgroundView.trailingAnchor.constraint(equalTo: view.trailingAnchor),
            backgroundView.bottomAnchor.constraint(equalTo: view.bottomAnchor)
        ])
    }
}

Now our alert presentation is complete; let's turn our attention to dismissing that alert.

Dismissing alerts

Just like presentation, dismissal will happen inside the AlertPresenter:

class AlertPresenter {
    //Omitted properties and other methods

    func dismissAlert(_ viewController: UIViewController) {
        // 1
        guard let alertWindow = alertWindows.first(where: { $0.viewController == viewController } )  else {
            return
        }

        os_log(.info, "Alert being dismissed")

        // 2
        alertWindow.dismiss { [weak self] in
            // 3
            self?.alertWindows.remove(alertWindow)
        }
    }
}

With the changes to AlertPresenter we:

1. Find the window that is showing the alert.
2. Call dismiss(completion:) on that window to begin the dismissal process.
3. Remove the window from alertWindows once the dismissal has completed.

There are a few ways we could have implemented the dismissal logic. In the end, I decided to require all custom alerts to call dismissAlert(_:) when they are ready to be dismissed. This direct calling approach has symmetry with how the alert is presented and is very simple.

If you try to run the above method you will get an error because AlertWindow doesn't yet have an viewController property or dismissAlert(_:) method so let's add them in:

class AlertWindow: UIWindow {
    // 1
    var viewController: UIViewController {
        return holdingViewController.containerViewController.childViewController
    }

    //Omitted other methods and properties

    // MARK: - Dismiss

    func dismiss(completion: @escaping (() -> Void)) {
        // 2
        holdingViewController.dismissAlert { [weak self] in
            // 3
            self?.resignKeyAndHide()
            completion()
        }
    }

    // MARK: - Resign

    private func resignKeyAndHide() {
        resignKey()
        isHidden = true
    }
}

With the above changes to AlertWindow we:

1. Added a new viewController property that gets the alert being displayed and returns it.
2. Pass the dismissal command onto the HoldingViewController instance.
3. Resign the window and hide it once the dismissal has completed.

Normally I don't like the level of method chaining shown inside the viewController property. However, I feel comfortable doing it here as HoldingViewController and AlertContainerViewController are private implementation details of AlertWindow.

Let's add a dismissAlert(_:) method into HoldingViewController:

class HoldingViewController: UIViewController, UIViewControllerTransitioningDelegate {
    //Omitted properties and other methods

    // MARK: - Dismiss

    func dismissAlert(completion: @escaping (() -> Void)) {
        containerViewController.dismiss(animated: true, completion: {
            completion()
        })
    }
}

With the above change, we call the standard UIKit dismiss(animation:completion:) on the containerViewController and trigger the completion closure when that dismiss action completes.

If you add in the above code changes to your project and run it by calling AlertPresenter.shared.dismiss(completion:) when your alert's dismissal button is pressed then your alert should be dismissed. However, it will still be using the standard modal dismissal animation. Just like with presenting, dismissing will need an animator:

class CustomAlertDismissAnimationController: NSObject, UIViewControllerAnimatedTransitioning {

    // MARK: - UIViewControllerAnimatedTransitioning

    func transitionDuration(using transitionContext: UIViewControllerContextTransitioning?) -> TimeInterval {
        return 0.2
    }

    func animateTransition(using transitionContext: UIViewControllerContextTransitioning) {
        guard let fromViewController = transitionContext.viewController(forKey: .from),
            let snapshot = fromViewController.view.snapshotView(afterScreenUpdates: true)
            else {
                return
        }

        let containerView = transitionContext.containerView
        let finalFrame = transitionContext.finalFrame(for: fromViewController)

        snapshot.frame = finalFrame

        containerView.addSubview(snapshot)
        fromViewController.view.isHidden = true

        let duration = transitionDuration(using: transitionContext)

        UIView.animate(withDuration: duration, animations: {
            snapshot.alpha = 0.0
        }) { _ in
            snapshot.removeFromSuperview()
            transitionContext.completeTransition(!transitionContext.transitionWasCancelled)
        }
    }
}

CustomAlertDismissAnimationController is mostly the opposite of CustomAlertPresentAnimationController but without the bounce effect.

Now HoldingViewController just has to return an CustomAlertDismissAnimationController instance at the appropriate moment:

class HoldingViewController: UIViewController, UIViewControllerTransitioningDelegate {
    //Omitted properties and other methods

    func animationController(forDismissed dismissed: UIViewController) -> UIViewControllerAnimatedTransitioning? {
        return CustomAlertDismissAnimationController()
    }
}

And that completes our custom alert presenter.

🍾💃🏻🕺🏻

Feeling hopeful? 🌞

To recap, we built a simple custom alert presentation system that takes an alert view-controller, places it in the centre of a semi-transparent container and presents it in a dedicated window, all the while doing so with the same feel as presenting a UIAlertController instance.

I spent quite a long time toying with the idea of having AlertPresenter accept view-models rather than view-controllers (with the view-model then being transformed into view-controllers inside the presenter). However, the view-model solution always ended up feeling very confused - was AlertPresenter part of the app's infrastructure or part of the app's UI. By using view-models AlertPresenter had to be both 😧. Each time someone created a new type of alert, AlertPresenter would need to be modified to know about the new view-model - breaking the open/closed principle and opening the door to unexpected consequences rippling through the app (you can see this approach on this branch) via the AlertPresenter. By moving the alert view-controller creation from AlertPresenter to the component that wanted to use AlertPresenter, I was able to give AlertPresenter a clear purpose: AlertPresenter takes a view-controller and presents it in the same manner as an UIAlertController alert. This clear division of responsibility between alert creator and alert presenter, I believe, has meant that AlertPresenter is a simple, easy-to-understand component that should very rarely need to be modified.

To see the above code snippets together in a working example, head over to the repo and clone the project.

Being British, queuing is essential to me. I take every opportunity I can get to queue: posting a package ✅, paying for my groceries ✅, fleeing a burning building ✅, etc. So every time I need to present a sequence of alerts, I come up against the uncomfortable truth that UIAlertController doesn't care as much about queuing as I do 😔.

This article is a look at how UIAlertController can be made a little more British through embracing a strong queuing etiquette.

Let's get queuing 🇬🇧

A queue is a first-in, first-out data structure so that the oldest element is the first to be removed. Swift doesn't come with a built in queue type so lets build one:

struct Queue<Element> {
    private var elements = [Element]()

    // MARK: - Operations

    mutating func enqueue(_ element: Element) {
        elements.append(element)
    }

    mutating func dequeue() -> Element? {
        guard !elements.isEmpty else {
            return nil
        }

        return elements.removeFirst()
    }
}

The above class is a relatively trivial generic queue implementation that allows only two operations - enqueue and dequeue.

When queuing alerts, it's important to ensure that all alerts are queued on the same instance. To achieve this, we need a specialised singleton class that will control access to the queue and be the central component for all alert presentation logic:

class AlertPresenter {
    private var alertQueue = Queue<UIAlertController>()

    static let shared = AlertPresenter()

    // MARK: - Present

    func enqueueAlertForPresentation(_ alertController: UIAlertController) {
        alertQueue.enqueue(alertController)

        //TODO: Present
    }
}

In the above class, AlertPresenter holds a private queue specialised for UIAlertController elements and a method for enqueuing alerts. But as of yet, no way to present the queued alert. Before we can implement a presentation method, let's look at how alerts are normally presented:

let alertController = ...

present(alertController, animated: true, completion: nil)

UIAlertController (which is a subclass of UIViewController) is presented modally like any other view-controller by calling present(_:animated:completion:) on the presenting view-controller. A consequence of queuing alerts in AlertPresenter is that it brokes the usual alert presentation flow. As AlertPresenter isn't a subclass of UIViewController we can't use it directly for presenting. Instead, we need to get a view-controller to present from. As well as requiring a view-controller to be injected into AlertPresenter, our indirect alert presentation also exacerbates an existing subtle issue with presenting alerts - simultaneous navigation events (presentation/dismissal) occurring at the same time resulting in one of those events being cancelled and the following error being generated:

Without AlertPresenter being involved, the alert would be presented directly from the view-controller, allowing that view-controller to prevent other navigation events occurring until after the alert is dismissed. However, by queueing the alert, the view-controller has no way of knowing when it will be shown so no way of knowing when it should or shouldn't prevent other navigation events.

While it's possible for AlertPresenter to query the topmost view-controller and determine if it is in the process of presenting or being dismissed, doing so requires logic that gets messy quickly 🤢. Instead of having to accommodate events happening on the navigation stack that AlertPresenter doesn't control, we can raise AlertPresenter above all navigation concerns by using a dedicated UIWindow instance (with a dedicated view-controller) to present the queued alerts from. As the navigation stack in one window is independent of any other window's navigation stack, an alert can be presented on its window at the same time as a view controller is being pushed on its window without navigation collisions 🥳.

Before delving into how to use a dedicated window to present alerts, let's get to know windows better.

If you are comfortable with how UIWindow works, feel free to skip ahead.

Getting to know windows 💭

UIWindow is a subclass of UIView that acts as the container for an app's visible content - it is the top of the view hierarchy. All views that are displayed to the user need to be added to a window. An app can have multiple windows, but only windows that are visible can have their content displayed to the user - by default windows are not visible. Multiple windows can be visible at once. Each window's UI stack is independent of other window's UI stacks. Where a window is displayed in regards to other visible windows is controlled by setting that window's windowLevel property. The higher the windowLevel the nearer to the user that window is. UIWindow has three default levels:

1. .normal
2. .statusBar
3. .alert

With .alert > .statusBar > .normal. If a more fine-grain level of control is needed it's possible to use a custom level:

window.windowLevel = .normal + 25

As of iOS 13: .alert has a raw value of 2000, .statusBar has a raw value of 1000 and .normal has a raw value of 0.

Where two or more windows have the same level, their ordering is determined by the order they were made visible in - the last window made visible is nearest one to the user.

It's unusual to directly add subviews to a window instead each window should have a root view-controller who's view is used as the window's initial subview.

As well as displaying content, UIWindow is also responsible for forwarding any events (touch, motion, remote-control or press) to interested parties in it's responder chain. While all touch events are forwarded to the responder chain of the window that the event occurred on, events that are outside of the app's UI such as motion events or keyboard entry, are forwarded to the key window. Only one window can be key at any one given time (which window is key can change throughout the lifetime of the app).

An iOS app needs at least one window, a reference to this window can be found in the app delegate:

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {

    var window: UIWindow?

    //Omitted methods
}

If you're using storyboards to layout your interface, then most of the work of setting up this window is happening under the hood. When the app is launched a UIWindow instance is created that fills the screen. This window is then assigned to the window property (declared in the UIApplicationDelegate protocol), configured with the view-controller declared as the storyboard entry point from the project's main storyboard as it's rootViewController and finally the window is made key and visible.

If you are not using storyboards you can better see this setup:

@UIApplicationMain
class AppDelegate: UIResponder, UIApplicationDelegate {

    var window: UIWindow?

    // MARK - AppLifecycle

    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
        window = UIWindow()
        window?.rootViewController = YourViewController()
        window?.makeKeyAndVisible()

        return true
    }
}

Using windows

As this new window will only display alerts, let's subclass UIWindow for this single purpose:

class AlertWindow: UIWindow {
    // MARK: - Init

    init(withAlertController alertController: UIAlertController) {
        super.init(frame: UIScreen.main.bounds)

        rootViewController = alertController

        windowLevel = .alert
    }

    @available(*, unavailable)
    required init?(coder aDecoder: NSCoder) {
        fatalError("Unavailable")
    }

    // MARK: - Present

    func present() {
        makeKeyAndVisible()
    }
}

In the above AlertWindow class an alert is passed into the init'er and set to the window's rootViewController, the window is then configured to have a .alert window level - this will put it above the app's main window (which by default has a .normal window level).

If you create an instance of AlertWindow and make it visible, you will notice that it's not being presented with an animation - it just appears which feels weird. A window's root view-controller can not be animated on-screen so to keep the alert's animation we need that alert to be presented from an intermediate view-controller which can be the window's root view-controller:

class HoldingViewController: UIViewController {
    private let alertController: UIAlertController

    // MARK: - Init

    init(withAlertController alertController: UIAlertController) {
        self.alertController = alertController
        super.init(nibName: nil, bundle: nil)
    }

    required init?(coder aDecoder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }

    // MARK: - ViewLifecycle

    override func viewDidLoad() {
        super.viewDidLoad()
        view.backgroundColor = .clear
    }

    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)

        present(alertController, animated: true, completion: nil)
    }
}

The above HoldingViewController class only has one responsibility - presenting an alert once viewDidAppear(_:) has been called.

Trying to present an alert earlier will cause an animation error due to the HoldingViewController instance having not finished its own "animation" on screen. While (at the time of writing - iOS 13) this doesn't seem to affect the actual presentation of the alert, waiting for HoldingViewController to be presented before attempting another presentation ensures that the error isn't produced.

Lets use HoldingViewController:

class AlertWindow: UIWindow {

    // MARK: - Init

    init(withAlertController alertController: UIAlertController) {
        super.init(frame: UIScreen.main.bounds)

        rootViewController = HoldingViewController(withAlertController: alertController)

        windowLevel = .alert
    }

    //Omitted methods
}

In the above class, instead of the UIAlertController instance being set directly as the rootViewController it is used to create an HoldingViewController instance which is set as the window's rootViewController.

Each AlertWindow instance is intended to be single-use - to present one alert. This single-use nature allows for its mere presence to be the determining factor in whether a queued alert should be presented or not:

class AlertPresenter {
    //Omitted properties

    private var alertWindow: AlertWindow?

    // MARK: - Present

    func enqueueAlertForPresentation(_ alertController: UIAlertController) {
        alertQueue.enqueue(alertController)

        showNextAlertIfPresent()
    }

    private func showNextAlertIfPresent() {
        guard alertWindow == nil,
            let alertController = alertQueue.dequeue() else {
                return
        }

        let alertWindow = AlertWindow(withAlertController: alertController)
        alertWindow.present()

        self.alertWindow = alertWindow
    }
}

With the above changes, alertWindow holds a reference to the window that is being used to present the alert. Inside showNextAlertIfPresent() if alertWindow is nil and there is a queued alert then a new AlertWindow instance is created, presented and assigned to alertWindow. Making the window key and visible sets off the chain of activity that results in the alert being animated on screen.

The example so far while functional is limited - it can present only one alert. AlertPresenter needs to be informed when an alert has been dismissed so it can move onto the next alert.

It turns out knowing when an alert has been dismissed is one of the trickier things to know about in an iOS app 😕. I had to work through a number of different solutions before I got somewhere I was happy:

1. My first attempt involved subclassing UIAlertController and overriding viewDidDisappear(_:) with a call to AlertPresenter - everyone would then use this subclass instead of UIAlertController directly. However, this approach is explicitly warned against in the documentation for UIAlertController:

👎

2. My second attempt involved requiring each developer to explicitly call AlertPresenter from inside each UIAlertAction closure. However, this approach puts too much of a burden on each developer to remember to include those calls in each and every action. A missed call from any action closure (that was substantially triggered) would cause the queue to be jammed from that alert until the app was killed. This is too easy a requirement to forget when writing or reviewing 👎.

3. My third attempt involved using a view model to abstract the creation of UIAlertController instances to a factory method where a call to AlertPresenter could then be injected into each UIAlertAction closure (during the conversion from view model to UIAlertController instance). However, this approach would require a lot of custom code to be written, maintained and tested - using UIAlertController directly would avoid all effort 👎.

Tricky, tricky 🤔.

Instead of thinking about how to get UIAlertController to tell us about what was happening to it, I decided to start thinking about what impact UIAlertController had on its surroundings. UIViewController has a great suite of methods for when it will appear and disappear - these appearance events can tell us what has happened to the presented alert. When an alert is dismissed, it calls func dismiss(animated:completion:) on the view-controller it was presented from. We can hook into this behaviour to inform AlertWindow about the dismissal:

protocol HoldingDelegate: class {
    func viewController(_ viewController: HoldingViewController, didDismissAlert alertController: UIAlertController)
}

class HoldingViewController: UIViewController {
    //Omitted properties

    weak var delegate: HoldingDelegate?

    // Omitted methods

    override func dismiss(animated flag: Bool, completion: (() -> Void)? = nil) {
        //Called when a UIAlertController instance is dismissed
        super.dismiss(animated: flag) {
            completion?()

            self.delegate?.viewController(self, didDismissAlert: self.alertController)
        }
    }
}

👍

You may be thinking: "Why is func dismiss(animated:completion:) being called on a view-controller that isn't being dismissed instead of viewDidAppear(_:) as it is actually reappearing"?. The same thought occurred to me, and this mismatch between expectations vs reality left me a little uncomfortable 😒. However, in the end, I decided that being able to use UIAlertController without having to care about the queue outweighed my discomfort. If Apple changes this to better match the behaviour of other UIViewController instances, modifying this solution to handle both cases will be trivial.

When an alert has been dismissed, the presenting AlertWindow instance has served its purpose and can itself be dismissed:

class AlertWindow: UIWindow, HoldingDelegate {
      //Omitted properties

      // MARK: - Init

      init(withAlertController alertController: UIAlertController) {
          super.init(frame: UIScreen.main.bounds)

          let holdingViewController = HoldingViewController(withAlertController: alertController)
          holdingViewController.delegate = self

          rootViewController = holdingViewController

          windowLevel = .alert
      }

      //Omitted other methods

      // MARK: - HoldingDelegate

      func viewController(_ viewController: HoldingViewController, didDismissAlert alertController: UIAlertController) {
          resignKeyAndHide()
      }

      // MARK: - Resign

      private func resignKeyAndHide() {
          resignKey()
          isHidden = true
      }

}

All that is left to do is inform AlertPresenter that the alert has been dismissed, again the delegation pattern can be used here:

protocol AlertWindowDelegate: class {
    func alertWindow(_ alertWindow: AlertWindow, didDismissAlert alertController: UIAlertController)
}

class AlertWindow: UIWindow, HoldingDelegate {
    weak var delegate: AlertWindowDelegate?

    //Omitted properties and methods

    func viewController(_ viewController: HoldingViewController, didDismissAlert alertController: UIAlertController) {
        resignKeyAndHide()
        delegate?.alertWindow(self, didDismissAlert: alertController)
    }

    //Omitted methods
}

AlertPresenter then just needs to present the next alert (if present):

class AlertPresenter: AlertWindowDelegate {
    //Omitted properties and methods

    private func showNextAlertIfPresent() {
        guard alertWindow == nil,
            let alertController = alertQueue.dequeue() else {
                return
        }

        let alertWindow = AlertWindow(withAlertController: alertController)
        alertWindow.delegate = self

        alertWindow.present()

        self.alertWindow = alertWindow
    }

    // MARK: - AlertWindowDelegate

    func alertWindow(_ alertWindow: AlertWindow, didDismissAlert alertController: UIAlertController) {
        self.alertWindow = nil
        showNextAlertIfPresent()
    }
}

All queued out 🍾

Congratulations on having made it to the end of an article about queuing. You've shown a level of patience that even many Brits would struggle to achieve.

To recap, in this article we implemented a minimally intrusive alert queuing mechanism that allows us to continue using UIAlertController without having to ask the creators of those alerts to do anything more complicated than calling AlertPresenter.shared.enqueueAlertForPresentation(_:).

To see the above code snippets together in a working example, head over to the repo and clone the project.

Many of us feel nervous when meeting a group of people for the first time. What are the dynamics of the group, what are the in-jokes, will I find common ground with someone - are just a few questions that can plague you. A lot of your hard work at suppressing your crippling social self-doubt can unravel with a shaky introduction or misplaced comment. You may even find yourself wondering:

"Are they laughing with me because I'm funny or at me because of the overly pointy brogues that I'm wearing?"

Of course, that's nonsense. Pointy brogues are the height of fashion 👞.

It's not just you who can be anxious about new introductions; your app can be too. Computers are notoriously fickle about who they speak to and how they expect to be addressed. A barrage of errors await any app that breaks these social norms - it can be enough to make your app want to stay quietly in the corner rather than jump in and face potential rejection.

Thankfully, this social anxiety can be eased if your app already knows someone who understands the dynamics of the group - someone cool and suave like SSDP.

This post will gradually build up to a working example however if you want to jump ahead then head on over to the completed example and take a look at SSDPSearchSessionConfiguration, SSDPSearchSession, UDPSocketController, UDPSocket, SSDPServiceParser and SSDPService to see how things end up.

If you are already familiar with SSDP, feel free to skip the next section and head directly to building a working Swift solution.

Getting to know SSDP

SSDP (Simple Service Discovery Protocol) is a discovery protocol used to determine what services are available on a network. It is defined as part of the UPnP spec. SSDP is a zero-configuration networking protocol designed to allow nodes to be added and removed from a network without any involvement from a central service such as DNS or by assigning static IP addresses to specific nodes. This decentralised, dynamic approach is possible because SSDP uses UDP as it's underlying transportation protocol which allows for multicast communication.

Multicasting allows a node to transmit one message onto the network and for that message to be forwarded on to all interested nodes on the network without the sender node having to know what other network nodes are available (forwarding happens at the IP routing level). SSDP takes advantage of this forwarding functionality to allow any node to ask other nodes if they support a particular service, or conversely for a node which offers services to tell other nodes about those services.

For multicast messages, IANA has reserved the IPv4 address 239.255.255.250 and port 1900 for SSDP.

SSDP messages conform to the header field format of HTTP 1.1. It's important to note that SSDP does not allow any message to contain a body; everything is shared via those header fields.

An SSDP node is either a root device or control point. A root device offers one or more SSDP services; a control point uses SSDP services.

When a root device is responding to a discovery message it does so by sending a unicast (to a single specific node) message directly to the control point.

That's a lot of information to take in 😥. If it doesn't all make sense that's ok, most of the details will come up again later, and when seen in context, those details are easier to understand.

SSDP messages fall into two categories:

1. Discovery
2. Advertisement

This post is mainly concerned with Discovery messages but to ensure that we have the fullest possible understanding of SSDP I'll cover Advertisement messages as well (feel free to skip the Advertisement section).

Discovery

Discovery involves two message types:

1. Request
2. Response

A request message is when a control point transmits an M-SEARCH message onto the network looking for a particular service (or as we shall see soon, any service) e.g.

M-SEARCH * HTTP/1.1
HOST: 239.255.255.250:1900
MAN: "ssdp:discover"
MX: 1
ST: urn:dial-multiscreen-org:service:dial:1

An M-SEARCH request message contains:

• The host and port (HOST) the message will be sent to. Typically an M-Search message is multicast (like the example above) but can be unicast.
• The message type (MAN), for an M-Search this is always ssdp:discover.
• The search target (ST) of the service the search request is attempting to discover.
• The maximum wait response time (MX) in seconds that a root device can take before responding. The MX field is an attempt to overcome a scaling issue implicit with SSDP. SSDP is a chatty protocol, in a network with a significant number of nodes that host SSDP services, sending an M-SEARCH message could result in accidentally DDOS-ing the questing node due to too many services responding at once. The MX field instructs the root device to wait a random time between 0 and MX before attempting to respond - this should allow the responses to be spaced out enough to ease the processing strain on the control point. The MX value should be between 1 and 5. Even with the MX workaround, SSDP is recommended only to be used in home or small office networks.

A root device should only respond with services that match the search target field of the the request e.g.

HTTP/1.1 200 OK
CACHE-CONTROL: max-age=3600
ST: urn:dial-multiscreen-org:service:dial:1
USN: uuid:0175c106-5400-10f8-802d-b0a7374360b7::urn:dial-multiscreen-org:service:dial:1
EXT:
SERVER: Roku UPnP/1.0 MiniUPnPd/1.4
LOCATION: http://192.168.1.104:8060/dial/dd.xml

An M-Search response message contains:

• The cache-control (CACHE-CONTROL) value to determine for how long the message is valid.
• The search target (ST) of the service that is responding. ST should be common across all devices of this type.
• The unique service name (USN) to identify the service.
• The server system information (SERVER) value providing information in the following format: [OS-Name] UPnP/[Version] [Product-Name]/[Product-Version].
• The location URL (LOCATION) to allow the control point to gain more information about this service.

The EXT field is required for backwards compatibility with UPnP 1.0 but can otherwise be ignored.

Advertisement

An advertisement message is when a root device shares the status of each service it offers with the other nodes on the network.

There are three types of advertisement:

1. Alive
2. Update
3. ByeBye

An alive message allows interested devices to know that a service is available. An alive message is a multicast NOTIFY message e.g.

NOTIFY * HTTP/1.1
HOST: 239.255.255.250:1900
CACHE-CONTROL: max-age=3600
NT: urn:dial-multiscreen-org:service:dial:1
NTS: ssdp:alive
LOCATION: http://192.168.1.104:8060/dial/dd.xml
USN: uuid:0175c106-5400-10f8-802d-b0a7374360b7::urn:dial-multiscreen-org:service:dial:1

An alive message contains:

• The host and port (HOST) the message will be sent to.
• The cache-control (CACHE-CONTROL) value to determine for how long the message is valid.
• The notification type (NT) that defines the service it offers (the equivalent of ST in an M-Search message).
• The notification subtype (NTS), for an alive message this will always be ssdp:alive (the equivalent of MAN in an M-Search message).
• The location URL (LOCATION) to allow a receiving control point to gain more information about this service.
• The unique service name (USN) to identify the service.

An update message allows changes to a service to be shared. An update message is also a multicast NOTIFY message like the alive message e.g.

NOTIFY * HTTP/1.1
HOST: 239.255.255.250:1900
NT: urn:dial-multiscreen-org:service:dial:1
NTS: ssdp:update
LOCATION: http://192.168.1.160:8060/dial/dd.xml
USN: uuid:0175c106-5400-10f8-802d-b0a7374360b7::urn:dial-multiscreen-org:service:dial:1

An update message has the same header fields as an alive message with only the NTS value differing between them.

A byebye message allows any interested nodes to know when a service is about to be removed from the network. A byebye message should be sent for each valid (non-expired) alive message that was sent. A byebye message is a multicast NOTIFY message e.g.

NOTIFY * HTTP/1.1
HOST: 239.255.255.250:1900
NT: urn:dial-multiscreen-org:service:dial:1
NTS: ssdp:byebye
USN: uuid:0175c106-5400-10f8-802d-b0a7374360b7::urn:dial-multiscreen-org:service:dial:1

Again a byebye message has a very similar structure to both an alive and update message only omitting the LOCATION header field and having a different NTS value.

Now that we have an understanding of what SSDP is let's get back to the solution.

Getting to know who's there 🔭

As SSDP communication is built on top of UDP, it isn't possible to use URLSession to send an M-Search message instead we need to read and write from a socket manually. I was tempted to dive in and write a simple socket layer to handle this, but the more I read up about the various network setups I would have to support, the more writing a socket layer started to look like an unforgiving task. Instead, I decided to introduce a 3rd-party dependency into the project: BlueSocket. BlueSocket will handle the nitty-gritty socket communication and free us up to focus on sending and receiving SSDP messages.

In the example project, CocoaPods is used to manage this dependency.

At the time of writing (April, 2019) the new(ish) Network framework doesn't support UDP multicasting.

Since I wrote this article, Apple has released iOS 14, which contained a range of user privacy improvements. One area of focus was around how an app accesses the local network. There are two changes that affect the below example; firstly, before accessing the local network, iOS will request permission from the user as it does for, e.g. accessing the camera and secondly, any app that wants to multicast on a network must also have the com.apple.developer.networking.multicast entitlement enabled - this entitlement needs to be requested from Apple. Without both the user permission and the entitlement, the below example won't be able to send multicast messages from a device (everything will still work on the simulator) - see this note for more details.

Before we get started, let's look at what we are going to build:

• SSDPSearchSessionConfiguration represents the configuration of an SSDP search session.
• SSDPSearchSession is responsible for triggering a search request, parsing and filtering received responses and passing back those services that match our search terms to whoever requested the search. A typical search session will write multiple search requests to the socket to try and overcome the unreliable nature of UDP.
• SSDPSearchSessionDelegate is a protocol that informs the delegator of any found services, any errors encounter and ultimately that the search session has ended.
• SocketControllerFactory is responsible for producing sockets controllers configured to use sockets that use a specific transport protocol.
• UDPSocketController is a wrapper around a Socket instance configured to use UDP. UDPSocketController helps to hide the messiness of socket communication.
• UDPSocketControllerDelegate a protocol that informs the delegator of any responses received and any errors encountered.
• SocketFactory is responsible for producing fully configured sockets that are ready to be written/read to/from.
• UDPSocket a Socket instance configured to use UDP. As Socket is from the BlueSocket dependency, Socket is wrapped in a protocol to limit the spread of BlueSocket in the project.
• SSDPServiceParser is responsible for parsing an SSDP response into Service instance. If the SSDP response is invalid, no service is created.
• SSDPService represents an SSDP service.

Don't worry if that doesn't all make sense yet, we will look into each class in greater depth below.

Now that we know where we are going let's start at the bottom with UDPSocket and work our way up.

Ideally when adding a 3rd-party dependency to our projects we want to hide that dependency behind a facade which should make removing that dependency easier by limiting it's spread in the project. In Swift we can use a protocol and an extension to wrap that 3rd-party dependency inside our facade:

enum UDPSocketError: Error {
    case addressCreationFailure
    case writeError(underlayingError: Error)
    case readError(underlayingError: Error)
}

protocol UDPSocketProtocol {
    func write(_ string: String, to host: String, on port: UInt) throws
    func readDatagram(into data: inout Data) throws
    func close()
}

extension Socket: UDPSocketProtocol {
    // 1
    func write(_ string: String, to host: String, on port: UInt) throws {
        guard let signature = self.signature, signature.socketType == .datagram, signature.proto == .udp else {
            fatalError("Only UDP sockets can use this method")
        }

        guard let address = Socket.createAddress(for: host, on: Int32(port)) else {
            throw(UDPSocketError.addressCreationFailure)
        }
        do {
            try write(from: string, to: address)
        } catch {
            throw(UDPSocketError.writeError(underlayingError: error))
        }
    }

    // 2
    func readDatagram(into data: inout Data) throws {
        guard let signature = self.signature, signature.socketType == .datagram, signature.proto == .udp else {
            fatalError("Only UDP sockets can use this method")
        }

        do {
            let (_,_) = try readDatagram(into: &data)
        } catch {
            throw(UDPSocketError.readError(underlayingError: error))
        }
    }
}

Socket does a lot more than what we need for our SSDP solution. UDPSocketProtocol reduces the range of tasks that can be performed on a Socket instance to only the three that we need: write, read and close. As well as reducing scope, UDPSocketProtocol also simplifies the interface of Socket by wrapping the Socket methods in our own methods.

1. The Socket type in BlueSocket can be used to create sockets with different configurations. In UDPSocketProtocol, we only want to cater for sockets configured that using UDP to send datagram messages, so in write(_:to:on:) a check is made to determine if this Socket instance has been configured for UDP and datagram - if it hasn't then a fatal error is thrown as this is a developer error. The write(from:to:) method on Socket takes an Address type which is defined in Socket - as mentioned above we want to limit the spread of BlueSocket in the project, so the write method defined in UDPSocketProtocol doesn't use Address but rather sticks with two String parameters host and port. In the extension, these parameters are used to create an Address instance which is then forwarded to the write(from:to:) method on Socket. If for some reason an Address instance can't be created, an exception is thrown. It's also possible for an exception to be thrown when writing to the socket, if an exception is thrown that exception is caught, wrapped inside an UDPSocketError case before a new exception is thrown.
2. Just like with write(_:to:on:), in readDatagram(into:) a check is made to ensure that we are dealing with a socket configured to UDP and datagram. The readDatagram(into:) method on Socket returns the number of bytes read as an Int and the address that sent those bytes as an Address. We aren't interested in either of those return values, so the readDatagram(into:) defined in SocketProtocol doesn't have a return type - our readDatagram(into:) just eats those details. If so some reason when reading from the socket an exception is thrown, this exception is caught, wrapped inside an UDPSocketError case before a new exception is thrown.

The close() on Socket fits our needs perfectly so we don't need to wrap that method.

UDPSocketProtocol does a good job of hiding Socket but it can be called on with all configurations of Socket when we really only intend for it be called on UDP sockets sending datagram messages so lets make making that socket configuration as easy as possible:

extension Socket {
    static func createUDPSocket() throws -> UDPSocketProtocol {
        return try Socket.create(type: .datagram, proto: .udp)
    }
}

You will need to import the Socket module into any class that uses Socket.

Now that we have UDPSocketProtocol, lets build Socket instances wrapped in it:

protocol SocketFactoryProtocol {
    func createUDPSocket() -> UDPSocketProtocol?
}

class SocketFactory: SocketFactoryProtocol {

    // MARK: - UDP

    func createUDPSocket() -> UDPSocketProtocol? {
        guard let socket = try? Socket.createUDPSocket() else {
            return nil
        }

        return socket
    }
}

The above factory, attempts to create a UDP socket, if successful that Socket instance is returned hidden behind the UDPSocketProtocol protocol; if unsuccessful, nil is returned.

SocketFactory conforms to SocketFactoryProtocol so that when it is injected into UDPSocketController - during unit testing we can replace the actual factory with a mock factory that conforms to that protocol. I use this technique throughout this example, so any other protocols that are named after a concrete type are for this purpose.

Now that we have our socket, let's try writing to it:

protocol UDPSocketControllerProtocol: AnyObject {
    var state: UDPSocketControllerState { get }

    func write(message: String)
    func close()
}

// 1
enum UDPSocketControllerState {
    case ready
    case active
    case closed

    var isReady: Bool {
        self == .ready
    }

    var isActive: Bool {
        self == .active
    }

    var isClosed: Bool {
        self == .closed
    }
}

class UDPSocketController: UDPSocketControllerProtocol {
    private(set) var state: UDPSocketControllerState = .ready

    private let socket: UDPSocketProtocol

    private let host: String
    private let port: UInt

    private let callbackQueue: OperationQueue
    private let socketWriterQueue = DispatchQueue(label: "com.williamboles.udpsocket.writer.queue",  attributes: .concurrent)

    // MARK: - Init

    // 2
    init?(host: String, port: UInt, socketFactory: SocketFactoryProtocol) {
        guard let socket = socketFactory.createUDPSocket() else {
            return nil
        }

        self.host = host
        self.port = port
        self.socket = socket
    }

    // MARK: - Write

    // 3
    func write(message: String) {
        guard !state.isClosed else {
            os_log(.info, "Attempting to write to a closed socket")
            return
        }

        state = .active

        write(message: message, on: socketWriterQueue)
    }

    // 4
    private func write(message: String, on queue: DispatchQueue) {
        queue.async {
            do {
                try self.socket.write(message, to: self.host, on: self.port)
            } catch {
                self.closeAndReportError(error)
            }
        }
    }

    // MARK: - Close

    private func closeAndReportError(_ error: Error) {
        close()
        os_log(.info, "Error received: \r%{public}@", error)
        //TODO: Implement reporting error
    }

    func close() {
        state = .closed
        socket.close()
    }
}

Let's look at what we did above:

1. A socket can be in 3 states: ready, active and closed - UDPSocketControllerState represents these states. As UDPSocketController interacts with its socket, it will move through states. This will ensure that we don't try to write to a socket that can't be written to.
2. To write a message to a socket, we need the host and port on the remote machine that we want to communicate with. The init'er of UDPSocketController accepts this host and port. It also accepts a socket factory instance which is used to create a UDP socket.
3. When writing to a socket, we only want to allow communication with a socket that hasn't been closed so the first action of the write(message:) is to check if state is in a closed state.
4. Writing to a socket can be a time-consuming operation, so the operation is pushed onto a background queue: socketWriterQueue. As a write operation can throw an exception, we wrap that operation in a do...catch whereupon catching an exception the socket is closed, and any error reported (we will implement this shortly).

UDPSocketController is designed to be tied to one host and port. If that host and port needs to be changed, then a new instance of UDPSocketController needs to be created.

Let's build a factory to produce socket controllers:

protocol SocketControllerFactoryProtocol {
    func createUDPSocketController(host: String, port: UInt, socketFactory: SocketFactoryProtocol) -> UDPSocketControllerProtocol?
}

class SocketControllerFactory: SocketControllerFactoryProtocol {

    // MARK: - UDP

    func createUDPSocketController(host: String, port: UInt, socketFactory: SocketFactoryProtocol) -> UDPSocketControllerProtocol? {
        UDPSocketController(host: host, port: port, socketFactory: socketFactory)
    }
}

Now that we have a socket controller and a socket to write to let's look at how to give it an M-Search message to send.

3 pieces of information are required to send an M-Search message:

1. Search target (ST).
2. The IP address and port (HOST).
3. Maximum wait response time (MX).

These pieces of information can be represented as:

struct SSDPSearchSessionConfiguration {
    let searchTarget: String
    let host: String
    let port: UInt
    let maximumWaitResponseTime: TimeInterval

    // MARK: - Init

    init(searchTarget: String, host: String, port: UInt, maximumWaitResponseTime: TimeInterval) {
        assert(maximumWaitResponseTime >= 1 && maximumWaitResponseTime <= 1 5 5, "maximumwaitresponsetime should be between and (inclusive)") self.searchtarget="searchTarget" self.host="host" self.port="port" self.maximumwaitresponsetime="maximumWaitResponseTime" } }< code>

SSDPSearchSessionConfiguration has a custom initialiser to allow for an assertion to be performed on the value of maximumWaitResponseTime (which needs to be between 1 and 5 inclusive). I believe having to manually write this custom initialiser is a price worth paying to allow for quicker feedback during development if an invalid value is passed in (by causing the app to crash).

While it's possible to send unicast M-Search messages, here we are only interested in sending multicast messages so to make things easier let's add a small factory method to return a preconfigured multicast SSDPSearchSessionConfiguration instance:

extension SSDPSearchSessionConfiguration {

    static func createMulticastConfiguration(forSearchTarget searchTarget: String, maximumWaitResponseTime: TimeInterval = 3) -> SSDPSearchSessionConfiguration {
        let configuration = SSDPSearchSessionConfiguration(searchTarget: searchTarget, host: "239.255.255.250", port: 1900, maximumWaitResponseTime: maximumWaitResponseTime)

        return configuration
    }
}

Setting the searchTarget to ssdp:all should cause all root devices to respond with their full range of SSDP services.

With this configuration it is possible to build a simple searcher class to drive any UDPSocketController instance:

protocol SSDPSearchSessionProtocol {
    func startSearch()
    func stopSearch()
}

class SSDPSearchSession: SSDPSearchSessionProtocol {
    private let socketController: UDPSocketControllerProtocol
    private let configuration: SSDPSearchSessionConfiguration

    // 1
    private lazy var mSearchMessage = {
        // Each line must end in `\r\n`
        return "M-SEARCH * HTTP/1.1\r\n" +
            "HOST: \(configuration.host):\(configuration.port)\r\n" +
            "MAN: \"ssdp:discover\"\r\n" +
            "ST: \(configuration.searchTarget)\r\n" +
            "MX: \(Int(configuration.maximumWaitResponseTime))\r\n" +
        "\r\n"
    }()

    // MARK: - Init

    // 2
    init?(configuration: SSDPSearchSessionConfiguration, socketControllerFactory: SocketControllerFactoryProtocol = SocketControllerFactory()) {
        guard let socketController = socketControllerFactory.createUDPSocketController(host: configuration.host, port: configuration.port, socketFactory: SocketFactory()) else {
            return nil
        }
        self.socketController = socketController
        self.configuration = configuration
    }

    deinit {
        stopSearch()
    }

    // MARK: - Search

    // 3
    func startSearch() {
        os_log(.info, "SSDP search session starting")
        writeMessageToSocket(mSearchMessage)
    }

    func stopSearch() {
        os_log(.info, "SSDP search session stopping")
        close()
    }

    // MARK: - Close

    // 4
    private func close() {
        if socketController.state.isActive {
            socketController.close()
        }
    }

    // MARK: - Write

    private func writeMessageToSocket(_ message: String) {
        os_log(.info, "Writing to socket: \r%{public}@", message)
        socketController.write(message: message)
    }
}

SSDPSearchSession above:

1. Configures the M-Search message using the SSDPSearchSessionConfiguration instance.
2. Creates an UDPSocketController instance.
3. Writes the M-Search message to the socket controller.
4. Closes the socket controller if it's active.

With the M-Search message you may have noticed the \r\n character sequence at the end of each line - don't be tempted to remove this from the M-Search message as the \r\n sequence is part of the protocol spec.

SSDPSearchSession is designed to be a single-use instance - once stopSearch() is called and the socket closed, a new instance of SSDPSearchSession is needed to perform another search.

Don't forget to add import os to get the os_log statements to compile.

Calling startSearch() should cause an M-Search message to be written to the network however as grizzly, well-travelled developers we know that trusting our code to do something without testing it, is a recipe for disappointment 😞. To test that this message is being written to the network, we can snoop on our network traffic using tcpdump.

To test on the simulator, open your terminal and run:

sudo tcpdump -vv -A -s 0 'port 1900 and host 239.255.255.250 and udp'

-vv: verbose output.
-A: print each packet in ASCII.
-s 0: sets the amount of captured data for each frame. Using 0 sets the amount to the default: 65535 bytes - we explicitly set this here for backwards compatibility with recent older versions of tcpdump.
'port 1900 and host 239.255.255.250 and udp' is using Berkeley Packet Filter (BPF) syntax to filter what traffic we see.

The above command will capture all the UDP traffic using host 239.255.255.250 on port 1900 (i.e. SSDP traffic) on your local machine.

If you are using a VPN, you may need to disable it to see anything in the console.

Testing on a device is slightly more difficult. We need to tell tcpdump which device to snoop on by creating a remote virtual interface using rvictl:

Connect your device and grab its UDID, then open terminal and run:

rvictl -s {UDID} && sudo tcpdump -vv -A -s 0 -i rvi0 'port 1900 and host 239.255.255.250 and udp'

Replacing {UDID} with the UDID of the device.

You should see similar traffic to what you would if testing on the simulator.

Run rvictl -x {UDID} to stop the remote virtual interface and Ctrl-C to kill tcpdump.

Now that we have confirmation that M-Search messages are being sent, let's build the functionality to parse any responses we may receive. First UDPSocketController needs to read anything sent to the socket:

class UDPSocketController: UDPSocketControllerProtocol {
    //Omitted other properties

    // 1
    private let socketListeningQueue = DispatchQueue(label: "com.williamboles.udpsocket.listen.queue",  attributes: .concurrent)

    //Omitted other methods

    // MARK: - Write

    func write(message: String) {
        //Omitted code

        // 2
        let shouldStartListening = state.isReady
        state = .active

        if shouldStartListening {
           startListening(on: socketListeningQueue)
        }

        //Omitted code
    }

    // MARK: - Listen

    // 3
    private func startListening(on queue: DispatchQueue) {
        queue.async {
            do {
                repeat {
                    var data = Data()
                    try self.socket.readDatagram(into: &data) //blocking call
                    self.reportResponseReceived(data)
                } while self.state.isActive
            } catch {
                if self.state.isActive { // ignore any errors for non-active sockets
                    self.closeAndReportError(error)
                }
            }
        }
    }

    private func reportResponseReceived(_ data: Data) {
        os_log(.info, "Response received: \r%{public}@", response)
        //TODO: Implement reporting response received
    }

    //Omitted other methods
}

With the above changes, UDPSocketController is now able to read from its socket.

1. Reading from a BlueSocket socket configured to read datagram messages using UDP is a blocking call - any thread that readDatagram(into:) is called on will be blocked at that line until there is data there to be read. To avoid the app from freezing, reading from the socket must be pushed off the caller queue and onto a background queue: socketWriterQueue.
2. We only need to configure the socket to listen once - on the first write.
3. Once a response is received, that response is converted into a string and (for the moment) logged. Finally, if the controller is still listening for responses, the socket is polled again. As a read operation can throw an exception, we wrap that operation in a do...catch whereupon catching an exception the socket is closed, and any error reported (we will implement this shortly). An interesting point to note is that closing a socket that is being polled will throw an exception. So when an exception is thrown during polling, we only care about that exception if the session is listening.

If you have devices on your network that support SSDP, you should start to see responses in the Console when running the above code. However, if you don't, it's possible to fake a response using netcat. You will need to extract the host and port from the M-Search request via tcpdump and run the following command:

echo "HTTP/1.1 200 OK\r\nCache-Control: max-age=3600\r\nST: urn:dial-multiscreen-org:service:dial:1\r\nUSN: uuid:0175c106-5400-10f8-802d-b0a7374360b7::urn:dial-multiscreen-org:service:dial:1\r\nExt: \r\nServer: Roku UPnP/1.0 MiniUPnPd/1.4\r\nLOCATION: http://192.168.1.104:8060/\r\n\r\n" | nc -u {host} {port}

Replacing {host} {port} with the extracted values.

The above command will send a response pretending to be a Roku set-top box.

Now that it is possible to read and write from a socket, lets pass any responses (and any errors) out of our socket controller:

protocol UDPSocketControllerDelegate: AnyObject {
    func controller(_ controller: UDPSocketControllerProtocol, didReceiveResponse response: Data)
    func controller(_ controller: UDPSocketControllerProtocol, didEncounterError error: Error)
}
protocol UDPSocketControllerProtocol: AnyObject {
    //Omitted other properties
    var delegate: UDPSocketControllerDelegate? { get set }

    //Omitted methods
}

class UDPSocketController: UDPSocketControllerProtocol {
    //Omitted other properties

    weak var delegate: UDPSocketControllerDelegate?

    private let callbackQueue: OperationQueue

    init?(host: String, port: UInt, socketFactory: SocketFactoryProtocol, callbackQueue: OperationQueue) {
        //Omitted rest of method

        self.callbackQueue = callbackQueue
    }

    //Omitted other methods

    private func reportResponseReceived(_ data: Data) {
        callbackQueue.addOperation {
           self.delegate?.controller(self, didReceiveResponse: data)
        }
    }

    private func closeAndReportError(_ error: Error) {
        close()
        callbackQueue.addOperation {
            self.delegate?.controller(self, didEncounterError: error)
        }
    }
}

With the above changes, SSDPSearchSession can now add itself as the delegate of UDPSocketControllerDelegate. To make the communication via that delegate more predictable when creating that socket, the thread that the communication will happen on is passed in.

Let's update the SocketControllerFactory to support the callbackQueue:

protocol SocketControllerFactoryProtocol {
    func createUDPSocketController(host: String, port: UInt, socketFactory: SocketFactoryProtocol, callbackQueue: OperationQueue) -> UDPSocketControllerProtocol?
}

class SocketControllerFactory: SocketControllerFactoryProtocol {

    // MARK: - UDP

    func createUDPSocketController(host: String, port: UInt, socketFactory: SocketFactoryProtocol, callbackQueue: OperationQueue) -> UDPSocketControllerProtocol? {
        UDPSocketController(host: host, port: port, socketFactory: socketFactory, callbackQueue: callbackQueue)
    }
}

Now, let's update SSDPSearchSession to be the delegate of UDPSocketControllerDelegate:

class SSDPSearchSession: SSDPSearchSessionProtocol, UDPSocketControllerDelegate {
    //Omitted properties

    init?(configuration: SSDPSearchSessionConfiguration, socketControllerFactory: SocketControllerFactoryProtocol = SocketControllerFactory()) {
        guard let socketController = socketControllerFactory.createUDPSocketController(host: configuration.host, port: configuration.port, socketFactory: SocketFactory(), callbackQueue: .main) else {
            return nil
        }

        //Omitted other assignments

        self.socketController.delegate = self
    }

    //Omitted other methods

    // MARK: - UDPSocketControllerDelegate

    func controller(_ controller: UDPSocketControllerProtocol, didReceiveResponse response: Data) {
        os_log(.info, "Received response: \r%{public}@", response)
        //TODO: Implement
    }

    func controller(_ controller: UDPSocketControllerProtocol, didEncounterError error: Error) {
        os_log(.info, "Encountered socket error: \r%{public}@", error.localizedDescription)
        close()
        //TODO: Implement
    }
}

SSDPSearchSession can now receive responses from its UDPSocketController, let's turn those responses into something useful:

struct SSDPService {
    let cacheControl: Date
    let location: URL
    let server: String
    let searchTarget: String
    let uniqueServiceName: String
    let otherHeaders: [String: String]
}

An M-Search response has mandatory and optional/custom header fields. In SSDPService, the mandatory header fields are mapped to named properties, and the optional/custom header fields are mapped to the otherHeaders property. Each optional/custom header field is represented as a dictionary with the field name as the dictionary key and field's value as dictionary value.

To get an SSDPService instance, it needs to be parsed:

private enum SSDPServiceResponseKey: String {
    case cacheControl = "CACHE-CONTROL"
    case location = "LOCATION"
    case server = "SERVER"
    case searchTarget = "ST"
    case uniqueServiceName = "USN"
}

protocol SSDPServiceParserProtocol {
    func parse(_ data: Data) -> SSDPService?
}

class SSDPServiceParser: SSDPServiceParserProtocol {
    private let dateFactory: DateFactoryProtocol

    // Init

    init(dateFactory: DateFactoryProtocol =  DateFactory()) {
        self.dateFactory = dateFactory
    }

    // MARK: - Parse

    func parse(_ data: Data) -> SSDPService? {
        guard let responseString = String(data: data, encoding: .utf8) else {
            return nil
        }

        os_log(.info, "Received SSDP response: \r%{public}@", responseString)

        // 1
        var responseDict = parseResponseIntoDictionary(responseString)

        // 2
        guard let cacheControl = parseCacheControl(responseDict[SSDPServiceResponseKey.cacheControl.rawValue]),
            let location = parseLocation(responseDict[SSDPServiceResponseKey.location.rawValue]),
            let server = responseDict[SSDPServiceResponseKey.server.rawValue],
            let searchTarget = responseDict[SSDPServiceResponseKey.searchTarget.rawValue],
            let uniqueServiceName = responseDict[SSDPServiceResponseKey.uniqueServiceName.rawValue] else {
                return nil
        }

        // 3
        responseDict.removeValue(forKey: SSDPServiceResponseKey.cacheControl.rawValue)
        responseDict.removeValue(forKey: SSDPServiceResponseKey.location.rawValue)
        responseDict.removeValue(forKey: SSDPServiceResponseKey.server.rawValue)
        responseDict.removeValue(forKey: SSDPServiceResponseKey.searchTarget.rawValue)
        responseDict.removeValue(forKey: SSDPServiceResponseKey.uniqueServiceName.rawValue)

        // 4
        return SSDPService(cacheControl: cacheControl, location: location, server: server, searchTarget: searchTarget, uniqueServiceName: uniqueServiceName, otherHeaders: responseDict)
    }

    private func parseResponseIntoDictionary(_ response: String) -> [String: String] {
        var elements = [String: String]()
        for element in response.split(separator: "\r\n") {
            let keyValuePair = element.split(separator: ":", maxSplits: 1)
            guard keyValuePair.count == 2 else {
                continue
            }

            let key = String(keyValuePair[0]).uppercased().trimmingCharacters(in: CharacterSet.whitespacesAndNewlines)
            let value = String(keyValuePair[1]).trimmingCharacters(in: CharacterSet.whitespacesAndNewlines)

            elements[key] = value
        }

        return elements
    }

    private func parseCacheControl(_ value: String?) -> Date? {
        guard let cacheControlRange = value?.range(of: "[0-9]+$", options: .regularExpression),
            let cacheControlString = value?[cacheControlRange],
            let cacheControlTimeInterval = TimeInterval(cacheControlString) else {
                return nil
        }

        let currentDate = dateFactory.currentDate()
        return currentDate.addingTimeInterval(cacheControlTimeInterval)
    }

    private func parseLocation(_ value: String?) -> URL? {
        guard let urlString = value,
            let url = URL(string: urlString) else {
                return nil
        }

        return url
    }
}

SSDPServiceParser above takes the string response and attempts to parse it into an SSDPService instance by:

1. Splitting the string into a dictionary using \r\n to determine fields and : to determine key and value pairs.
2. The response dictionary is checked to ensure that all mandatory fields are present. If any of the mandatory fields are missing, the SSDP response is considered invalid, and nil is returned.
3. As an SSDP response can contain non-mandatory fields, the mandatory fields are stripped from the response dictionary so leaving the only the non-mandatory fields present.
4. An SSDPService is created using the mandatory and non-mandatory fields.

I could have combined SSDPService and SSDPServiceParser into the one type with the init'er accepting the string response, but I think having an independent parser makes the code easier to read.

The DateFactory is used to make unit-testing this parser possible:

protocol DateFactoryProtocol {
    func currentDate() -> Date
}

class DateFactory: DateFactoryProtocol {

    // MARK: - Current

    func currentDate() -> Date {
        return Date()
    }
}

Let's update SSDPSearchSession to use SSDPServiceParser:

class SSDPSearchSession: SSDPSearchSessionProtocol, UDPSocketControllerDelegate {
    //Omitted other properties

    private let parser: SSDPServiceParserProtocol

    // MARK: - Init

    init?(configuration: SSDPSearchSessionConfiguration, socketControllerFactory: SocketControllerFactoryProtocol = SocketControllerFactory(), parser: SSDPServiceParserProtocol = SSDPServiceParser()) {
        //Omitted other assignments
        self.parser = parser
    }

    //Omitted methods

    // MARK: - UDPSocketControllerDelegate

    func controller(_ controller: UDPSocketControllerProtocol, didReceiveResponse response: Data) {
        guard !response.isEmpty,
            let service = parser.parse(response) else {
                return
        }

        os_log(.info, "Received service \r%{public}@", service)
    }

    //Omitted methods
}

Once you start parsing responses, you will notice that some root devices respond to any M-Search message they receive rather than just those discovery requests that match one of their services. To counter these chatty root devices we need to ensure that the parsed SSDPService instance is the searched-for-service:

class SSDPSearchSession {
    //Omitted properties & methods

    // MARK: - UDPSocketControllerDelegate

    func controller(_ controller: UDPSocketControllerProtocol, didReceiveResponse response: Data) {
       guard !response.isEmpty,
           let service = parser.parse(response),
           searchedForService(service) else {
               return
       }

       os_log(.info, "Received service \r%{public}@", service)
    }

    private func searchedForService(_ service: SSDPService) -> Bool {
       return service.searchTarget.contains(configuration.searchTarget) || configuration.searchTarget == "ssdp:all"
    }

    //Omitted methods
}

If the search target is set to the special ssdp:all value, all services that respond are treated as valid.

SSDPSearchSession is doing good work but isn't able to share the fruits of its labour with anyone. Let's add in a delegate to tell interested parties how things are going with the search:

protocol SSDPSearchSessionDelegate: AnyObject {
    func searchSession(_ searchSession: SSDPSearchSession, didFindService service: SSDPService)
    func searchSession(_ searchSession: SSDPSearchSession, didEncounterError error: SSDPSearchSessionError)
    func searchSessionDidStopSearch(_ searchSession: SSDPSearchSession, foundServices: [SSDPService])
}

enum SSDPSearchSessionError: Error {
    case searchAborted(Error)
}

class SSDPSearchSession: SSDPSearchSessionProtocol, UDPSocketControllerDelegate {
    //Omitted properties

    // 1
    private var servicesFoundDuringSearch = [SSDPService]()

    weak var delegate: SSDPSearchSessionDelegate?

    //Omitted methods

    func stopSearch() {
        os_log(.info, "SSDP search session stopping")
        close()

        // 2
        delegate?.searchSessionDidStopSearch(self, foundServices:servicesFoundDuringSearch)
    }

    //Omitted methods

    // MARK: - UDPSocketControllerDelegate

    func controller(_ controller: UDPSocketControllerProtocol, didReceiveResponse response: Data) {
        guard !response.isEmpty,
            let service = parser.parse(response),
            searchedForService(service) else {
                return
        }

        os_log(.info, "Received a valid service response")

        servicesFoundDuringSearch.append(service)

        // 3
        delegate?.searchSession(self, didFindService: service)
    }

    func controller(_ controller: UDPSocketControllerProtocol, didEncounterError error: Error) {
        os_log(.info, "Encountered socket error: \r%{public}@", error.localizedDescription)

        // 4
        let wrappedError = SSDPSearchSessionError.searchAborted(error)
        delegate?.searchSession(self, didEncounterError: wrappedError)
        close()
    }

    //Omitted methods
}

With the above changes, we now:

1. Store all valid services that have been received during that search session.
2. Inform the delegate when the search has been stopped, returning all valid services found.
3. Inform the delegate when a valid service has been found.
4. Wrap any received error in an SSDPSearchSessionError error and inform the delegate of that error.

It's interesting to note that searchSession(_:, didFindService:) is called as soon as a valid SSDPService instance is parsed rather than waiting for all services to be parsed. This will allow the app to respond immediately to any found services.

An alternative to delegation would have been to pass a closure into startSearch(). In fact, using a closure was my preferred option to begin with. However, after experimenting, I felt that having one closure handling three possible states resulted in code that was very busy and that readability suffered because of this.

Every M-Search message contains an MX value that represents the maximum time a service can wait before responding. When this time has elapsed, it can be confidently assumed that all services that can respond, have responded. Meaning that MX can be used as a timeout for the search session:

class SSDPSearchSession: SSDPSearchSessionProtocol, UDPSocketControllerDelegate {
    //Omitted other properties

    private let searchTimeout: TimeInterval

    private var timeoutTimer: Timer?

    // MARK: - Init

    init?(configuration: SSDPSearchSessionConfiguration, socketControllerFactory: SocketControllerFactoryProtocol = SocketControllerFactory(), parser: SSDPServiceParserProtocol = SSDPServiceParser()) {
        //Omitted other assignments

        self.searchTimeout = configuration.maximumWaitResponseTime + 0.1
    }

    //Omitted methods

    // MARK: - Search

    func startSearch() {
        //Omitted code

        timeoutTimer = Timer.scheduledTimer(withTimeInterval: searchTimeout, repeats: false, block: { [weak self] (timer) in
            self?.searchTimedOut()
        })
    }

    private func searchTimedOut() {
        os_log(.info, "SSDP search timed out")
        stopSearch()
    }

    //Omitted methods

    // MARK: - Close

    private func close() {
        timeoutTimer?.invalidate()
        timeoutTimer = nil

        //Omitted code
    }

    //Omitted methods
}

With the above changes, the search session is ended after maximumWaitResponseTime seconds causing the socket to be closed. The more eagle-eyed reader may have spotted that timeoutTimer has a trigger time that is 0.1 seconds longer than maximumWaitResponseTime - this is to allow any responses from root devices that waited the full maximumWaitResponseTime seconds before responding to reach the app and be processed before the search session is ended.

Some people are harder to talk to than others

If you have been combining the above code snippets into a working project, you will now be able to search for SSDP services and parse any response received. However, every so often, you may notice that an SSDP service that you know exists on the network does not respond.

🤔

As described above, SSDP uses the unreliable UDP transportation protocol (because UDP supports multicasting). UDP is unreliable because there is no acknowledgement if a message made it to its destination. This means there is no way of knowing if a service hasn't responded because the message was dropped along the way or that the service is no longer available. Unreliability isn't a great characteristic for a discovery service to have. While not foolproof, it is possible to increase the reliability of an SSDP based discovery service by sending multiple M-Search messages over the lifecycle of an SSDPSearchService instance. Sending multiple M-Search messages will increase the chances that at least one message makes it to each root device on the network. For this to be possible, our SSDPSearchService instance must exist longer than the MX value before timing out:

struct SSDPSearchSessionConfiguration {
    //Omitted properties

    let maximumSearchRequestsBeforeClosing: UInt

    // MARK: - Init

    init(searchTarget: String, host: String, port: UInt, maximumWaitResponseTime: TimeInterval, maximumSearchRequestsBeforeClosing: UInt) {
        //Omitted other assignments

        self.maximumSearchRequestsBeforeClosing = maximumSearchRequestsBeforeClosing
    }
}

extension SSDPSearchSessionConfiguration {

    static func createMulticastConfiguration(forSearchTarget searchTarget: String, maximumWaitResponseTime: TimeInterval = 3, maximumSearchRequestsBeforeClosing: UInt = 3) -> SSDPSearchSessionConfiguration {
        let configuration = SSDPSearchSessionConfiguration(searchTarget: searchTarget, host: "239.255.255.250", port: 1900, maximumWaitResponseTime: maximumWaitResponseTime, maximumSearchRequestsBeforeClosing: maximumSearchRequestsBeforeClosing)

        return configuration
    }
}

maximumSearchRequestsBeforeClosing will control how many M-Search messages are sent before the search session is closed. maximumSearchRequestsBeforeClosing needs to be at least 1, or no M-Search will be sent.

An alternative to using a count property would have been to use a total-duration property. However, the total-duration approach would have required the config-maintainer to ensure that this timeout property was always a multiple of the maximumWaitResponseTime. As having a value which isn't a multiple would result in the situation where the session was stopped before the last M-Search iteration's maximumWaitResponseTime had expired - potentially resulting in ignored responses because the root devices waited until the maximumWaitResponseTime value before responding. It's easy to imagine this mistake happening. By expressing the timeout as the value to multiply maximumWaitResponseTime by, we ensure that this error scenario can never happen.

class SSDPSearchSession: SSDPSearchSessionProtocol, UDPSocketControllerDelegate {
    //Omitted properties

    private var searchRequestTimer: Timer?

    //Omitted methods

    // MARK: - Init

    init?(configuration: SSDPSearchSessionConfiguration, socketControllerFactory: SocketControllerFactoryProtocol = SocketControllerFactory(), parser: SSDPServiceParserProtocol = SSDPServiceParser()) {
        //Omitted other assignments

        // 1
        self.searchTimeout = (TimeInterval(configuration.maximumSearchRequestsBeforeClosing) * configuration.maximumWaitResponseTime) + 0.1
    }

    // MARK: - Search

    func startSearch() {
        // 2
        guard configuration.maximumSearchRequestsBeforeClosing > 0 else {
            delegate?.searchSessionDidStopSearch(self, foundServices: servicesFoundDuringSearch)
            return
        }

        os_log(.info, "SSDP search session starting")
        sendMSearchMessages()

        //Omitted code
    }

    //Omitted methods

    // MARK: - Close

    private func close() {
        //Omitted code

        // 3
        searchRequestTimer?.invalidate()
        searchRequestTimer = nil

        //Omitted code
    }

    // MARK: Write

    private func sendMSearchMessages() {
        let message = mSearchMessage

        // 4
        if configuration.maximumSearchRequestsBeforeClosing > 1 {
            let window = searchTimeout - configuration.maximumWaitResponseTime
            let interval = window / TimeInterval((configuration.maximumSearchRequestsBeforeClosing - 1))

            searchRequestTimer = Timer.scheduledTimer(withTimeInterval: interval, repeats: true, block: { [weak self] (timer) in
                self?.writeMessageToSocket(message)
            })
        }
        writeMessageToSocket(message)
    }

    //Omitted methods
}

With the above changes, a new M-Search message is sent every maximumWaitResponseTime seconds until the desired number of messages have been sent.

1. Increased the total search time for multiple M-Search message writes.
2. Check that maximumSearchRequestsBeforeClosing is greater than 0 and return if it isn't.
3. Invalidate new search request timer.
4. Send multiple M-Search messages inside the maximum send window.

However, while making SSDPSearchSession a more reliable discovery service, sending multiple M-Search messages creates a new problem. In ideal network conditions, the same SSDP service would respond to each sent M-Search message. If these responses were just blindly passed to the delegate, the SSDPSearchService instance would in effect be spamming that delegate with the same service multiple times. Thankfully each service parsed is already stored in the servicesFoundDuringSearch array (to be used when searchSessionDidStopSearch(_:, foundServices:) is called), so to prevent becoming a spammer a check can be made to determine if an SSDPService instance representing the same service has already been passed to the delegate:

class SSDPSearchSession: SSDPSearchSessionProtocol, UDPSocketControllerDelegate {
    //Omitted properties and methods

    // MARK: - UDPSocketControllerDelegate

    func controller(_ controller: UDPSocketControllerProtocol, didReceiveResponse response: Data) {
        guard !response.isEmpty,
            let service = parser.parse(response),
            searchedForService(service),
            // 1
            !servicesFoundDuringSearch.contains(service) else {
                return
        }

        //Omitted code
    }

    //Omitted methods
}

With the above change:

1. A check is made to see if servicesFoundDuringSearch already contains the service that has been parsed.

In order to get the above code to work, SSDPService needs to conform to Equatable:

struct SSDPService: Equatable {
    //Omitted properties

    // MARK: - Equatable

    static func == (lhs: SSDPService, rhs: SSDPService) -> Bool {
        return lhs.location == rhs.location &&
            lhs.server == rhs.server &&
            lhs.searchTarget == rhs.searchTarget &&
            lhs.uniqueServiceName == rhs.uniqueServiceName
    }
}

The above custom equality check excludes cacheControl as this will change with each response because it's based on the date when the service was parsed.

🎉🎉🎉

And that's everything you need for discovering what SSDP services are available on a network - congratulations.

Happy to get to know everyone 🎯

Social situations can be tricky. It's easy to think that you don't have anything of value to add and to allow that thought to leave you alone in the corner. However, with a little bit of effort (and bravery), you can reach out and get to know new people.

This is just as true for your app.

SSDP is a lightweight, widely supported protocol that makes it straightforward to discover services on a network. It has a few gotchas but provided that we treat it with care and don't 100% trust any root devices to behave as expected, SSDP can be a useful tool to have in the toolbox.

To see the above code snippets together in a working example, head over to the repo and clone the project.

There are very few certainties in app development, but one is that once your app is released it will change in unexpected ways. And no matter how flexible your architecture is, inevitably one of those changes will be a breaking change. Perhaps the most important breaking changes involve the user's data. If your app loses or corrupts user data you can expect at least some reputational damage and if the loss is severe enough you can end up doing your competitors marketing for them by turning your users into their users. If you have any doubt about the impact of data loss imagine how you would feel if a game you had been playing was updated and deleted your recently hard earned thingamabob - all that time and effort lost through no fault of yours. And that's just a game, now imagine how your users would feel when your far-more-important app starts losing their data.

In our iOS apps we often store these thingamabobs in Core Data. The structure of which is defined by a model/schema - a set of entities with attributes and relationships. To allow changes to be made to this model to meet your apps changing needs, Core Data has a built-in mechanism to migrate data from one model structure to another model structure.

In this post, we are going to build a simple system to manipulate the inbuilt Core Data migration mechanism to make migrations simpler and so reduce the risk of losing your user's data.

This post will gradually build up to a working example however if you're on a tight deadline and/or there is murderous look creeping into your manager's eyes 😡, then head on over to the completed example and take a look at CoreDataManager, CoreDataMigrator, CoreDataMigrationStep and CoreDataMigrationVersion to see how things end up.

The Migration Process

As mentioned above, Core Data allows the model to evolve through model versions. Typically a model version's changeable lifecycle (when it can be changed) is from when that version is created until it's released as an app update. Once released, a version is effectively "frozen" - any further changes made to that version would result in an app crash upon launch. To change an already released model, you need to create a new version of that model and migrate users from the old version to the new version. Thankfully, Core Data has a builtin migration system.

Migrations can be handled using one of two techniques:

1. Lightweight Migration - when Core Data can automatically infer how the migration should happen and creates the mapping model on the fly.
2. Standard Migration - when Core Data cannot infer how the migration should happen and so we must write a custom migration by providing a mapping model (xcmappingmodel) and/or a migration policy (NSEntityMigrationPolicy).

By default, Core Data will attempt to perform a migration automatically when it detects a mismatch between the model used in the persistent store and the bundle's current model. When this happens, Core Data will first attempt to perform a Standard migration by searching in the app's bundle for a mapping model that maps from the persistent store model to the current bundle model. If a custom mapping model isn't found, Core Data will then attempt to perform a Lightweight migration. If neither form of migration is possible an exception is thrown.

If you are using NSPersistentContainer, Lightweight migrations are enabled by default, however if you are still directly setting up the NSPersistentStoreCoordinator then you need to enable Lightweight migrations by passing in an options dictionary with both NSMigratePersistentStoresAutomaticallyOption and NSInferMappingModelAutomaticallyOption set to true when loading the persistent store.

These automatic migrations are performed as one-step migrations; directly from the source to destination model. So if we support 4 model versions, mapping models would exist for 1 to 4, 2 to 4 and 3 to 4. While this is the most efficient migration approach from a device performance point-of-view, it can actually be quite wasteful from a development point-of-view. For example if we added a new model version (5) we would need to create 4 new mapping models from 1 to 5, 2 to 5, 3 to 5 and 4 to 5 which as you can see doesn't reuse any of the mapping models for migrating to version 4. With a one-step migration approach, each newly added model version requires n-1 mapping models (where n is the number of supported model versions) to be created.

It's possible to reduce the amount of work required to perform a Core Data migration by disabling automatic migrations and so break the requirement to perform migrations in one-step. With a manual migration approach, we can perform the full migration by chaining multiple smaller migrations together. As the full migration is split into smaller migrations when adding a new model version we only need to handle migrating to the new model version from its direct predecessor rather than all it's predecessors e.g. 4 to 5 because we can reuse the existing 1 to 2, 2 to 3 and 3 to 4 mapping models. Not only do manual migrations reduce the amount of work involved they also help to reduce the complexity of the migration as the conceptional distance between the source and destination version is reduced when compared to one-step migrations i.e. version 4 is much nearer to the structure of version 5 than version 1 is - this should make it easier spot any issues with the migration.

Progressive migrations

In order to support progressive migrations we'll need to answer a few questions:

1. Which model version comes after version X?
2. What is a migration step?
3. How can we combine the migration steps into a migration path?
4. How do we trigger a migration?

These questions will be answered with the help of 4 separate types:

1. CoreDataMigrationVersion
2. CoreDataMigrationStep
3. CoreDataMigrator
4. CoreDataManager

These types will come together in the following class structure (along with several helper extensions):

Don't worry if that doesn't all make sense yet, we will look into each type in greater depth below.

Which model version comes after version X?

Each CoreDataMigrationVersion instance will represent a Core Data model version. As each Core Data model version is unique and known at compile time they can be perfectly represented as enum cases, with the raw value of each case being the Core Data model name:

enum CoreDataMigrationVersion: String, CaseIterable {
    case version1 = "CoreDataMigration_Example"

    // MARK: - Current

    static var current: CoreDataMigrationVersion {
        guard let current = allCases.last else {
            fatalError("no model versions found")
        }

        return current
    }

    // MARK: - Migration

    func nextVersion() -> CoreDataMigrationVersion? {
        switch self {
        case .version1:
            return nil
        }
    }
}

Migrations are often concerned with what the latest model version is - the static current property allows easy access to this version. Before Swift 4.2 we would probably have had to hardcode this property to one case which would then lead to bugs if we forgot to update that property when adding a new version. However in Swift 4.2 we got the CaseIterable protocol which makes it possible to get an array of the cases in an enum in the order they were defined in via the allCases property. This means that to get the latest model version should be as simple as calling last on the allCases array - no need to hardcode anything.

In CoreDataMigrationVersion the nextVersion() method is where the real work happens as it determines which (if any) version comes after self.

You may be thinking:

"Why bother with nextVersion() when we can just always choose the next enum case?"

If you are reading this post before performing your first migration I congratulate you on your:

1. Excellent taste in selecting blog posts.
2. Organisational ability.

However, I'm guessing it's more likely that you've found this post having already performed a number of migrations and been hit by the inherent scaling issue with the default one-step migration approach. If you are in the latter camp then you will have already implemented one-step migrations having configured various mapping models and maybe even written a migration policy or two. Instead of throwing all that work away we can use it and tie it into the new progressive approach. In a hypothetical project that had 6 model versions which until model version 4 used the one-step migration approach before switching over to the progressive migration approach, then nextVersion would look like:

func nextVersion() -> CoreDataMigrationVersion? {
    switch self {
    case .version1, .version2, .version3:
        return .version4
    case .version4:
        return .version5
    case .version5:
        return .version6
    case .version6:
        return nil
    }
}

In the above code snippet, version1, version2 and version3 migrate directly to version4 and then version4 and version5 migrate to their direct successor. As you can see both these migration approaches can co-exist very happily with each other.

Even if you don't have any existing migrations, it's possible that at some point in the future a broken model version is released that corrupts your user's data upon migration. In order to minimise to the impact of this mistake, nextVersion could be configured to bypass that broken model version so that any currently unaffected user are never impacted:

func nextVersion() -> CoreDataMigrationVersion? {
    switch self {
    case .version1:
        return .version2
    case .version2:
        return .version4 // skipping corrupted .version3
    case .version3:
        return .version4
    case .version4:
        return nil
    }
}

Both these issues are easily bypassed using nextVersion() without adding too much complexity to the overall solution.

What is a migration step?

A migration happens between 2 model versions by having a mapping from the entities, attributes and relationships of the source model and their counterpoints in the destination model. As such CoreDataMigrationStep needs to contain 3 properties:

1. Source version model.
2. Destination version model.
3. Mapping model.

struct CoreDataMigrationStep {

    let sourceModel: NSManagedObjectModel
    let destinationModel: NSManagedObjectModel
    let mappingModel: NSMappingModel

    // MARK: Init

    init(sourceVersion: CoreDataMigrationVersion, destinationVersion: CoreDataMigrationVersion) {
        let sourceModel = NSManagedObjectModel.managedObjectModel(forResource: sourceVersion.rawValue)
        let destinationModel = NSManagedObjectModel.managedObjectModel(forResource: destinationVersion.rawValue)

        guard let mappingModel = CoreDataMigrationStep.mappingModel(fromSourceModel: sourceModel, toDestinationModel: destinationModel) else {
            fatalError("Expected modal mapping not present")
        }

        self.sourceModel = sourceModel
        self.destinationModel = destinationModel
        self.mappingModel = mappingModel
    }

    // MARK: - Mapping

    private static func mappingModel(fromSourceModel sourceModel: NSManagedObjectModel, toDestinationModel destinationModel: NSManagedObjectModel) -> NSMappingModel? {
        guard let customMapping = customMappingModel(fromSourceModel: sourceModel, toDestinationModel: destinationModel) else {
            return inferredMappingModel(fromSourceModel:sourceModel, toDestinationModel: destinationModel)
        }

        return customMapping
    }

    private static func inferredMappingModel(fromSourceModel sourceModel: NSManagedObjectModel, toDestinationModel destinationModel: NSManagedObjectModel) -> NSMappingModel? {
        return try? NSMappingModel.inferredMappingModel(forSourceModel: sourceModel, destinationModel: destinationModel)
    }

    private static func customMappingModel(fromSourceModel sourceModel: NSManagedObjectModel, toDestinationModel destinationModel: NSManagedObjectModel) -> NSMappingModel? {
        return NSMappingModel(from: [Bundle.main], forSourceModel: sourceModel, destinationModel: destinationModel)
    }
}

It's possible to have multiple mapping models between versions, (this can be especially useful when migrating large data sets) in this post in an attempt to keep things simple I assume only one mapping model.

CoreDataMigrationStep takes the source model and destination model and attempts to find a way to map between them. As we know there are two types of migrations: Lightweight and Standard - both of which use a NSMappingModel instance to hold the mapping path between the versions. Because of this shared output type mappingModel(fromSourceModel:toDestinationModel) handles searching for a mapping model using either Lightweight and Standard migration. First, a search is made for a custom migration mapping existing in the bundle (Standard migration) and then if no custom mapping model is found Core Data is asked to try and infer a mapping model (Lightweight migration). If a mapping model can't be found using either approach, a fatal error is thrown as this migration path isn't supported.

How can we combine the migration steps into a migration path?

CoreDataMigrator is at the heart of our migration solution and has 3 tasks:

1. Determining if there needs to be a migration.
2. Ensuring the persistent store is ready to be migrated.
3. Performing the migration.

As CoreDataManager (we will see this later) holds a reference to CoreDataMigrator we can make our lives easier by wrapping CoreDataMigrator in a protocol so that it's easier to mock when writing tests for CoreDataManager:

protocol CoreDataMigratorProtocol {
    func requiresMigration(at storeURL: URL, toVersion version: CoreDataMigrationVersion) -> Bool
    func migrateStore(at storeURL: URL, toVersion version: CoreDataMigrationVersion)
}

Now that we have that protocol lets look at how CoreDataMigrator implements the first of those protocol methods:

class CoreDataMigrator: CoreDataMigratorProtocol {

    // MARK: - Check

    func requiresMigration(at storeURL: URL, toVersion version: CoreDataMigrationVersion) -> Bool {
        guard let metadata = NSPersistentStoreCoordinator.metadata(at: storeURL) else {
            return false
        }

        return (CoreDataMigrationVersion.compatibleVersionForStoreMetadata(metadata) != version)
    }

    //Omitted other methods
}

In the above method, the persistent store's metadata is loaded and checked to see if it's compatible with the current bundle model's metadata. To support this we need to extend CoreDataMigrationVersion to include:

private extension CoreDataMigrationVersion {

    // MARK: - Compatible

    static func compatibleVersionForStoreMetadata(_ metadata: [String : Any]) -> CoreDataMigrationVersion? {
        let compatibleVersion = CoreDataMigrationVersion.allCases.first {
            let model = NSManagedObjectModel.managedObjectModel(forResource: $0.rawValue)

            return model.isConfiguration(withName: nil, compatibleWithStoreMetadata: metadata)
        }

        return compatibleVersion
    }
}

The above method attempts to find a compatible model for the metadata by iterating through the model associated with a case of CoreDataMigrationVersion. If a compatible model is found the associated version is returned, else nil is returned.

Now that we know if a migration is required or not, lets look at how that migration happens by implementing the next protocol method:

class CoreDataMigrator: CoreDataMigratorProtocol {

    //Omitted other methods

    // MARK: - Migration

    func migrateStore(at storeURL: URL, toVersion version: CoreDataMigrationVersion) {
        forceWALCheckpointingForStore(at: storeURL)

        var currentURL = storeURL
        let migrationSteps = self.migrationStepsForStore(at: storeURL, toVersion: version)

        for migrationStep in migrationSteps {
            let manager = NSMigrationManager(sourceModel: migrationStep.sourceModel, destinationModel: migrationStep.destinationModel)
            let destinationURL = URL(fileURLWithPath: NSTemporaryDirectory(), isDirectory: true).appendingPathComponent(UUID().uuidString)

            do {
                try manager.migrateStore(from: currentURL, sourceType: NSSQLiteStoreType, options: nil, with: migrationStep.mappingModel, toDestinationURL: destinationURL, destinationType: NSSQLiteStoreType, destinationOptions: nil)
            } catch let error {
                fatalError("failed attempting to migrate from \(migrationStep.sourceModel) to \(migrationStep.destinationModel), error: \(error)")
            }

            if currentURL != storeURL {
                //Destroy intermediate step's store
                NSPersistentStoreCoordinator.destroyStore(at: currentURL)
            }

            currentURL = destinationURL
        }

        NSPersistentStoreCoordinator.replaceStore(at: storeURL, withStoreAt: currentURL)

        if (currentURL != storeURL) {
            NSPersistentStoreCoordinator.destroyStore(at: currentURL)
        }
    }

    private func migrationStepsForStore(at storeURL: URL, toVersion destinationVersion: CoreDataMigrationVersion) -> [CoreDataMigrationStep] {
        guard let metadata = NSPersistentStoreCoordinator.metadata(at: storeURL), let sourceVersion = CoreDataMigrationVersion.compatibleVersionForStoreMetadata(metadata) else {
            fatalError("unknown store version at URL \(storeURL)")
        }

        return migrationSteps(fromSourceVersion: sourceVersion, toDestinationVersion: destinationVersion)
    }

    private func migrationSteps(fromSourceVersion sourceVersion: CoreDataMigrationVersion, toDestinationVersion destinationVersion: CoreDataMigrationVersion) -> [CoreDataMigrationStep] {
        var sourceVersion = sourceVersion
        var migrationSteps = [CoreDataMigrationStep]()

        while sourceVersion != destinationVersion, let nextVersion = sourceVersion.nextVersion() {
            let migrationStep = CoreDataMigrationStep(sourceVersion: sourceVersion, destinationVersion: nextVersion)
            migrationSteps.append(migrationStep)

            sourceVersion = nextVersion
        }

        return migrationSteps
    }

    // MARK: - WAL

    func forceWALCheckpointingForStore(at storeURL: URL) {
        guard let metadata = NSPersistentStoreCoordinator.metadata(at: storeURL), let currentModel = NSManagedObjectModel.compatibleModelForStoreMetadata(metadata) else {
            return
        }

        do {
            let persistentStoreCoordinator = NSPersistentStoreCoordinator(managedObjectModel: currentModel)

            let options = [NSSQLitePragmasOption: ["journal_mode": "DELETE"]]
            let store = persistentStoreCoordinator.addPersistentStore(at: storeURL, options: options)
            try persistentStoreCoordinator.remove(store)
        } catch let error {
            fatalError("failed to force WAL checkpointing, error: \(error)")
        }
    }
}

There is quite a bit of code there, let's break it down into smaller pieces and explore each separately, building the migration process up from bottom-to-top.

Before attempting a migration, we need to undertake some housekeeping on our persistent store.

Since iOS 7, Core Data has used the Write-Ahead Logging (WAL) option on SQLite stores to provide the ability to recover from crashes by allowing changes to be rolled back until the database is stable. If you have ever had to perform a rollback before, the WAL approach may work a little differently from what you are expecting. Rather than directly writing changes to the sqlite file and having a pre-write copy of the changes to rollback to, in WAL mode the changes are first written to the sqlite-wal file and at some future date those changes are transferred to the sqlite file. The sqlite-wal file is in effect an up-to-date copy of some of the data stored in the main sqlite file.

The sqlite-wal and sqlite files store their data using the same structure to allow data to be transferred easily between them. However, this shared structure causes issues during migration as Core Data only migrates the data stored in the sqlite file to the new structure, leaving the data in the sqlite-wal file in the old structure. The resulting mismatch in structure will lead to a crash when Core Data attempts to update/use data stored in the sqlite-wal file 😞 . To avoid this crash, we need to force any data in the sqlite-wal file into the sqlite file before we perform a migration - a process known as checkpointing:

func forceWALCheckpointingForStore(at storeURL: URL) {
    guard let metadata = NSPersistentStoreCoordinator.metadata(at: storeURL), let currentModel = NSManagedObjectModel.compatibleModelForStoreMetadata(metadata) else {
        return
    }

    do {
        let persistentStoreCoordinator = NSPersistentStoreCoordinator(managedObjectModel: currentModel)

        let options = [NSSQLitePragmasOption: ["journal_mode": "DELETE"]]
        let store = persistentStoreCoordinator.addPersistentStore(at: storeURL, options: options)
        try persistentStoreCoordinator.remove(store)
    } catch let error {
        fatalError("failed to force WAL checkpointing, error: \(error)")
    }
}

The above method, forces checkpointing to occur. A side effect of checkpointing is that the empty sqlite-wal file is deleted for us so removing the store from the persistentStoreCoordinator is all the cleanup that we need to perform.

An easy mistake to make when checkpointing is using the bundle's model rather than the store's model - remember we want to perform checkpointing on the live (store) model before attempting to migrate to the latest (bundle) model.

Before a migration can be performed Core Data must first construct the individual migration steps into a migration path:

private func migrationStepsForStore(at storeURL: URL, toVersion destinationVersion: CoreDataMigrationVersion) -> [CoreDataMigrationStep] {
    guard let metadata = NSPersistentStoreCoordinator.metadata(at: storeURL), let sourceVersion = CoreDataMigrationVersion.compatibleVersionForStoreMetadata(metadata) else {
        fatalError("unknown store version at URL \(storeURL)")
    }

    return migrationSteps(fromSourceVersion: sourceVersion, toDestinationVersion: destinationVersion)
}

private func migrationSteps(fromSourceVersion sourceVersion: CoreDataMigrationVersion, toDestinationVersion destinationVersion: CoreDataMigrationVersion) -> [CoreDataMigrationStep] {
    var sourceVersion = sourceVersion
    var migrationSteps = [CoreDataMigrationStep]()

    while sourceVersion != destinationVersion, let nextVersion = sourceVersion.nextVersion() {
        let migrationStep = CoreDataMigrationStep(sourceVersion: sourceVersion, destinationVersion: nextVersion)
        migrationSteps.append(migrationStep)

        sourceVersion = nextVersion
    }

    return migrationSteps
}

In the above methods, the migration path is built by looping through the appropriate model versions until the destination model version is reached. This migration path will take the users data from the persistent store's model version to the bundle model version in a progressive migration:

func migrateStore(at storeURL: URL, toVersion version: CoreDataMigrationVersion) {
    forceWALCheckpointingForStore(at: storeURL)

    var currentURL = storeURL
    let migrationSteps = self.migrationStepsForStore(at: storeURL, toVersion: version)

    for migrationStep in migrationSteps {
        let manager = NSMigrationManager(sourceModel: migrationStep.sourceModel, destinationModel: migrationStep.destinationModel)
        let destinationURL = URL(fileURLWithPath: NSTemporaryDirectory(), isDirectory: true).appendingPathComponent(UUID().uuidString)

        do {
            try manager.migrateStore(from: currentURL, sourceType: NSSQLiteStoreType, options: nil, with: migrationStep.mappingModel, toDestinationURL: destinationURL, destinationType: NSSQLiteStoreType, destinationOptions: nil)
        } catch let error {
            fatalError("failed attempting to migrate from \(migrationStep.sourceModel) to \(migrationStep.destinationModel), error: \(error)")
        }

        if currentURL != storeURL {
            //Destroy intermediate step's store
            NSPersistentStoreCoordinator.destroyStore(at: currentURL)
        }

        currentURL = destinationURL
    }

    NSPersistentStoreCoordinator.replaceStore(at: storeURL, withStoreAt: currentURL)

    if (currentURL != storeURL) {
        NSPersistentStoreCoordinator.destroyStore(at: currentURL)
    }
}

In the above method, we iterate through each migration step and attempt to perform a migration using NSMigrationManager. The result of each completed migration step is saved to a temporary persistent store, only once the migration is complete is the original persistent store overwritten. If there is a failure during any individual migration step a fatal error is thrown - this is especially useful during the development of a custom migration path.

In the above code snippets, we've seen a number of methods used that are not part of the standard API so I've included the extensions that contain these methods below. As with most extensions, the methods are used to reduce boilerplate code:

extension NSPersistentStoreCoordinator {

    // MARK: - Destroy

    static func destroyStore(at storeURL: URL) {
        do {
            let persistentStoreCoordinator = NSPersistentStoreCoordinator(managedObjectModel: NSManagedObjectModel())
            try persistentStoreCoordinator.destroyPersistentStore(at: storeURL, ofType: NSSQLiteStoreType, options: nil)
        } catch let error {
            fatalError("failed to destroy persistent store at \(storeURL), error: \(error)")
        }
    }

    // MARK: - Replace

    static func replaceStore(at targetURL: URL, withStoreAt sourceURL: URL) {
        do {
            let persistentStoreCoordinator = NSPersistentStoreCoordinator(managedObjectModel: NSManagedObjectModel())
            try persistentStoreCoordinator.replacePersistentStore(at: targetURL, destinationOptions: nil, withPersistentStoreFrom: sourceURL, sourceOptions: nil, ofType: NSSQLiteStoreType)
        } catch let error {
            fatalError("failed to replace persistent store at \(targetURL) with \(sourceURL), error: \(error)")
        }
    }

    // MARK: - Meta

    static func metadata(at storeURL: URL) -> [String : Any]?  {
        return try? NSPersistentStoreCoordinator.metadataForPersistentStore(ofType: NSSQLiteStoreType, at: storeURL, options: nil)
    }

    // MARK: - Add

    func addPersistentStore(at storeURL: URL, options: [AnyHashable : Any]) -> NSPersistentStore {
        do {
            return try addPersistentStore(ofType: NSSQLiteStoreType, configurationName: nil, at: storeURL, options: options)
        } catch let error {
            fatalError("failed to add persistent store to coordinator, error: \(error)")
        }
    }
}

extension NSManagedObjectModel {

    // MARK: - Resource

    static func managedObjectModel(forResource resource: String) -> NSManagedObjectModel {
        let mainBundle = Bundle.main
        let subdirectory = "CoreDataMigration_Example.momd"
        let omoURL = mainBundle.url(forResource: resource, withExtension: "omo", subdirectory: subdirectory) // optimised model file
        let momURL = mainBundle.url(forResource: resource, withExtension: "mom", subdirectory: subdirectory)

        guard let url = omoURL ?? momURL else {
            fatalError("unable to find model in bundle")
        }

        guard let model = NSManagedObjectModel(contentsOf: url) else {
            fatalError("unable to load model in bundle")
        }

        return model
    }
}

I won't go into detail about what these extension methods do as I believe their names do a good enough job.

How do we trigger a migration?

CoreDataManager handles both setting up the Core Data stack and triggering a migration (if needed):

class CoreDataManager {

    let migrator: CoreDataMigratorProtocol
    private let storeType: String

    lazy var persistentContainer: NSPersistentContainer = {
        let persistentContainer = NSPersistentContainer(name: "CoreDataMigration_Example")
        let description = persistentContainer.persistentStoreDescriptions.first
        description?.shouldInferMappingModelAutomatically = false //inferred mapping will be handled else where
        description?.shouldMigrateStoreAutomatically = false
        description?.type = storeType

        return persistentContainer
    }()

    lazy var backgroundContext: NSManagedObjectContext = {
        let context = self.persistentContainer.newBackgroundContext()
        context.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy

        return context
    }()

    lazy var mainContext: NSManagedObjectContext = {
        let context = self.persistentContainer.viewContext
        context.automaticallyMergesChangesFromParent = true

        return context
    }()

    // MARK: - Singleton

    static let shared = CoreDataManager()

    // MARK: - Init

    init(storeType: String = NSSQLiteStoreType, migrator: CoreDataMigratorProtocol = CoreDataMigrator()) {
        self.storeType = storeType
        self.migrator = migrator
    }

    // MARK: - SetUp

    func setup(completion: @escaping () -> Void) {
        loadPersistentStore {
            completion()
        }
    }

    // MARK: - Loading

    private func loadPersistentStore(completion: @escaping () -> Void) {
        migrateStoreIfNeeded {
            self.persistentContainer.loadPersistentStores { description, error in
                guard error == nil else {
                    fatalError("was unable to load store \(error!)")
                }

                completion()
            }
        }
    }

    private func migrateStoreIfNeeded(completion: @escaping () -> Void) {
        guard let storeURL = persistentContainer.persistentStoreDescriptions.first?.url else {
            fatalError("persistentContainer was not set up properly")
        }

        if migrator.requiresMigration(at: storeURL, toVersion: CoreDataMigrationVersion.current) {
            DispatchQueue.global(qos: .userInitiated).async {
                self.migrator.migrateStore(at: storeURL, toVersion: CoreDataMigrationVersion.current)

                DispatchQueue.main.async {
                    completion()
                }
            }
        } else {
            completion()
        }
    }
}

If you have ever seen a Core Data stack setup before, you will instantly notice how little code the CoreDataManager contains. Over the years Core Data has evolved and become more developer friendly, above we are taking advantage of a relatively new piece of the Core Data family - NSPersistentContainer which was introduced in iOS 10:

lazy var persistentContainer: NSPersistentContainer = {
    let persistentContainer = NSPersistentContainer(name: "CoreDataMigration_Example")
    let description = persistentContainer.persistentStoreDescriptions.first
    description?.shouldInferMappingModelAutomatically = false //inferred mapping will be handled else where
    description?.shouldMigrateStoreAutomatically = false
    description?.type = storeType

    return persistentContainer
}()

NSPersistentContainer simplifies the creation of the managed object model, persistent store coordinator and the managed object contexts by making smart assumptions on how we want our persistent store configured. It's still possible to access the NSManagedModel, NSPersistentStoreCoordinator and NSManagedObjectContext instances via this container but we no longer have to handle their set-up code.

Our example project is called CoreDataMigration-Example however as you can see when creating the NSPersistentContainer we give CoreDataMigration_Example as our model's name - see Apple's documentation on why the - became a _.

As we only have one Core Data stack, CoreDataManager is a singleton:

static let shared = CoreDataManager()

init(storeType: String = NSSQLiteStoreType, migrator: CoreDataMigratorProtocol = CoreDataMigrator()) {
    self.storeType = storeType
    self.migrator = migrator
}

CoreDataManager is a little odd when it comes to being a singleton in that it has an explicit init implementation. This explicit init method allows for changing the type of persistent store used - by default it's NSSQLiteStoreType however when unit testing we will actually create multiple instances of CoreDataManager using NSInMemoryStoreType to avoid persisting data between tests (and having tests potentially pollute each other). A persistent store type of NSInMemoryStoreType will cause our Core Data stack to only be created in-memory and so be more cheaply torn down and set up than if we used NSSQLiteStoreType. In the accompanying example project, you can see how this is used in the CoreDataManagerTests class.

Loading the persistent store involves interacting with the disk which compared to memory interactions is more expensive ⏲️, as such the loadPersistentStores(completionHandler:) method on NSPersistentContainer is asynchronous. This is mirrored by the setup(), loadPersistentStore(completion:) and migrateStoreIfNeeded(completion:) methods:

func setup(completion: @escaping () -> Void) {
   loadPersistentStore {
       completion()
   }
}

private func loadPersistentStore(completion: @escaping () -> Void) {
   migrateStoreIfNeeded {
       self.persistentContainer.loadPersistentStores { description, error in
           guard error == nil else {
               fatalError("was unable to load store \(error!)")
           }

           completion()
       }
   }
}

private func migrateStoreIfNeeded(completion: @escaping () -> Void) {
    guard let storeURL = persistentContainer.persistentStoreDescriptions.first?.url else {
        fatalError("persistentContainer was not set up properly")
    }

    if migrator.requiresMigration(at: storeURL, toVersion: CoreDataMigrationVersion.current) {
        DispatchQueue.global(qos: .userInitiated).async {
            self.migrator.migrateStore(at: storeURL, toVersion: CoreDataMigrationVersion.current)

            DispatchQueue.main.async {
                completion()
            }
        }
    } else {
        completion()
    }
}

Before an attempt is made to load the persistent store, we check if the model needs to be migrated by calling migrateStoreIfNeeded(completion:).

If the answer is yes - the migrator attempts to migrate the user's data. As migrating can be a relatively slow process, the migration happens on a background queue to avoid hanging the UI. Once the migration is completed the completion closure is called on the main queue.

If the answer is no - the completion closure is called straight away.

Once the persistent store is successfully loaded, the setup() method calls its completion closure and the stack finishes setting up.

This setup method is called in the AppDelegate:

func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
    CoreDataManager.shared.setup {
        self.presentMainUI()
    }

    return true
}

The above code snippet is from the example project where the user is shown a loading screen while the Core Data stack is being set up. Only once the setup is complete is the user allowed into the app proper. presentMainUI switches out the window's root view controller for a navigation stack that can freely use Core Data. While this is strictly not necessary, by splitting the UI into pre and post Core Data stack set up it is possible to avoid race conditions where the app is attempting to use Core Data before it has finished setting up.

💃🥂🎉🕺

Congratulations, that's all there is to the progressive migration approach.

The rest of this post is devoted to putting the above migration approach into practice by migrating an app through 3 Core Data model versions.

Colourful Posts

Colourful Posts is a simple app that allows the user to create posts that are persisted in Core Data. Each post consists of:

• A unique ID.
• A random associated colour represented as a hex string.
• The body/content of the post.
• The date the post was created on.

So that the model looks like:

Each post that the user creates is then displayed in a tableview as a brightly coloured cell.

To keep this post to a responsible length I won't show any code from Colourful Posts that isn't connected to performing a migration.

It's a simple, fun app that we submit to Apple for approval 🤞.

Migrating to version 2

Despite not being able to edit posts, Apple not only approves Colourful Posts, they love it. So much so that they feature it on the Today tab. Colourful Posts is instantly propelled to the top of the charts. After hundreds of thousands of downloads, we decide to hire a new developer to help steer the success-train we find ourselves on 🚂. However, in their first week the new developer mistakes the information stored in the color property on Post to be using RGB rather hex to store the color as a string. Unfortunately we don't catch this mismatch until it's in production and leads to the app crashing on launch 😞. To avoid this issue happening when we hire more developers we decide to rename color to hexColor. As this is a change to the model we need to create a new model version and handle the migration between the old and new version.

To create a new model version, select the *.xcdatamodel (it may be called *.xcdatamodeld) file in the Project Navigator, open the Editor menu from the top bar and click on the Add Model Version... option. In the wizard that opens, this new model will already be given a name, this typically follows [ModelName] [Number] so CoreDataMigration_Example 2 but this can be changed to whatever you want.

Lightweight migrations are typically a less intensive form of migration than Standard migrations (both from a developer and performance POV) because of this I prefer to perform Lightweight migrations whenever possible. Lightweight migrations can handle the following transformations to the model:

• Adding an attribute.
• Removing an attribute.
• Changing a non-optional attribute to be optional.
• Changing an optional attribute to non-optional (by defining a default value).
• Renaming an entity, attribute or relationship (by providing a Renaming ID).
• Adding a relationship.
• Removing a relationship.
• Changing the entity hierarchy.

An impressive list of transformations that we get free (or almost free) with Lightweight migrations. The color to hexColor change is covered by the Renaming an entity, attribute or relationship which has a small caveat: by providing a Renaming ID. The Renaming ID creates a link between the old attribute and the new attribute. All it requires is to add the old attribute name to the new attribute's metadata:

With this information, Core Data now knows that color and hexColor are the same attribute just with different names and that rather than discarding color during a Lightweight migration the value should be transferred to hexColor.

With that change the only thing that's left to do is update CoreDataMigrationVersion to allow migrations from CoreDataMigration_Example to CoreDataMigration_Example 2:

enum CoreDataMigrationVersion: String, CaseIterable {
    case version1 = "CoreDataMigration_Example"
    case version2 = "CoreDataMigration_Example 2"

    //Omitting methods

    func nextVersion() -> CoreDataMigrationVersion? {
    switch self {
    case .version1:
        return .version2
    case .version2:
        return .nil
    }
}

A new case was added to CoreDataMigrationVersion - version2. As with version1, this new version has a raw value which maps to the name of its respective model version - CoreDataMigration_Example 2. nextVersion() has also been updated so that there is a migration path from version1 to version2.

Now that we have a migration path, let's look at unit testing it. Unit testing a migration path requires:

1. Populating a SQLite database using the CoreDataMigration_Example model.
2. Copying that SQLite database into the test target.
3. Asserting that the contents of that SQLite database migrated as expected.

Before copying your SQLite database, it's important to ensure it is in fact populated with test data. As we discussed above, Core Data uses Write-Ahead Logging to improve performance so your data could be residing in the sqlite-wal file rather than the sqlite file. The easiest way to force any uncommitted changes is to fake a migration - add a breakpoint just after the forceWALCheckpointingForStore(at:) method, open the Application Support folder, copy the sqlite file and then abort the migration.

class CoreDataMigratorTests: XCTestCase {

    var sut: CoreDataMigrator!

    // MARK: - Lifecycle

    override class func setUp() {
        super.setUp()

        FileManager.clearTempDirectoryContents()
    }

    override func setUp() {
        super.setUp()

        sut = CoreDataMigrator()
    }

    override func tearDown() {
        sut = nil

        super.tearDown()
    }

    func tearDownCoreDataStack(context: NSManagedObjectContext) {
        context.destroyStore()
    }

    // MARK: - Tests

    // MARK: SingleStepMigrations

    func test_individualStepMigration_1to2() {
        let sourceURL = FileManager.moveFileFromBundleToTempDirectory(filename: "CoreDataMigration_Example_1.sqlite")
        let toVersion = CoreDataMigrationVersion.version2

        sut.migrateStore(at: sourceURL, toVersion: toVersion)

        XCTAssertTrue(FileManager.default.fileExists(atPath: sourceURL.path))

        let model = NSManagedObjectModel.managedObjectModel(forResource: toVersion.rawValue)
        let context = NSManagedObjectContext(model: model, storeURL: sourceURL)
        let request = NSFetchRequest.init(entityName: "Post")
        let sort = NSSortDescriptor(key: "postID", ascending: false)
        request.sortDescriptors = [sort]

        let migratedPosts = try? context.fetch(request)

        XCTAssertEqual(migratedPosts?.count, 10)

        let firstMigratedPost = migratedPosts?.first

        let migratedDate = firstMigratedPost?.value(forKey: "date") as? Date
        let migratedHexColor = firstMigratedPost?.value(forKey: "hexColor") as? String
        let migratedPostID = firstMigratedPost?.value(forKey: "postID") as? String
        let migratedContent = firstMigratedPost?.value(forKey: "content") as? String

        XCTAssertEqual(migratedDate?.timeIntervalSince1970, 1547494150.058821)
        XCTAssertEqual(migratedHexColor, "1BB732")
        XCTAssertEqual(migratedPostID, "FFFECB21-6645-4FDD-B8B0-B960D0E61F5A")
        XCTAssertEqual(migratedContent, "Test body")

        tearDownCoreDataStack(context: context)
    }
}

There is no need to test every object stored in the persistent store rather we just have to assert that each entity has the correct number of objects and then select one object per entity and assert the values on that object.

In the above test, a migration is triggered between the CoreDataMigration_Example and CoreDataMigration_Example 2 models. An interesting point to note is that rather than making use of the Post subclass of NSManagedObject, the above test uses a plain NSManagedObject instance and KVC to determine if the migration was a success. This is to handle the very likely scenario that the Post structure defined in the CoreDataMigration_Example 2 model will not be the final Post structure. If we used Post instances then as the Post entity changed in later versions of the model, those changes would be mirrored in Post NSManagedObject subclass which would result in this test potentially breaking. By using plain NSManagedObject instances and KVC it is possible to ensure that this test is 100% accurate to the structure of the Post entity as defined in CoreDataMigration_Example 2 model.

As changes are being made to the file system the last thing the test does is tear down the Core Data stack using the tearDownCoreDataStack(context:) method.

Just deleting the migrated SQLite files from the file system would result in a rather serious sounding error BUG IN CLIENT OF libsqlite3.dylib: database integrity compromised by API violation: vnode unlinked while in use:.... being printed to the console. This is because the store would be being deleted from under an active Core Data stack. While the active Core Data stack in question will then be discarded resulting in this error not actually creating any issues, having it clutter the console would make it that much harder to read it and spot any genuine issues printed there so best to tear things down properly.

In the above test class there are a few extensions being used to make things easier:

extension FileManager {

    // MARK: - Temp

    static func clearTempDirectoryContents() {
        let tmpDirectoryContents = try! FileManager.default.contentsOfDirectory(atPath: NSTemporaryDirectory())
        tmpDirectoryContents.forEach {
            let fileURL = URL(fileURLWithPath: NSTemporaryDirectory(), isDirectory: true).appendingPathComponent($0)
            try? FileManager.default.removeItem(atPath: fileURL.path)
        }
    }

    static func moveFileFromBundleToTempDirectory(filename: String) -> URL {
        let destinationURL = URL(fileURLWithPath: NSTemporaryDirectory(), isDirectory: true).appendingPathComponent(filename)
        try? FileManager.default.removeItem(at: destinationURL)
        let bundleURL = Bundle(for: CoreDataMigratorTests.self).resourceURL!.appendingPathComponent(filename)
        try? FileManager.default.copyItem(at: bundleURL, to: destinationURL)

        return destinationURL
    }
}

extension NSManagedObjectContext {

    // MARK: Model

    convenience init(model: NSManagedObjectModel, storeURL: URL) {
        let persistentStoreCoordinator = NSPersistentStoreCoordinator(managedObjectModel: model)
        try! persistentStoreCoordinator.addPersistentStore(ofType: NSSQLiteStoreType, configurationName: nil, at: storeURL, options: nil)

        self.init(concurrencyType: .mainQueueConcurrencyType)

        self.persistentStoreCoordinator = persistentStoreCoordinator
    }

    // MARK: - Destroy

    func destroyStore() {
        persistentStoreCoordinator?.persistentStores.forEach {
            try? persistentStoreCoordinator?.remove($0)
            try? persistentStoreCoordinator?.destroyPersistentStore(at: $0.url!, ofType: $0.type, options: nil)
        }
    }
}

As stated above, I won't expand on the extension methods.

Migrating to version 3

After another successful release, we decide to expand our posting functionality by allowing the user to add multiple sections to a post. These sections will be stored alongside the post in Core Data. As with any model change we need to create a new model version: CoreDataMigration_Example 3.

Each section consists of:

• A title.
• A body.
• An index.

which in turn reduces a post to:

• A unique ID.
• A random associated colour represented as a hex string.
• The date the post was created on.
• A collection of sections.

Such that:

Migrating from CoreDataMigration_Example 2 to CoreDataMigration_Example 3 is slightly trickier than the previous migration as CoreDataMigration_Example 2 splits an existing entity into two entities and creates a relationship between them. This will require implementing both a mapping model and migration policy.

To create a mapping model open the File menu on the top bar then click on New File->New, in the window that opens scroll down to the Core Data section and double tap on Mapping Model. This will open a wizard where you can select your source and destination model versions so in this case: CoreDataMigration_Example 2 and CoreDataMigration_Example 3. After that you need to give the mapping a name and save it, I tend to follow Migration[sourceVersion]to[destinationVersion]ModelMapping as a naming convention so Migration2to3ModelMapping.

A mapping model defines the transformations required to migrate from the source model to the destination model. In Xcode, a mapping model is an xcmappingmodel file that when opened has a GUI that's very similar to the Core Data Model GUI. A mapping model handles mapping between entities, attributes and relationships. The mapping model GUI even allows for simple transformations. If the model had a percentage attribute that used to have a value between 0 - 100 but in the new model that value should be between 0 - 1, we could use the Expression field on that attribute to perform this transformation by setting the expression to: $source.percentage/100. Despite the range of transformations possible within the mapping model GUI some changes are just too complex and require a more custom approach - this is handled by creating a migration policy. A migration policy is an NSEntityMigrationPolicy subclass that defines how to map between two entities from two different model versions using the full Core-Data/Swift toolkit.

Migrating from CoreDataMigration_Example 2 to CoreDataMigration_Example 3 will require a custom migration policy as we will need to move the current content attribute's value on Post to both the title and body attributes on a newly created Section instance:

final class Post2ToPost3MigrationPolicy: NSEntityMigrationPolicy {

    override func createDestinationInstances(forSource sourcePost: NSManagedObject, in mapping: NSEntityMapping, manager: NSMigrationManager) throws {
        try super.createDestinationInstances(forSource: sourcePost, in: mapping, manager: manager)

        guard let destinationPost = manager.destinationInstances(forEntityMappingName: mapping.name, sourceInstances: [sourcePost]).first else {
            fatalError("was expected a post")
        }

        let sourceBody = sourcePost.value(forKey: "content") as? String
        let sourceTitle = sourceBody?.prefix(4).appending("...")

        let section = NSEntityDescription.insertNewObject(forEntityName: "Section", into: destinationPost.managedObjectContext!)
        section.setValue(sourceTitle, forKey: "title")
        section.setValue(sourceBody, forKey: "body")
        section.setValue(destinationPost, forKey: "post")
        section.setValue(0, forKey: "index")

        var sections = Set()
        sections.insert(section)

        destinationPost.setValue(sections, forKey: "sections")
    }
}

Just like with mapping models I have a naming convention for migration policies: [Entity][Version]To[Entity][Version]MigrationPolicy, this way I can know at a glance exactly what the migration policy is doing.

The above migration policy overrides createDestinationInstances(forSource:in:manager) to allow for transforming existing CoreDataMigration_Example 2 model Post instances into CoreDataMigration_Example 3 model Post and Section instances. Again in order to interact with attributes on each Post instance, we need to use KVC. First, a new CoreDataMigration_Example 3 model Post (destinationPost) is created using the mapping rules defined in the mapping model (these rules are set in the mapping model GUI). Then a Section instance from the new Section entity. As the old Post didn't have the concept of a title, we take the first 4 characters of that older post's body value and combine it with ... so that it can be used as the title of the new Section instance. After setting the other properties of the section, a relationship between this section and the new post is created.

In order for this migration policy to be used during the migration we need to add it to the mapping model by setting the Custom Policy on the PostToPost entity mapping:

It's important to note that the migration policy class name is prefixed with the module name.

All that's left to do is to update: CoreDataMigrationVersion by introducing a version3 case and updating nextVersion:

enum CoreDataMigrationVersion: String, CaseIterable {
    case version1 = "CoreDataMigration_Example"
    case version2 = "CoreDataMigration_Example 2"
    case version3 = "CoreDataMigration_Example 3"

    //Omitting methods

    func nextVersion() -> CoreDataMigrationVersion? {
    switch self {
    case .version1:
        return .version2
    case .version2:
        return .version3
    case .version3:
        return nil
    }
}

And that's it - we now have a migration path from not only CoreDataMigration_Example 2 to CoreDataMigration_Example 3 but also from CoreDataMigration_Example to CoreDataMigration_Example 3.

Check out CoreDataMigratorTests for the unit test that supports this migration.

Migrating to version 4

The success of Colourful Posts knows no bounds and we decide to release our next killer feature: deleting posts. This deletion functionality is actually a soft delete which means that the post will still exist in Core Data but won't be shown to the user. We can achieve this by adding a new attribute to the Post entity - softDelete. Of course, this change will require a new model version and for us to handle the migration to that version. This migration can be handled as a Lightweight migration and in fact requires very little effort on our part. We only need to add a new case to CoreDataMigrationVersion and update nextVersion:

enum CoreDataMigrationVersion: String, CaseIterable {
    case version1 = "CoreDataMigration_Example"
    case version2 = "CoreDataMigration_Example 2"
    case version3 = "CoreDataMigration_Example 3"
    case version4 = "CoreDataMigration_Example 4"

    // Omitted methods

    // MARK: - Migration

    func nextVersion() -> CoreDataMigrationVersion? {
        switch self {
        case .version1:
            return .version2
        case .version2:
            return .version3
        case .version3:
            return .version4
        case .version4:
            return nil
        }
    }
}

Check out CoreDataMigratorTests for the unit test that supports this migration.

We got there 🏁

Core Data migration can often seem like a tedious and cumbersome process that punishes developers for mutating their models. However (hopefully) this post shows that by diverging from the default one-step migration approach we can simplify the process and significantly cut down the amount of work required to perform a successful migration. This simplification makes it much easier to treat our user's data with the care that I hope others treat my data with.

I want to acknowledge that I based the above approach on the migration example shown in the excellent Core Data book by Florian Kugler and Daniel Eggert which you can get here. I would highly recommend that you give that book a read as it's a treasure trove of Core Data knowledge.

In our day-to-day life, dates are pretty straight forward - we read them all the time, make plans around them, and share them with other people. Apart from the occasional missed birthday party all of these date-based tasks go surprisingly smoothly. Which is remarkable when you stop to think how complex our date systems are. It works this smoothly because everyone is making some pretty large, unspoken assumptions around the dates that they see - what calendar is used, what timezone is used, the ordering of date elements, etc. While for most of us these assumptions rarely create issues in our day-to-day lives, if we want to build a system that uses dates we need to discover what assumptions we are making and remove them. Take for example the following date:

02/12/06

If you are from the UK then you would read this as:

2nd of December 2006

whereas if you are from the US then you would read this as:

February 12th, 2006

Both of these interpretations are valid however only one is correct. If the developer is from the UK then the former interpretation is correct and any US users are going to be either confused when they start seeing dates like 14/12/06 😕 or angry when they show up to an appointment on the wrong day 🤬. It's easy to fall into this date ordering trap especially if everyone in the development and testing teams is working off the same set of date assumptions/conventions - this is the real danger with assumptions, often we don't know that we are making them.

Getting to know what's local

When it comes to formatting dates we can choose to take the user's conventions into account or not, so for example, with the above date example 02/12/06 if we want to confuse our US users we could hardcode the date element order to match UK conventions by:

let dateFormatter = DateFormatter()
dateFormatter.dateFormat = "dd/MM/yy"
let formattedDate = dateFormatter.string(from: date)

It's not uncommon to see this approach to configuring a DateFormatter and while there are valid scenarios for hardcoding the dateFormat value (we will see an example of that later) I can't think of any valid scenarios for doing so when displaying a date to a user. By hardcoding the dateFormat value we are instructing the DateFormatter to ignore the date element ordering for the user's conventions and instead use the same element ordering for all users. By not meeting our user's date element ordering expectations we degrade that users experience by making something that should be as simple as reading a date into a surprise maths puzzle 📖. I've seen a number of novel but naive approaches on how to solve the expectation mismatch a hardcoded dateFormat produces. These solutions tend to centre on two approaches:

1. Add conditional logic and set a dateFormat value that's unique for each user's conventions.
2. Move the dateFormat value into the localisation .strings file.

While it's possible to meet our user's expectations with either of these solutions both have two major drawbacks:

1. The developer needs to actively decide which conventions each DateFormatter will support and.
2. The developer needs to take on the responsibility of ensuring that any date and time formatting rules are correct for each supported set of conventions.

In order to define the formatting rules for each dateFormat value, the developer would need to answer questions such as:

• Which calendar to use?
• Which clock to use: 24-hour or 12-hour?
• What is the ordering of date elements?
• What is the date separator character(s)?

And the list goes on.

Answering all these questions correctly is a massive task and as it turns out, a totally unnecessary one as iOS already answers them (and many more besides) for us. In iOS, the linguistic, cultural and technological conventions are collected within the Locale class (each locale's conventions are provided by the Unicode Common Locale Data Repository (CLDR) project). The conventions defined within each locale should match our user's expectations for how their world is measured and represented; as such Locale plays a vital role in making our users comfortable when using our apps to complete their tasks. By working with the user's locale conventions, it's possible to produce formatted dates that match our user's expectations. Even better than just improving our user's in-app experience, is that this can all be achieved without actually having to know or care what those locale conventions actually are.

While each locale comes with default conventions, some of them can be customised by the user e.g. switching from 12-hour to 24-hour clock representation. So even if two users have the same locale, it would be a mistake to assume these locales where identical.

For the rest of this post I'm going to assume your locale is the default US locale. To keep the examples comparable, each will use a date based off of the Unix epoch timestamp: 1165071389 which equates to 2nd of December, 2006 at 14:56:29. You can see the completed playground with all of the below examples by following this link.

Staying local

When it comes to presenting date information to the user, it's best to leave all locale concerns to DateFormatter - there are two ways to do this:

1. Using DateFormatter.Style
2. Using a template

Let's explore both these approaches in more detail.

Using DateFormatter.Style

One way to ensure that a Date is always shown to the user using a formatting style that they are expecting is to use the DateFormatter.Style enum. The DateFormatter.Style enum is a set of predefined values that correspond to a date format, at the time of writing (iOS 12) this enum has 5 possible cases to choose from:

1. none
2. short
3. medium
4. long
5. full

Each case will take into account the user's locale settings when formatting dates. DateFormatter has two DateFormatter.Style properties: dateStyle and timeStyle. Each property works semi-independently of the other and as such each can take a different value - this flexibility allows for a wide range of formatting options (25 possible permutations):

let dateFormatter = DateFormatter()
dateFormatter.dateStyle = .full
dateFormatter.timeStyle = .medium
let formattedDate = dateFormatter.string(from: date)

formattedDate would be set to Saturday, December 2, 2006 at 2:56:29 PM.

In the above example, we can see the user's locale influencing the formattedDate value in a number of ways:

1. The date format ordering - EEEE, MMMM d, y 'at' h:m:s a.
2. The names of the months and days.
3. The separator between the various date elements - / for calendar date elements and : for time date elements.

It's also interesting to note that DateFormatter is handling combining the calendar date and time elements together into a sentence structure that is appropriate for both DateFormatter.Style values. So in the above example at is being used as a connector between those two elements however if dateStyle was changed to .short then at becomes , and we get 12/2/06, 2:56:29 PM as the formatted date should be shown in a more compacted form - pretty powerful stuff 🤯.

To demonstrate the power of this further, if we set the locale to Locale(identifier: "de_DE") the above .full example becomes Samstag, 2. Dezember 2006 um 14:56:29 and the .short example becomes 02.12.06, 14:56:29 - all this localisation without me having to know anything about German date conventions or even anything about the German language - 🤯 in 🇩🇪.

Another way to use the DateFormatter.Style approach is by using the available class method:

let formattedDate = DateFormatter.localizedString(from: date, dateStyle: .short, timeStyle: .medium)

formattedDate has the same value as the property based approach: 12/2/06, 2:56:29 PM.

When deciding which option to use I tend to favour the property(s) approach when I either need to cache the DateFormatter instance being used (see "Sneaky date formatters exposing more than you think" for more details on how to cache date formatters) or further customise it (outside of just the dateStyle and/or timeStyle properties). For all other scenarios, I favour the class method approach as I think it's easier to reason about and reads better (from the method name I know that I'm getting a localised string value back).

This is a helpful cheatsheet showing the output for each DateFormatter.Style case. To see the conventions used in different locales set the locale property for the DateFormatter instance to the locale you are interested in e.g. dateFormatter.locale = Locale(identifier: "en_CA").

Using DateFormatter.Style works really well if you to want to display a date in one of the available formats but if you want to display a custom format you need to go down a different path.

Using a template

A template is a customised instruction to DateFormatter of the date elements that the formatted date should contain - these date elements are specified using the same symbols as used with a fixed string date format approach: d, MM, y etc. The beauty of the template approach is that it will take this customised instruction and produce a formatted date that takes into account the user's locale when displaying those elements. For example to produce a formatted date only containing the day and month, the template could look like:

let dateFormatter = DateFormatter()
dateFormatter.setLocalizedDateFormatFromTemplate("d MM")
let formattedDate = dateFormatter.string(from: date)

formattedDate would be set to 12/2. The more eagle-eyed among you will have spotted that two transformations have occurred here, the first is that a locale-specific separator was added between the date elements and that the date elements have switched positions from the ordering in the template i.e. from d MM to MM d 🚀.

It's important to note that when defining the above template I added a space between the different date elements, strictly speaking this space wasn't needed - dMM would have resulted in the same formattedDate value. I included the space here only to make the template easier to read.

Just like with the DateFormatter.Style, the template approach has both an instance and class interface:

let localisedDateFormat = DateFormatter.dateFormat(fromTemplate: "d MM", options: 0, locale: Locale.current)
let dateFormatter = DateFormatter()
dateFormatter.dateFormat = localisedDateFormat
let formattedDate = dateFormatter.string(from: date)

Again, formattedDate would be set to 12/2. The class approach is little more verbose than the instance approach but could be more useful if you wanted to pass the localised formatting string around rather a DateFormatter instance.

Going remote

You may be thinking at this point:

"Is using dateFormat with a fixed string ever the correct approach?"

The simple answer:

Yes.

The date formatting examples shown so far have been concerned with presenting a formatted date to the user. However as iOS developers we (often) have another consumer of our date data: the backend. Typically the backend will be expecting all date values to be sent using a fixed, locale-neutral format. In order to ensure that the user's locale does not affect these dates when converting from a Date instance to the formatted date value we need to take full control of our DateFormatter instances and hardcode the locale, timeZone and dateFormat properties to match the backend's expectations:

let dateFormatter = DateFormatter()
dateFormatter.locale = Locale(identifier: "en_US_POSIX")
dateFormatter.timeZone = TimeZone(identifier: "UTC")
dateFormatter.dateFormat = "yyyy-MM-dd'T'HH:mm:ss'Z'"
let formattedDate = dateFormatter.string(from: date)

formattedDate would be set to 2006-12-02T14:56:29Z regardless of the user's locale settings. An interesting side effect of setting the locale property is that the formatter's calendar property is always set to the default calendar for that locale (which for this local is the Gregorian calendar). It's important to note that en_US_POSIX is not the same locale as en_US - en_US_POSIX while based off of en_US is a special locale that isn't tied to any country/region so shouldn't change even if en_US does change.

You may have noticed that the dateFormatter is using the ISO 8601 string format, in this case (and if your project has a base iOS version of at least iOS 10) I would recommend using ISO8601DateFormatter instead of the more generic DateFormatter class.

Letting go of (some) control

When displaying a date in our apps, it makes sense to do so in a format that the user is expecting and can easily understand- this isn't as simple as it first seems. Thankfully, DateFormatter has two very straight forward ways to achieve this:

1. Using DateFormatter.Style
2. Using a template

Both approaches work well and require very little effort to solve what is a real iceberg of a problem. The only thing it costs us is that we need to give up a little bit of control on exactly how the date is formatted - that's a price I'm happy to pay to ensure that my users are shown dates in the most convenient format for them and I don't need to think about which calendar a certain locale uses or if month comes before day, etc.

Now I just need to convince those pixel-perfect designers on my team that giving up some control is actually a good thing... 😅

If you are interested, there is an accompanying playground that contains all of the above code snippets.

Ask any iOS developer about DateFormatter and one of the first things you will hear is that creating a DateFormatter instance is an expensive operation. The second thing you will hear is that after creating one you need to cache it.

In this post, I want to look at how expensive it is to create DateFormatter instances and how we can cache them effectively.

To determine just how expensive they are we will need to conduct a small experiment 🔬.

If you'd prefer not to have to manually build up the below examples, you can download the completed playground here.

The expense experiment

Our experiment will have two scenarios:

1. Create a new instance of DateFormatter for each date conversion.
2. Reuse the same instance of DateFormatter for all date conversions.

class DateConverter {

    let dateFormat = "y/MM/dd @ HH:mm"

    func convertDatesWithUniqueFormatter(_ dates: [Date]) {
        for date in dates {
            let dateFormatter = DateFormatter()
            dateFormatter.locale = Locale(identifier: "en_US_POSIX")
            dateFormatter.dateFormat = dateFormat

            _ = dateFormatter.string(from: date)
        }
    }

    func convertDatesWithReusedFormatter(_ dates: [Date]) {
        let dateFormatter = DateFormatter()
        dateFormatter.locale = Locale(identifier: "en_US_POSIX")
        dateFormatter.dateFormat = dateFormat

        for date in dates {
            _ = dateFormatter.string(from: date)
        }
    }
}

convertDatesWithUniqueFormatter represents the first scenario and convertDatesWithReusedFormatter the second scenario. Both methods follow a similar structure - loop over an array of dates and format each date into a string representation, the only difference is in how DateFormatter is used.

The XCTest framework makes it straight forward to measure the performance of our code using the appropriately named measure(_:) method. measure(_:) tracks the time in seconds that it takes for the code under test to finish executing. As it's possible for factors outside of our control to affect performance, measure(_:) executes the same test 10 times and reports the average time as its result ⏲️.

class DateConverterTests: XCTestCase {
    var sut: DateConverter!
    let dates = Array(repeating: Date(), count: 100)

    // MARK: - Lifecycle

    override func setUp() {
        super.setUp()
        sut = DateConverter()
    }

    override func tearDown() {
        sut = nil
        super.tearDown()
    }

    // MARK: - Tests

    func test_convertDatesWithUniqueFormatter_performance() {

        measure {
            sut.convertDatesWithUniqueFormatter(dates)
        }
    }

    func test_convertDatesWithReusedFormatter_performance() {

        measure {
            sut.convertDatesWithReusedFormatter(dates)
        }
    }
}

Running the two above tests results in generating the following console output:

Of course running the tests on your computer will give you different results.

The above results show that when converting 100 dates, reusing a DateFormatter instance takes on average 20% (0.02 vs 0.004 seconds) of the time that creating a unique DateFormatter instance for each conversion does. While in real terms the difference in time isn't much, as a percentage the difference is pretty conclusive - reusing a DateFormatter instance is 5x cheaper. If we are thinking in terms of UX, the additional overhead from always creating a new instance of DateFomatter could be the difference between a smooth scrolling table view and a jumpy one.

Premature optimisation is the mother of many bugs so before deciding to improve the performance of any one area, it's important to make sure that that area is actually a performance bottleneck - Instruments and Xcode itself are great tools for profiling. By only optimising actual performance bottlenecks we can ensure that we are not wasting time and not unnecessarily making our codebase more complex than it needs to be.

Working with performant DateFormatters

Now that it's been determined that reusing a DateFormatter instance offers a performance improvement and we've identified that this performance improvement will lead to a better user experience, the question is:

"How do we reuse it?"

It's simple enough to extract the DateFormatter instance into a local variable or private property so it can be reused and I won't explore these here - things really start to get interesting when the same formatter is needed across the project.

While normally derided as an overused pattern, I think sharing formatters would be greatly served by using the Singleton pattern:

class DateFormattingHelper {

    // MARK: - Shared

    static let shared = DateFormattingHelper()

    // MARK: - Formatters

    let dobDateFormatter: DateFormatter = {
        let dateFormatter = DateFormatter()
        dateFormatter.locale = Locale(identifier: "en_US_POSIX")
        dateFormatter.dateFormat = "y/MM/dd @ HH:mm"

        return dateFormatter
    }()
}

If we then wanted to format Date instances into date-of-birth strings, we could add a method similar to:

func formatDOB(_ date: Date, with dateFormatter: DateFormatter) -> String {
    let formattedDate = dateFormatter.string(from: date)
    return ("Date of birth: \(formattedDate)")
}

And when it's called, pass in the shared DateFormatter instance that is a property on DateFormattingHelper singleton like:

let dateFormatter = DateFormattingHelper.shared.dobDateFormatter
let dobFormattedString = formatDOB(date, with: dateFormatter)

It's a simple, easy to read and easy to test approach that ensures that DateFormatter instances are being reused. It's not too hard to imagine adding more DateFormatter properties to DateFormattingHelper and more helper methods like formatDOB to our view controllers to meet all of our date formatting needs.

But before we get too carried away with our success, let's examine our use of the Singleton pattern here in more detail. A key design consideration when using any pattern to expose shared state is: mutability. Ideally shared state should be immutable to prevent a situation where different parts of the codebase can change that state and so indirectly affect other parts that depend on that shared state. However, as we have seen DateFormattingHelper only has one property and that property is a constant (let) so mutability shouldn't be an issue here 👏.

Or is it? 🤔

Swift has two categories of Type:

1. Value - structs, enums, or tuples
2. Reference - classes, functions or closures

The main difference between the types is shown when they are used in an assignment.

When a value-type instance is assigned, an exact copy of that instance is made and it is this copy that is then assigned. This results in two instances of that value-type existing, with both instances (at least initially) having the same property values:

struct PersonValueTypeExample: CustomStringConvertible {
    var name: String
    var age: Int

    var description: String {
        return ("name: \(name), age: \(age)")
    }
}

var a = PersonValueTypeExample(name: "Susie", age: 29)
var b = a
b.name = "Samantha"
a.age = 56

print("a: \(a)") // prints "a: name: Susie, age: 56"
print("b: \(b)") // prints "b: name: Samantha, age: 29"

CustomStringConvertible is used here only to override the description property and ensure a consistent printout format between the value-type and reference-type examples.

As the above example shows, when a is assigned to b, a copy of a is made and assigned to b. Any change made to either a or b will be limited to that instance only - this can be seen in the description string printed for both.

Copying a value-type instance can be an expensive process so a number of techniques are used to try and avoid making the copy. These techniques range from compiler optimisations to implementation details in the struct itself such as Copy-On-Write. These optimisation techniques can result in our value-type instances behaving more like reference-types under the hood. While this is useful to know about, the important thing to note is that these optimisations are transparent to us as users of these value-types. So we can build a system which is dependent upon the difference between value and reference types without having to care how the instances of these types are actually stored in memory and accessed.

When a reference-type instance is assigned, a reference (or pointer) to that instance is created and it is this reference that is then actually assigned (rather than the instance itself). This can result in an instance having multiple references so that the instance becomes an informal shared instance to those references:

class PersonReferenceTypeExample: CustomStringConvertible {
    var name: String
    var age: Int

    var description: String {
    return ("name: \(name), age: \(age)")
    }

    init(name: String, age: Int) {
        self.name = name
        self.age = age
    }
}

let c = PersonReferenceTypeExample(name: "Susie", age: 29)
var d = c
d.name = "Samantha"
c.age = 56

print("c: \(c)") // prints "c: name: Samantha, age: 56"
print("d: \(d)") // prints "d: name: Samantha, age: 56"

As the above example shows when c is assigned to d, a new reference to the same instance that c is pointing at is created and assigned to d. Any change made to either c or d will affect the other - this can be seen in the description string printed for both.

The sharper eyed among you may have spotted that the above examples differ in one additional aspect other than just class/struct and the naming of the variables - in the struct example a is defined as a var whereas in the class example c is a let. When a value-type instance is assigned to a constant, all of its properties are treated as constant properties (and any mutating methods are no longer available) - this isn't true for reference-type instances where var and let are always respected.

You may have heard of pass-by-reference and pass-by-value and currently be thinking that these terms are connected to the above types. This is a common misunderstanding, by default Swift uses pass-by-value when passing both value-type and reference-type instances. The only difference as we have seen is what is copied - with a value-type it's the entire instance, with a reference-type instance it's a reference/pointer. Why this is the case took me a while to fully get my head around, I found the following Stack Overflow question very informative, as well as the wiki page on Evaluation Strategy especially the "Call by sharing" section which I think best describes what happens when we pass a reference-type into a method and why that isn't pass-by-reference. Using the linked wiki definition of pass-by-reference, the nearest that Swift gets to it is using the inout keyword.

While an interesting detour, you may be thinking:

"What does any of it have to do with DateFormattingHelper?"

Well, dobDateFormatter is a reference-type so when passing it around, new references are being made that point to the same shared DateFomatter instance. On first glance that's fine because dobDateFormatter is a constant property and so is immutable however the trouble lays in that some of the properties on DateFormatter are mutable. As we seen above (unlike a value-type instance) a reference-type instance's property declarations are not affected by how that instance was defined. So while it looks like in DateFormattingHelper we have immutable shared state we actually have mutable shared state - just hidden one layer deeper.

A bug that is introduced by changing shared state is often discovered not where the shared state is changed but elsewhere in the codebase where the previous state of the shared state was expected - think about what the consequences of changing the badgeFormat value on dobDateFormatter would be. Even then the other part of the app will only surface the bug if the code that introduces the bug is called first i.e. the bug is dependent on how the user moves through the app 😱. A nightmare of a bug that can take a long time to track down and solve. We can avoid the many sleepless nights this bug would cause by turning hidden mutable properties into immutable properties.

Going from mutable to immutable is actually surprisingly straightforward in this example, we just need to hide our DateFomatter instances behind a protocol. When it comes to using the dobDateFormatter property above from outside the DateFormattingHelper class we are actually only interested in using a DateFomatter to convert a Date into a String so we can produce an extremely limited protocol:

protocol DateFormatterType {
    func string(from date: Date) -> String
}

We just need to conform DateFomatter to our custom DateFormatterType using an extension:

extension DateFormatter: DateFormatterType { }

Now that we have this protocol we can return DateFomatter instances wrapped up as DateFormatterType instance:

let dobDateFormatter: DateFormatterType = {
    let dateFormatter = DateFormatter()
    dateFormatter.locale = Locale(identifier: "en_US_POSIX")
    dateFormatter.dateFormat = "y/MM/dd @ HH:mm"

    return dateFormatter
}()

So whenever we use DateFomatter we would use DateFormatterType instead so:

func formatDOB(_ date: Date, with dateFormatter: DateFormatterType) -> String {
    let formattedDate = dateFormatter.string(from: date)
    return ("Date of birth: \(formattedDate)")
}

In fact we can take this even further by absorbing the formatDOB method into DateFormattingHelper and hide DateFormatterType altogether:

private protocol DateFormatterType {
    func string(from date: Date) -> String
}

class DateFormattingHelper {

    // MARK: - Shared

    static let shared = DateFormattingHelper()

    // MARK: - Formatters

    private let dobDateFormatter: DateFormatterType = {
        let dateFormatter = DateFormatter()
        dateFormatter.locale = Locale(identifier: "en_US_POSIX")
        dateFormatter.dateFormat = "y/MM/dd @ HH:mm"

        return dateFormatter
    }()

    // MARK: - DOB

    func formatDOBDate(_ date: Date) -> String {
        let formattedDate = dobDateFormatter.string(from: date)
        return ("Date of birth: \(formattedDate)")
    }
}

DateFormatterType and dobDateFormatter are now private and so only accessible from inside DateFormattingHelper. At this point, you may be thinking:

"What's the point of still having the DateFormatterType protocol?"

I think it's important to understand that the issue that DateFormatterType is solving isn't one of malicious intent from someone deliberately trying to crash the app but from a trusted colleague who through either clumsiness or a lack of domain knowledge, accidentally misuses a DateFormatter instance. So if we simply made the dobDateFormatter property private, it would still be possible to introduce this type of bug into DateFormattingHelper (by modifying the properties of e.g. dobDateFormatter from inside formatDOBDate) however by continuing to use the protocol approach we can actually reduce this risk further. So by being both private and hidden behind a protocol, we are protecting against unintended mutations of a shared instance both external and internal to DateFormattingHelper.

Going a step beyond

Now that we have the basic mechanisms in place, lets look at a more complete version of DateFormattingHelper:

class DateFormattingHelper {

    // MARK: - Shared

    static let shared = DateFormattingHelper()

    // MARK: - Formatters

    private let dobDateFormatter: DateFormatterType = {
        let dateFormatter = DateFormatter()
        dateFormatter.locale = Locale(identifier: "en_US_POSIX")
        dateFormatter.dateFormat = "y/MM/dd @ HH:mm"

        return dateFormatter
    }()

    private let dayMonthTimeDateFormatter: DateFormatterType = {
        let dateFormatter = DateFormatter()
        dateFormatter.locale = Locale(identifier: "en_US_POSIX")
        dateFormatter.dateFormat = "dd MMM @ HH:mm"

        return dateFormatter
    }()

    private let hourMinuteDateFormatter: DateFormatterType = {
        let dateFormatter = DateFormatter()
        dateFormatter.locale = Locale(identifier: "en_US_POSIX")
        dateFormatter.dateFormat = "HH:mm"

        return dateFormatter
    }()

    private let dayMonthYearDateFormatter: DateFormatterType = {
        let dateFormatter = DateFormatter()
        dateFormatter.locale = Locale(identifier: "en_US_POSIX")
        dateFormatter.dateFormat = "d MMM 'of' y"

        return dateFormatter
    }()

    // MARK: - DOB

    func formatDOBDate(_ date: Date) -> String {
        let formattedDate = dobDateFormatter.string(from: date)
        return ("Date of birth: \(formattedDate)")
    }

    // MARK: - Account

    func formatLastActiveDate(_ date: Date, now: Date = Date()) -> String {
        let yesterday = Calendar.current.date(byAdding: .day, value: -1, to: now)!

        var dateFormatter = dayMonthTimeDateFormatter
        if date > yesterday {
            dateFormatter = hourMinuteDateFormatter
        }

        let formattedDate = dateFormatter.string(from: date)
        return ("Last active: \(formattedDate)")
    }

    // MARK: - Post

    func formatPostCreatedDate(_ date: Date) -> String {
        let formattedDate = dayMonthYearDateFormatter.string(from: date)
        return formattedDate
    }

    // MARK: - Commenting

    func formatCommentedDate(_ date: Date) -> String {
        let formattedDate = dayMonthTimeDateFormatter.string(from: date)
        return ("Comment posted: \(formattedDate)")
    }
}

This gives a better idea of what DateFormattingHelper can become. However, you can quickly see how just adding three more properties has bloated this class. There is room for further optimisation here.

In the above example the only configuration difference between the DateFomatter instances is the dateFormat value. This shared configuration can be used to our advantage to remove the distinct DateFormatter properties:

class CachedDateFormattingHelper {

    // MARK: - Shared

    static let shared = CachedDateFormattingHelper()
    
    // MARK: - Queue
    
    let cachedDateFormattersQueue = DispatchQueue(label: "com.boles.date.formatter.queue")

    // MARK: - Cached Formatters

    private var cachedDateFormatters = [String : DateFormatterType]()

    private func cachedDateFormatter(withFormat format: String) -> DateFormatterType {
        return cachedDateFormattersQueue.sync {
            let key = format
            if let cachedFormatter = cachedDateFormatters[key] {
                return cachedFormatter
            }
            
            let dateFormatter = DateFormatter()
            dateFormatter.locale = Locale(identifier: "en_US_POSIX")
            dateFormatter.dateFormat = format
            
            cachedDateFormatters[key] = dateFormatter
            
            return dateFormatter
        }
    }

    // MARK: - DOB

    func formatDOBDate(_ date: Date) -> String {
        let dateFormatter = cachedDateFormatter(withFormat: "y/MM/dd @ HH:mm")
        let formattedDate = dateFormatter.string(from: date)
        return ("Date of birth: \(formattedDate)")
    }

    // MARK: - Account

    func formatLastActiveDate(_ date: Date, now: Date = Date()) -> String {
        let yesterday = Calendar.current.date(byAdding: .day, value: -1, to: now)!

        var dateFormatter = cachedDateFormatter(withFormat: "dd MMM @ HH:mm")
        if date > yesterday {
            dateFormatter = cachedDateFormatter(withFormat: "HH:mm")
        }

        let formattedDate = dateFormatter.string(from: date)
        return ("Last active: \(formattedDate)")
    }

    // MARK: - Post

    func formatPostCreatedDate(_ date: Date) -> String {
        let dateFormatter = cachedDateFormatter(withFormat: "d MMM 'of' y")
        let formattedDate = dateFormatter.string(from: date)
        return formattedDate
    }

    // MARK: - Commenting

    func formatCommentedDate(_ date: Date) -> String {
        let dateFormatter = cachedDateFormatter(withFormat: "dd MMM @ HH:mm")
        let formattedDate = dateFormatter.string(from: date)
        return ("Comment posted: \(formattedDate)")
    }
}

Above we have added a private dictionary (cachedDateFormatters) where all the DateFormatter instances are stored and an accessor method (cachedDateFormatter) to determine if a new instance of DateFormatter needs to be created or not based on if it exists in that dictionary, with the dateFormat value itself acting as the key.

Setting or getting a cached DateFormatter instance has been wrapped in a sync operation on its own queue to make using cachedDateFormatter thread safe - effectively, the cachedDateFormattersQueue queue here is acting as lock.

Credit for the above dictionary approach must be given to Jordon Smith, who describes it in this post.

Of course, if further customisation is needed in each of the formatters this approach isn't suitable, however for this example I think it offers a very elegant solution.

In the completed playground accompanying this post, I've added the same tests to CachedDateFormattingHelper that I use for DateFormattingHelper to show that externally both implementations behave the same way.

Looking back at those dates

In this post we questioned the common belief that creating DateFormatter instances is expensive (yes they are), looked at ways that we could avoid paying too much for them (local variables, properties and singletons), delved deep on why exposing shared DateFormatter instances was a bad idea (unexpected shared mutable state), how we could avoid that shared mutable state (helper methods and an immutable protocol) and even how we can take caching DateFormatter instances a step beyond (hidden dictionary). Overall we covered a lot of ground and if you've made it this far (as you have) I believe you deserve whatever you've been denying yourself all day 😉.

For me it's a lovely hot chocolate with marshmallows ☕ - I can guarantee no unexpected shared state there!

Special thanks to Paul Pires for helping to come up with this idea of how to share DateFormatter instances.

Networking is central to how most apps add value to their users which is why significant effort has been put into creating an easy to use networking layer by both Apple and the open-source community.

When iOS was first released, NSURLConnection was the native networking suite it shipped with. However the NSURLConnection story actually began long before iOS as it was first released in 2003 with Safari, this meant that NSURLConnection was never designed for the magnitude of tasks that it ended up having to support. These add-ons resulted in NSURLConnection having a number of sharp edges - such as how much programming effort was required to construct the response of a request: gradually building up an NSMutableData instance by responding to the delegate calls. Making networking trickier than it needed to be. A lot of the code for making network requests was boilerplate code which allowed us to abstract it away into base/parent classes that could then be reused for multiple requests. This not-quite-fit-for-purpose networking approach coupled with boilerplate code resulted in us creating generic, well-encapsulated networking layers to hide all that nasty away.

Then great third-party libraries like AFNetworking began to appear that operated on top of NSURLConnection and looked to smooth out some of those sharp edges for us. AFNetworking did this by implementing the boilerplate that NSURLConnection required and presenting to the developer an easy to use block/closure based interface. This closure based networking approach proved to be hugely successful and eventually Apple decided to embrace it with URLSession. However not everything was rosy 🥀 - due to the ease of which we could now make networking requests and process the responses, you started to see a lot more networking code implemented directly in view controllers 💥. These bloated view controllers became responsible for multiple different domains - interacting with views, making network requests and parsing network responses into model objects. Classes that combine multiple domains are trickier to reason about, maintain and extend which is why the single responsibility principle is a very common design practice.

In this post, I want to explore how we can build an app that avoids bringing multiple domains together in its view controllers by having a distinct networking layer using URLSession and Operation.

Thinking about that layer

Whenever I think of layers in software engineering I always picture that great engineering marvel: the Hoover Dam.

A very visible, physical divider between two different domains - the reservoir before and the river after. In order for the water to continue to flow through the canyons, it must pass through the dam in the way that the dam dictates - just like how data flows around a system based on an agreed contract.

(Of course like any good metaphor, the dam one doesn't quite work because water should only flow one direction unlike our data but it will serve here as a visual aid)

Our networking layer will have 3 main responsibilities:

1. Scheduling network requests
2. Performing networking requests
3. Parsing the response from network requests

These responsibilities will come together to produce the following class structure:

Just like with the Hoover Dam our networking layer will have a well encapsulated outer edge that will transform data requests into distinct tasks that will then be scheduled for execution. Each task will consist of the actual network request and its response parsing 🌹.

Recapping

At the heart of this approach will be the ability to transform a network request and the subsequent parsing of its response into one task to be completed together. With OperationQueue and Operation, iOS already has a simple, well-supported mechanism of isolating distinct tasks within our apps. Combined OperationQueue and Operation form one of the options to support the concurrent execution of tasks.

Before continuing, let's take a small recap of what OperationQueue and Operation are.

An OperationQueue is a queue that controls the order of execution of its operations by their readiness value (must be true in order to start execution) and then scoring each operation by its priority level and time spent in the queue. This means that an OperationQueue is built on top of GCD which allows it to take advantage of an iOS device's multiple cores without the developer having to know anything about how it does this. By default, an OperationQueue will attempt to execute as many operations in parallel as the device is capable of supporting. As an OperationQueue maintains a link to its operations, it's possible to query those operations and perform additional tasks on them such as pausing or cancelling operations (which can be useful if the user logs out).

Operation is an abstract class which need to be subclassed to undertake a specific task. An Operation typically runs on a separate thread from the one that created it. Each operation is controlled via an internal state machine, the possible states are:

• Pending indicates that our Operation subclass has been added to the queue.
• Ready indicates that our Operation subclass is good to go and if there is space on the queue, this operation's task can be started
• Executing indicates that our Operation subclass is actually doing work at the moment.
• Finished indicates that our Operation subclass has completed its task and should be removed from the queue.
• Cancelled indicates that our Operation subclass has been cancelled and should stop its execution.

A typical operations lifecycle will move through the following states:

It's important to note that cancelling an executing operation will not actually stop that operation on its own, instead it is up to the individual operation to clean up after itself and move into the Finished state.

Operations come in two flavours:

• Non-Concurrent
• Concurrent

Non-Concurrent operations perform all their work on the same thread so that when the main method returns the operation is moved into the Finished state. The queue is then be notified of this and removes the operation from its active operation pool, freeing resources for the next operation.

Concurrent operations can perform some of their work on a different thread so returning from the main method can no longer be used to move the operation into a Finished state. Instead, when we create a concurrent operation we are assuming the responsibility for moving the operation between the Ready, Executing and Finished states.

Building the concurrent operation 🏗️

A networking operation is a specialised concurrent operation because when an URLSession is making a network request, it does so on a different thread from the thread that resumed that task. Rather than cramming everything into one operation, this solution will focus on building an abstract concurrent operation and more specific operations that actually do the networking.

This post will gradually build up to a working example however if time is tight, then head on over to the completed example and take a look at ConcurrentOperation, QuestionsRetrievalOperation, QuestionsDataManager and QueueManager to see how things end up.

As mentioned, a concurrent operation takes responsibility for ensuring that its internal state is correct. This state is controlled by manipulating the isReady, isExecuting and isFinished properties. However, these are read-only so these properties will need to overridden and an operations' current state tracked via a different property. When mapping state I often find it best to use an enum type:

class ConcurrentOperation: Operation {

    // MARK: - State

    private enum State {
        case ready
        case executing
        case finished
    }

    private var state = State.ready

    override var isReady: Bool {
        return super.isReady && state == .ready
    }

    override var isExecuting: Bool {
        return state == .executing
    }

    override var isFinished: Bool {
        return state == .finished
    }
}

In the code snippet above, the private mutable property state will track this internal state with the state value then being used in each of the overridden properties to return the correct boolean result. The cases of the State enum map directly to the state properties that are being overridden.

It's important to note that the isReady property is a little more complex than isExecuting or isFinished. One of the advantages of using operations is that it's possible to chain operations together so that one operation is dependent on another operation finishing before it can begin execution. If isReady just checked state == .ready, ConcurrentOperation would lose the ability to create dependencies, as an operation that has an unfinished dependency has an isReady value of false.

(I've seen examples of concurrent operation implementations that use 3 separate, settable state properties e.g. _isReady, _isExecuting and _isFinished. While this is a valid approach, I find having 6 properties that are so similarly named hard to read and led to class bloat)

In concurrent operations, it's not uncommon to see the isAsynchronous property being overridden to always return true however in this example these operations are always going to be executed via an operation queue so there is no need to actually override this property as the queue will just ignore that value.

OperationQueue uses KVO to know when its operations change state however so far ConcurrentOperation will not inform any observers. The key paths that the queue will observe, map directly with the enum cases. So we can transform our basic State enum into a String backed enum and use the raw-value of each to hold the expected key path string:

enum State: String {
    case ready = "isReady"
    case executing = "isExecuting"
    case finished = "isFinished"
}

With this new powerful enum, let's add property observers:

var state = State.ready {
    willSet {
        willChangeValue(forKey: newValue.rawValue)
        willChangeValue(forKey: state.rawValue)
    }
    didSet {
        didChangeValue(forKey: oldValue.rawValue)
        didChangeValue(forKey: state.rawValue)
    }
}

ConcurrentOperation will now trigger KVO notifications correctly but so far the state property isn't actually being mutated. The first state change is from Ready -> Executing which is handled by overriding the start method:

override func start() {
    guard !isCancelled else {
        finish()
        return
    }

    if !isExecuting {
        state = .executing
    }

    main()
}

The start method is the entry point into an operation, it is called when the operation should begin its work. In the above code snippet a check is made to see if the operation has been moved into the Cancelled state (yep, an operation can begin in a cancelled state) - if it has, finish is immediately called (this will be shown soon) which will move the operation into the Finished state and then return. If the operation hasn't been cancelled, it is moved into the Executing state and main is called (for the actual work to begin). Each subclass of ConcurrentOperation would implement main to perform its specific task. main is actually the entry point for non-concurrent operations and technically for a concurrent operation any method could have been chosen as the work method (provided each subclass implements it). However, by choosing main the cognitive load on any future developer is lessened by allowing them to transfer the expectation of how non-concurrent operations work to our concurrent operation implementation.

(In older versions of iOS, concurrent operations were responsible for calling main on a different thread - this is no longer the case)

Its important to note that super.start() isn't being called on purpose, as by overriding start this operation assumes full control of maintaining its state.

func finish() {
    if isExecuting {
        state = .finished
    }
}

finish acts as a nice symmetrical opposite of the start method and ensures that when an operation has finished that it is moved into the Finished state. It's essential that all operations eventually call this method - if you are experiencing odd behaviour where your queue seems to have jammed and no operations are being processed, one of your operations is probably missing a finish call somewhere.

Finally the cancel method needs to be overridden:

override func cancel() {
    super.cancel()

    finish()
}

It's important that regardless of how a concurrent operation ends that it moves into the Finished state and it does so when cancelled.

And that's it. By taking control of its state, the above operation has transformed itself from a boring non-concurrent operation into....a...well, slightly less boring concurrent operation.

Adding networking specific add-ons 📡

Before we start using ConncurentOperation however let's make one more change to make it more useful by adding a custom completion closure that will use the widespread Result enum type to return the outcome of the operations task:

enum Result {
    case success(T)
    case failure(Error)
}

class ConcurrentOperation: Operation {

    typealias OperationCompletionHandler = (_ result: Result) -> Void

    var completionHandler: (OperationCompletionHandler)?

    // omitting properties and methods that we have already seen

    func complete(result: Result) {
        finish()

        if !isCancelled {
            completionHandler?(result)
        }
    }
}

ConcurrentOperation now supports returning results of an unknown type T which each operation subclass will define. The custom completion closure is then triggered when the complete method is called.

And that's it, we have an abstract concurrent networking operation 🎓.

Now, let's build some concrete operations.

Making that call

StackOverflow has a wonderful, open API that will be used below to build a networking operation. The networking operation will retrieve and parse the latest questions which are tagged with iOS via the /questions endpoint.

Instead of building the operation piece by piece like we did above, lets look at the complete operation and then examine 👀 more closely the interesting parts:

class QuestionsRetrievalOperation: ConcurrentOperation {

    private let session: URLSession
    private let urlRequestFactory: QuestionsURLRequestFactory
    private var task: URLSessionTask?
    private let pageIndex: Int

    // MARK: - Init

    init(pageIndex: Int, session: URLSession = URLSession.shared, urlRequestFactory: QuestionsURLRequestFactory = QuestionsURLRequestFactory()) {
        self.pageIndex = pageIndex
        self.session = session
        self.urlRequestFactory = urlRequestFactory
    }

    // MARK: - Main

    override func main() {
        let urlRequest = urlRequestFactory.requestToRetrieveQuestions(pageIndex: pageIndex)

        task = session.dataTask(with: urlRequest) { (data, response, error) in
            guard let data = data else {
                DispatchQueue.main.async {
                    if let error = error {
                        self.complete(result: .failure(error))
                    } else {
                        self.complete(result: .failure(APIError.missingData))
                    }
                }
                return
            }

            do {
                let page = try JSONDecoder().decode(QuestionPage.self, from: data)

                DispatchQueue.main.async {
                    self.complete(result: .success(page))
                }
            } catch let error {
                DispatchQueue.main.async {
                    self.complete(result: .failure(APIError.serialization))
                }
            }
        }

        task?.resume()
    }

    // MARK: - Cancel

    override func cancel() {
        task?.cancel()
        super.cancel()
    }
}

(I won't show the QuestionPage, QuestionsURLRequestFactory and APIError in this post but if you are interested you can see them by cloning the example project here. QuestionPage is a model struct that will be populated with the questions endpoint JSON response. QuestionsURLRequestFactory is a helper factory for returning a URLRequest object configured for accessing the /questions endpoint. APIError is a custom Error enum)

In the main method above, an instance of URLSession is used to create a URLSessionDataTask which will make a network request and parse its response. Depending on the outcome of the network request and parsing of its response, complete is called and either an error or a parsed QuestionPage model is passed back wrapped into a Result. When completing an operation, special attention needs to be paid to ensure that the completionHandler property is used directly as this would cause the operation to end but not move it into the Finished state and what would result eventually is a jammed queue (as spoken about above).

In the cancel method, the URLSessionTask instance used to make the network request is cancelled so ensuring that no bandwidth is needlessly wasted.

And that's pretty much it for what a networking operation looks like. Of course, each different network operation will be unique but the general structure will be similar to QuestionsRetrievalOperation (if you want to see another concurrent operation example, take a look at this one).

Getting the managers involved

So far 2 of the 3 responsibilities of the networking layer shown above have been built:

• Performing networking requests
• Parsing the response from network requests

Time to look at the final responsibility:

• Scheduling network requests.

If we refer back to the class structure above, this responsibility is handled by both the DataManager and Scheduler classes.

class QuestionsDataManager {

    private let queueManager: QueueManager

    // MARK: - Init

    init(withQueueManager queueManager: QueueManager = QueueManager.shared) {
        self.queueManager = queueManager
    }

    // MARK: - Retrieval

    func retrievalQuestions(pageIndex: Int, completionHandler: @escaping (_ result: Result) -> Void) {
        let operation = QuestionsRetrievalOperation(pageIndex: pageIndex)
        operation.completionHandler = completionHandler
        queueManager.enqueue(operation)
    }
}

The above manager is responsible for creating, configuring and scheduling operations on a queue. While this example only shows one method, the idea here is that this manager would handle all networking requests for the /questions endpoint.

The sharper reader will have spotted another custom class in the above code snippet: QueueManager. This class is responsible for the queue (or queues) that the operations will be added to - it acts as the Scheduler class.

class QueueManager {

    lazy var queue: OperationQueue = {
        let queue = OperationQueue()

        return queue;
    }()

    // MARK: - Singleton

    static let shared = QueueManager()

    // MARK: - Addition

    func enqueue(_ operation: Operation) {
        queue.addOperation(operation)
    }
}

The above manager creates a shared OperationQueue instance and controls access to it via the enqueue. To ensure that operations are consistently added to the same queue, this manager is a singleton. While this class only has the one queue it can be easily extended to hold multiple queues (perhaps one queue for user-driven events and another for system events - allowing greater control on suspending/cancelling queues) with each DataManager specifying which queue to add the operation to by passing an additional parameter when enqueuing that operation.

Looking at what we have

While using operations adds to the complexity of making a network request, when we step back and look at the benefit that having an isolated, well-encapsulated networking layer brings to our apps, I hope that we can all agree it is a cost worth paying 💰.

To see this networking layer in action, head over to the repo and clone the project.

In the past iOS was always about living in the moment - totally focused on whatever the user was attempting to do. This approach was good news for users as it allowed for a very responsive user experience on performance constrained devices however as app developers it limited what we could do because as soon as an app went into the background iOS suspended it. As devices have become more powerful and energy-efficient, iOS has eased the "living in the moment" restrictions while maintaining responsiveness.

One way that iOS has eased restrictions is allowing apps to start/continue download and upload requests in the background - collectively known as background-transfers. Support for background-transfers was first introduced in iOS 7, so it's not a recent change, but it's such a powerful tool that I wanted to explore it further and especially show a solution for recovering from a terminated state.

Different types of sessions

When Apple introduced the URLSession suite of classes, they addressed a long standing complaint - how to configure the networking stack to handle various types of network requests that an app has to make. Rather than having one networking stack like with NSURLConnection, NSURLSession instead allowed for multiple independent networking stacks (or sessions as they are known). The configurations of these independent sessions are controlled through URLSessionConfiguration. The main session configurations are:

1. Default - allowing for all types of network requests in the foreground.
2. Ephemeral - similar to default but more forgetful (doesn’t write caches, cookies, or credentials to disk).
3. Background - allowing for downloading and uploading content even when the app isn't running.

Of course, it's possible to further differentiate sessions within those configurations by adjusting the properties on each session.

At the time URLSession was introduced I was building a very media-heavy app, and the promise of being able to continue to upload or download content when the app wasn't running by using a background session was very appealing.

Peeking under the hood 🕵️

When scheduling a background-transfer on a background session, that transfer is passed to the nsurlsessiond daemon, which runs as a separate process from the app, to be actioned. As this daemon lives outside of the lifecycle of the app if the app is suspended or killed the transfer will continue unaffected - this is the main difference from scheduling an URLSessionDownloadTask or URLSessionUploadTask request on a default or ephemeral session where the request is tied to the lifecycle of the app. Once a background-transfer is complete, if the app has been suspended/terminated iOS will wake it up (in the background) and allow the app to perform any post-transfer work (within a limited time frame). If the app is in the foreground, control will be passed back to the app as if the transfer has been scheduled on default or ephemeral session (without a limited time frame for post-transfer processing).

Important to note here that you can schedule URLSessionDataTask requests on a background session but they will only be acted on when the app is in the foreground.

You're probably thinking:

"That sounds pretty amazing! Why isn't this the default behaviour for transfers?"

Well, there are a few reasons:

• The more background processing that the device has to support the quicker the battery will drain and the more bandwidth that will be consumed. So while Apple understands that background processing is needed, it wants to ensure that as app developers we use it responsibly and only where it's actually adding value.

• From a developer-POV supporting background-transfers requires a bit more programming than a foreground-only session. Rather than the simple closure based API that foreground-only sessions support, to use background sessions the developer needs to conform to various protocols. This additional programming while not hugely taxing (as we shall see shortly) does require each developer to know more about how URLSession works before they can make a network request. So in true Apple fashion, they have opted to not only keep it simple for the users but also for us developers.

Now that the hood has been peeked under, let's get back to how to use background-transfers to provide a better experience for our users.

Scenarios

Any background-transfer solution should be able to handle 3 different transfer scenarios:

1. Foreground - when the app is open, the transfer should behave the same way as if the session was using the default configuration.
2. Suspended/Backgrounded - when the app is in a suspended state and the asset is downloaded, the app should be able to be woken up to finish the transfer.
3. Terminated - when the app has been terminated by iOS, the app should be able to be relaunched to finish that transfer.

In the rest of this post, we will build an image download system that will look to handle the 3 scenarios described above. This example will populate a collectionview using data retrieved from Imgur with all image retrieval happening on a background session.

This post will focus solely on downloading using a background session as it's much easier to find services that allow us to retrieve content from their system than services that allow us to upload content.

This post will gradually build up to a working example however if you're on a tight deadline and/or there is murderous look creeping into your manager's eyes 😡, then head on over to the completed example and take a look at BackgroundDownloader, BackgroundDownloaderContext, BackgroundDownloadItem and AppDelegate to see how things end up - in order to run this you will need to follow the instructions below.

Let's get downloading

To support background-transfers we need to grant permission for Background Modes. This is done by opening the Capabilities tab in the target and switching the Background Modes toggle to ON.

As multiple viewcontrollers within the project could be retrieving images I decided to handle these download requests within their own class - BackgroundDownloader. BackgroundDownloader will be responsible for configuring our background session, scheduling download requests against that session and responding to any delegate callbacks.

Lets build the skeleton of the BackgroundDownloader class:

class BackgroundDownloader: NSObject {

    private var session: URLSession!

    // MARK: - Singleton

    static let shared = BackgroundDownloader()

    // MARK: - Init

    private override init() {
        super.init()

        let configuration = URLSessionConfiguration.background(withIdentifier: "background.download.session")
        session = URLSession(configuration: configuration, delegate: self, delegateQueue: nil)
    }
}

In the above code snippet, we create an instance of an URLSessionConfiguration instance and URLSession that will handle any background tasks. configuration is initialised using the background convenience init'er to allow any URLSession instances to support background-transfers. Each URLSessionConfiguration instances needs to have an identifier. This identifier plays a critical role in allowing a background-transfer that started on one instance of URLSession to be completed on another. This happens when the app has been terminated and iOS wakes the terminated app up to finish the transfer - configuration uses a static string as an identifier so it can be recreated (we will see this in action later). It's important to note here that each session needs to have a unique identifier within that app's execution lifecycle. As BackgroundDownloader is a singleton, only one session will have this identifier.

An interesting difference between background sessions and the other session types is that with background sessions we can't use the completionHandler methods when creating a URLSessionDownloadTask task (doing so won't result in the compiler throwing an error, but the app will throw an exception at run time). Instead of using the closure approach, BackgroundDownloader will need to implement the URLSessionDownloadDelegate protocol (and eventually URLSessionDelegate) - this need to implement these delegates is the reason that BackgroundDownloader is a subclass of NSObject.

The more eagle-eyed among you will have noticed that the session property is implicitly unwrapped. I usually try and avoid implicitly unwrapped properties but as session is a private property and is set in the init'er I felt making it implicitly unwrapped resulted in more readable code with very little danger of a crash. The reason that session can't be a let is that when creating the session, the BackgroundDownloader instance sets itself as the session's delegate so the BackgroundDownloader has to exist which means that the initialising of the session has to happen after the super.init() call is made.

So far, we have configured a session for performing background-transfers but don't yet have the ability to download anything so let's add that. A download needs 3 pieces of information:

1. Remote URL - URL of the asset to be downloaded.
2. File path URL - URL of where the downloaded asset should be moved to on the local file system.
3. Completion handler - a closure to be called when the download is complete.

Rather than store these pieces of information about each download in separate arrays, we can create a model class to group these related pieces of information:

typealias ForegroundDownloadCompletionHandler = ((_ result: DataRequestResult) -> Void)

class DownloadItem {

    let remoteURL: URL
    let filePathURL: URL
    var foregroundCompletionHandler: ForegroundDownloadCompletionHandler?

    // MARK: - Init

    init(remoteURL: URL, filePathURL: URL) {
        self.remoteURL = remoteURL
        self.filePathURL = filePathURL
    }
}

An instance of DownloadItem will be created when a new download request is made against BackgroundDownloader. This DownloadItem instance can then be stored in a dictionary to be used once the download is complete:

class BackgroundDownloader: NSObject {

    // Omitted properties

    private var downloadItems: [URL: DownloadItem] = [:]

    // Omitted methods

    func download(remoteURL: URL, filePathURL: URL, completionHandler: @escaping ForegroundDownloadCompletionHandler) {
        print("Scheduling download of: \(remoteURL)")

        let downloadItem = DownloadItem(remoteURL: remoteURL, filePathURL: filePathURL)
        downloadItem.foregroundCompletionHandler = completionHandler
        downloadItems[remoteURL] = downloadItem

        let task = session.downloadTask(with: remoteURL)
        task.resume()
    }
}

It's not unusual to have a scenario where before a download request is finished another request for that same download is made - think of a user scrolling a tableview up and down. There are many articles written on how to prevent these additional (unneeded) download requests so I won't complicate this example about background-transfers by implementing any of them here. However if you are really interested on how to prevent these additional download requests, check out my coalescing closures article on this subject which shows one possbile approach.

The URLSessionDownloadDelegate protocol has been mentioned a few times now so lets see how it's implemented:

extension BackgroundDownloader: URLSessionDownloadDelegate {

    func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) {
        guard let originalRequestURL = downloadTask.originalRequest?.url, let downloadItem = downloadItems[originalRequestURL] else {
            return
        }

        print("Downloaded: \(downloadItem.remoteURL)")

        do {
            try FileManager.default.moveItem(at: location, to: downloadItem.filePathURL)

            downloadItem.foregroundCompletionHandler?(.success(downloadItem.filePathURL))
        } catch {
            downloadItem.foregroundCompletionHandler?(.failure(APIError.invalidData))
        }

       downloadItems[originalRequestURL] = nil
    }
}

If the download has been a success, the asset is moved from its temporary location to its permanent location on the file system. Then the completionHandler is triggered with the success case of the DataRequestResult enum containing the permanent location URL.

If the download has failed the completionHandler is triggered with the failure case containing an error.

(This method is deliberately kept as simple as possible but you can easily imagine how it and DownloadItem could be extended to e.g. update Core Data or perform image manipulation)

Finally we have been using DataRequestResult enum as a generic result enum so lets look at as well:

enum DataRequestResult<T> {
    case success(T)
    case failure(Error)
}

And that's all that's required for performing foreground transfers using a background configuration.

If you want to see this version of BackgroundDownloader running, checkout out the foreground_only_transfers branch on the example project - to run this you will need to follow the instructions below.

Working behind the users back

If you run the app in the foreground you will notice that everything works as expected and the app is populated with downloaded assets however as soon as the app is put into the background, a stream of errors will begin to appear in the Xcode console, similar to:

So far BackgroundDownloader only works when the app is in the foreground, let's extend it to support downloads when the app enters a suspended/backgrounded state.

When an app is in a suspended/terminated state and a download is completed, iOS will wake the app up (in the background) by calling:

class AppDelegate: UIResponder, UIApplicationDelegate {

    // Omitted properties and methods

    func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @escaping () -> Void) {
        //TODO: Handle processing downloads
    }
}

Regardless of how many downloads are in progress, application(_:handleEventsForBackgroundURLSession:completionHandler:) is only called once - when all downloads have been completed. If the user foregrounds the app before the above method is called, urlSession(_:downloadTask:didFinishDownloadingTo:) is called for those downloads that have been completed.

In the above method, iOS has provided the app with the identifier of the session that the task was scheduled on and also a completion-closure. As our background session always uses the same identifier, we don't need to do anything with identifier however we do need to pass completionHandler onto BackgroundDownloader so that it can be triggered once all download processing has been completed:

class AppDelegate: UIResponder, UIApplicationDelegate {

    // Omitted properties and methods

    func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @escaping () -> Void) {
        BackgroundDownloader.shared.backgroundCompletionHandler = completionHandler
    }
}

As the completionHandler closure is passed to the shared BackgroundDownloader instance we need to add a new property to that class:

class BackgroundDownloader: NSObject {

    var backgroundCompletionHandler: (() -> Void)?

    // Omitted properties and methods
}

When all the downloads are are complete, just like before urlSession(_:downloadTask:didFinishDownloadingTo:) is be called for each download to allow any post-download processing to occur. After all the calls to urlSession(_:downloadTask:didFinishDownloadingTo:) have been made, a special background-transfer only call is made:

extension BackgroundDownloader: URLSessionDelegate {

    func urlSessionDidFinishEvents(forBackgroundURLSession session: URLSession) {
        DispatchQueue.main.async {
            self.backgroundCompletionHandler?()
            self.backgroundCompletionHandler = nil
        }
    }
}

The above method's main job is to allow backgroundCompletionHandler to be triggered. Triggering backgroundCompletionHandler instructs iOS to take a new snapshot of the app's UI for the app switcher preview - as such it needs to be called from the main queue (urlSessionDidFinishEvents(forBackgroundURLSession:) may be called from a background queue). It's important to ensure that your app does indeed call backgroundCompletionHandler - failure to do so will result in iOS treating the app as a bad citizen app; reducing the opportunity for any background processing in the future and resulting in the error that we seen at the start of this section.

While it's possible to make additional network requests with the app running in the background, they are subject to a rate limiter to prevent abuse of background sessions - this rate limiter delays the start of those network requests. With each separate network request the delay increases - this delay is reset when the user brings the app to the foreground or if your app does not make any additional network requests during the delay.

And that's all the changes required to continue network requests when the app is suspended/backgrounded.

If you want to see this version of BackgroundDownloader running, checkout out the foreground_and_suspended_transfers branch on the example project - to run this you will need to follow the instructions below. In order to better see the downloads happening in the background you may want to add a slight delay before the download actually starts so as to allow you to more leisurely put the app into the background:

task.earliestBeginDate = Date().addingTimeInterval(5)

Coming back from the dead 🧟

After an app has been terminated, all of the memory which was allocated to it is freed. This means that when iOS relaunches the app to complete a background-transfer we don't have the details in memory that we can use to determine how that background-transfer should be completed. Instead, these details need to be persisted in the file system between app executions. In our example, the only information that we really need to persist is filePathURL (as you will see the remoteURL is also persisted but this is primarily for informational purposes) so introducing something like Core Data to handle this would be overkill. Instead, let's stick with KISS principles and take advantage of the Codable protocol. By conforming DownloadItem to Codeable we can encode that DownloadItem object into its Data representation which can then be stored in User Defaults. The only issue with this approach is that the foregroundCompletionHandler property on DownloadItem can't conform to Codable, so the DownloadItem implementation of Codable will be slightly more involved than usual. Thankfully Codable has an easy way to declare which properties should and shouldn't be encoded.

class DownloadItem: Codable {

    // Properties omitted

    private enum CodingKeys: String, CodingKey {
        case remoteURL
        case filePathURL
    }

    // Methods omitted
}

With these changes, the DownloadItem class now has a dual nature:

1. Freshly created.
2. Restored from User Defaults.

This presents a problem with determining which class should be responsible for handling this dual nature - knowing how to save into, load from and clean up User Defaults. I toyed with placing this responsibility in DownloadItem but wanted to keep this class as simple as possible which ruled that approach out. Then I moved onto BackgroundDownloader but again felt that it wasn't the correct class for handling this - I didn't want to expose that some DownloadItem instances were freshly created while some had been restored from User Defaults to BackgroundDownloader whose primary responsibility centred on downloading rather than persistence. In the end, I drew inspiration from Core Data and decided to create a downloading context.

This context class needs to handle 3 different tasks:

1. Loading.
2. Saving.
3. Deleting.

Lets start building this BackgroundDownloaderContext class:

class BackgroundDownloaderContext {

    private var inMemoryDownloadItems: [URL: DownloadItem] = [:]
    private let userDefaults = UserDefaults.standard

}

Like before, a simple dictionary is being used as an in-memory store which will hold all active DownloadItem instances.

When it comes to loading/retrieving an existing DownloadItem instance, this can be from two possible stores:

1. In-memory.
2. User Defaults.

Again, we don't want to expose these details so let's add a load method to BackgroundDownloaderContext that can load from either of the two possible stores:

class BackgroundDownloaderContext {

    // Omitted properties

    func loadDownloadItem(withURL url: URL) -> DownloadItem? {
        if let downloadItem = inMemoryDownloadItems[url] {
            return downloadItem
        } else if let downloadItem = loadDownloadItemFromStorage(withURL: url) {
             inMemoryDownloadItems[downloadItem.remoteURL] = downloadItem

            return downloadItem
        }

        return nil
    }

    private func loadDownloadItemFromStorage(withURL url: URL) -> DownloadItem? {
        guard let encodedData = userDefaults.object(forKey: url.path) as? Data else {
            return nil
        }

        let downloadItem = try? JSONDecoder().decode(DownloadItem.self, from: encodedData)
        return downloadItem
    }
}

First, the in-memory store is checked for an existing DownloadItem using the network request's URL as the key. If an existing DownloadItem instance isn't found in the in-memory store, the User Defaults store is checked. If an object is found in User Defaults, an attempt is made to decode it from its raw Data representation to a new DownloadItem instance. As you will have spotted, decoding can potentially throw an exception, if an exception is thrown it will be silently ignored due to the try? statement. I chose this approach as an error during decoding wouldn't be repairable, instead just returning nil would suffice.

In order to get DownloadItem instances into User Defaults we need to save them:

class BackgroundDownloaderContext {

    // Omitted properties and methods

    func saveDownloadItem(_ downloadItem: DownloadItem) {
        inMemoryDownloadItems[downloadItem.remoteURL] = downloadItem

        let encodedData = try? JSONEncoder().encode(downloadItem)
        userDefaults.set(encodedData, forKey: downloadItem.remoteURL.path)
        userDefaults.synchronize()
    }
}

In the above code snippet, the DownloadItem instance is first placed into the in-memory store before being saved into User Defaults as a Data object.

Finally when a download is complete the in-memory and User Default objects need to be cleaned away:

class BackgroundDownloaderContext {

    // Omitted properties and methods

    func deleteDownloadItem(_ downloadItem: DownloadItem) {
      inMemoryDownloadItems[downloadItem.remoteURL] = nil
      userDefaults.removeObject(forKey: downloadItem.remoteURL.path)
      userDefaults.synchronize()
    }
}

By implementing the BackgroundDownloaderContext, the BackgroundDownloader logic can be kept pretty much the same as before despite this significant introduction of new functionality. First we need to add a new property to hold an instance of BackgroundDownloaderContext and remove the existing downloadItems property

class BackgroundDownloader: NSObject {

    private let context = BackgroundDownloaderContext()

    // Omitted properties and methods
}

Now let's use that context when requesting a new download:

class BackgroundDownloader: NSObject {

    // Omitted properties and methods

    func download(remoteURL: URL, filePathURL: URL, completionHandler: @escaping ForegroundDownloadCompletionHandler) {
        print("Scheduling download of: \(remoteURL)")

        let downloadItem = DownloadItem(remoteURL: remoteURL, filePathURL: filePathURL)
        downloadItem.foregroundCompletionHandler = completionHandler
        context.saveDownloadItem(downloadItem)

        let task = session.downloadTask(with: remoteURL)
        task.resume()
    }
}

Then when a download is completed, the DownloadItem instance needs to be deleted from the context:

extension BackgroundDownloader: URLSessionDownloadDelegate {

    func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) {
        guard let originalRequestURL = downloadTask.originalRequest?.url, let downloadItem = context.loadDownloadItem(withURL: originalRequestURL) else {
            return
        }

        print("Downloaded: \(downloadItem.remoteURL)")

        do {
            try FileManager.default.moveItem(at: location, to: downloadItem.filePathURL)

            downloadItem.foregroundCompletionHandler?(.success(downloadItem.filePathURL))
        } catch {
            downloadItem.foregroundCompletionHandler?(.failure(APIError.invalidData))
        }

       context.deleteDownloadItem(downloadItem)
    }
}

Again, here we have managed to avoid exposing to the BackgroundDownloader if that downloadItem was purely an in-memory object or if it also had a backing User Defaults entry.

As with the suspended/backgrounded code changes, when the app is awoken from a terminated state the app-delegate is called:

class AppDelegate: UIResponder, UIApplicationDelegate {

    // Omitted properties and methods

    func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @escaping () -> Void) {
      BackgroundDownloader.shared.backgroundCompletionHandler = completionHandler
    }
}

This is actually the exact same code as in the suspended/backgrounded example above but I wanted to highlight that when the app is awoken from a terminated state, this method not only passes the completionHandler along but also sets up the BackgroundDownloader (by calling shared) so that it can respond to the session delegate calls.

An interesting point around terminating the app is that if you manually force quit the app then iOS will take this as definite confirmation that you are finished with the app and so cancel any scheduled background-transfers. This presents a problem when it comes to testing how our solution handles termination. The easiest way I found to overcome this issue was to add a manual exit call to cause the app to terminate without any user involvement:

class AppDelegate: UIResponder, UIApplicationDelegate {

    // Omitted properties and methods

    func applicationDidEnterBackground(_ application: UIApplication) {
        //Exit app to test restoring app from a terminated state. Comment out to test restoring app from a suspended state.
        DispatchQueue.main.asyncAfter(deadline: .now()) {
            print("App is about to quit")

            if let documentsPath = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true).first {
                debugPrint("Gallery assets will be saved to: \(documentsPath)")
            }
            exit(0)
        }
    }
}

In the above code snippet, the file path of the directory that the downloaded assets will end up in is printed to the console and then the app terminates. With this file path, you can then navigate to that directory and see the folder fill up with downloaded assets.

Downloads keep going 🤯

With background sessions, we have a powerful weapon in ensuring that we can meet our user's wants without having to trap them in the app. As we seen above adding support for background sessions isn't too tricky and what extra complexity there is, can mostly be hidden from the wider app.

Congratulations on making it to the end 🎉.

Running the example project

In "Don't throw anything away with pausable downloads" I used Imgur as an online image database and I've used it again in this post to power the example project. Imgur has a great JSON based API and an extensive library of freely available media. Imgur API, while being free, does require us to register our example project to get a client-id which needs to be sent with each request.

As the example project does not require access to a user's personal data, when registering select the Anonymous usage without user authorization option for Authorization type. At the time of writing, you had to provide a URL in the Authorization callback URL field even for anonymous authentication - I found any URL value would work for this.

After you register, you should be given a unique client-id which will allow you access to Imgur's content. You will need to add your client-id as the value of the clientID property in RequestConfig. After you've added your client-id, you should be able to run the app and see how our background-transfer solution works.

It's 17:00 on a warm Friday evening, as you type the final characters into your PR description your mind drifts to the coming weekend. During that weekend you decide to work on that open-source you cloned a while back, so you head to GitHub to look through the open issues. Like many developers, you use the same GitHub account for both your personal and work projects so when you open GitHub, you can't avoid seeing that a colleague has spent some of their weekend leaving comments on your PR. You're now faced with a dilemma, do you:

A. Open your PR and read those comments.

Or

B. Try and ignore those comments.

Choosing A means that you spend a portion of your weekend completing work-related tasks when you should be relaxing and recharging. Do this too often and those out-of-work tasks you are completing will lead to your in-work performance dropping and may even lead to you burning out.

Choosing B means that throughout the rest of the weekend, you repeatedly return to that PR in your thoughts wondering what issues someone found in your solution. Not knowing what someone has commented on can be low-level stressful. This stress is because as humans, we are programmed to remind ourselves about any uncompleted tasks. Your colleague leaving comments means that your previously completed task is now incomplete. In psychology, this is known as the Zeigarnik effect and will stop you from leaving work behind to focus on whatever you are doing during the weekend.

As you can see choosing A or B leads to a negative outcome. The only way to truly enjoy your weekend is to leave work behind. In this case - won't you don't know can't hurt you.

I found that having my mixed GitHub account kept stressing me out. As I couldn't avoid seeing work-related updates on the GitHub homepage when doing some recreational programming. I don't want to stop the recreational programming because I love it, but I couldn't let my mental health continue to suffer. The only way I could think to prevent this unneeded stress was to disconnect my personal and work GitHub usage by creating multiple GitHub accounts.

However, creating and using multiple GitHub accounts on the same computer proved to be slightly trickier than I thought it would be.

Adding multiple accounts

After creating a work-only GitHub account, I needed to generate an SSH key so that I could access that account from my machine. The only difference from the documented Git setup is that I couldn't use id_rsa as the file name to store my work account's SSH key (as it was storing my personal SSH key) instead I had to use a different name, i.e. id_rsa_work.

Now that I had both SSH keys on my Mac, I needed a way to inform Git on which one to use with which repo.

There are a number of detailed articles already written on this subject and I ended up with a SSH config (~/.ssh/config) of:

Host personal
  HostName github.com
  User git
  AddKeysToAgent yes
  UseKeychain yes
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_rsa

Host work
  HostName github.com
  User git
  AddKeysToAgent yes
  UseKeychain yes
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_rsa_work

The idea being that when cloning a repo that I would specify which account to use and any actions after on that repo would happen as that account i.e.

git clone personal:wibosco/CoreDataServices.git

or

git clone git@personal:wibosco/CoreDataServices.git

However, sadly, I wasn't able to make this work on my Mac (Mojave or Catalina). No matter how I manipulated the structure of my SSH config I ended up back at the same issue - Git always attempted to use the last key added to the SSH Agent regardless of which account I specified when cloning that repo.

Just as I started to despair that there was no way to get this working, the thought occurred to me:

"Well, if SSH Agent is always serving the last key added I can just only tell it about the key for the account I want to use"

If the SSH Agent only knew about one SSH key, that key was the one it would serve.

So the first thing to do is make the SSH Agent forget about my personal and work SSH keys by removing them:

ssh-add -D

Once all SSH keys were removed (from the Agent, not the system) I could then add the correct key for the repo I wanted to access, so if it were a personal repo I would add my personal key to the SSH Agent:

ssh-add -K ~/.ssh/id_rsa

Then when I used any Git commands, they would be executed as my personal account. If I wanted to use my work account, I would remove the personal key and add my work key. While it was annoying having to keep switching between accounts, it worked!

Executing ssh-add -l shows you which keys are loaded.

Even with the inconvenience of having to manually add and remove keys from the SSH Agent I was able to improve my work-life balance by removing that general sense of unease I would get by being able to avoid work-related updates during my personal time. I don't have any data on how this change impacted my productivity, but I imagine that this made me more productive.

Recently, I've been experiencing the iOS equivalent of the movie Inception - putting a collection view inside a collection view. While exploring possible solutions, I stumbled upon this very informative article by Soroush Khanlou and his suggestion that the best way to implement a collection view inside a collection view was by using child view controllers - with each child view controller implementing its own collection view and having its view added as a subview on one of the parent view controller's cells.

If you haven't read that article, I recommend that you do as it sets out the argument for why you would want to put view controllers inside cells very well. And even if you don't need to be convinced I would still recommend as the rest of this article won't make sense if you don't 😉.

The article itself is a few years old with the examples are in Objective-C, so I converted it over to Swift and plugged the solution into my project. I ended with the following collection view cell subclass:

class HostedViewCollectionViewCell: UICollectionViewCell {

    // MARK: - HostedView

    // 1
    weak var hostedView: UIView? {
        didSet {
          // 2
          if let oldValue = oldValue {
              oldValue.removeFromSuperview()
          }

          // 3
          if let hostedView = hostedView {
              hostedView.frame = contentView.bounds
              contentView.addSubview(hostedView)
          }
        }
    }

    // MARK: - Reuse

    // 4
    override func prepareForReuse() {
        super.prepareForReuse()

        hostedView = nil
    }
}

1. Each cell holds a reference to a (child) view controller's view via the hostedView. Whenever that hostedView property is set, the above didSet observer code is triggered.
2. If hostedView was previously set that old hostedView is removed from the cells view hierarchy.
3. If the current hostedView value is non-nil, it is added as a subview of the cell's contentView.
4. To improve performance, a collection view will only create enough cells to fill the visible UI and then a few more to allow for smooth scrolling. When a cell scrolls off-screen, it is marked for reuse on a different index path. prepareForReuse() is called just before the cell is reused and gives us the opportunity to reset that cell. In the above prepareForReuse(), hostedView is set to nil so triggering its removal from the cells view hierarchy.

To begin with, this solution worked well. However, I started noticing that occasionally a cell would forget its content. It was infrequent and would be resolved by scrolling the collection view. However, I was pretty dissatisfied by experience and wanted to understand what was causing this UI breakdown.

Looking over that prepareForReuse() method, you can see that removeFromSuperview() is called to do the removing - what's interesting about removeFromSuperview() is that it takes no arguments and instead uses the soon-to-be-removed view's superview value to determine what view to remove the caller from. As a view can only have one superview if a view which already has a superview is added as a subview to a different view that original connection to the first superview is broken and replaced with this new connection. For the most part, this 1-to-M mapping between a superview and its subviews works just fine as most views once added as subviews do not tend to move around. However, cells are designed to be reused. The reusable nature of cells lies at the root of my cells forgetfulness. By using the solution above we end up the following unintended associations:

The diagram above shows how multiple cells can be associated (shown in purple) with the same view controller's view, but only one of those cells has the view of ViewController as a subview (shown in green). Because of the multiple references kept to the same view of ViewController it's possible for any of Cell A, Cell B or Cell C to remove the hostedView from Cell C by calling removeFromSuperview() in their own prepareForReuse() method. Of course, it was not intentional for multiple cells to have an active reference to a view controller's view if that view was no longer part of the cell's view hierarchy.

Once those unintended left-over hostedView references were spotted, the solution for the bug became straightforward - only remove the hostedView if it is still in the cell's view hierarchy:

class HostedViewCollectionViewCell: UICollectionViewCell {
    var hostedView: UIView? {
        didSet {
            if let oldValue = oldValue {
                if oldValue.isDescendant(of: self) { //Make sure that hostedView hasn't been added as a subview to a different cell
                    oldValue.removeFromSuperview()
                }
            }

            //Omitted rest of observer
        }
    }

    //Omitted methods
}

So now the cell will only remove the hostedView from its superview if that superview is part of the cell. This additional if statement addresses the forgetfulness that I was seeing. However, if you queried hostedView on Cell A or Cell B from the above diagram you would still get back a reference to the view that Cell C. With the above if statement, we have only resolved part of the bug. Let's make the cell only return a hostedView value if that hostedView is actually part of its view hierarchy:

class HostedViewCollectionViewCell: UICollectionViewCell {

    // MARK: - HostedView

    // 1
    private weak var _hostedView: UIView? {
        didSet {
            if let oldValue = oldValue {
                if oldValue.isDescendant(of: self) { //Make sure that hostedView hasn't been added as a subview to a different cell
                    oldValue.removeFromSuperview()
                }
            }

            if let _hostedView = _hostedView {
                _hostedView.frame = contentView.bounds
                contentView.addSubview(_hostedView)
            }
        }
    }

    // 2
    weak var hostedView: UIView? {
        // 3
        get {
            guard _hostedView?.isDescendant(of: self) ?? false else {
                _hostedView = nil
                return nil
            }

            return _hostedView
        }
        //4
        set {
            _hostedView = newValue
        }
    }

    //Omitted methods
}

1. The private _hostedView property assumes the responsibilities of the previous hostedView property implementation and acts as backing-store property to the new hostedView property implementation. The _hostedView is what now actually holds a reference to the view controller's view even though the outside world still thinks it is hostedView that holds the reference. Just like before there the didSet observer which checks if the the oldValue of the _hostedView isn't nil and if it isn't, removes that view from the cell's view hierarchy if _hostedView is still a subview. If the current value of _hostedView is not nil, _hostedView is added as a subview of the current cell.
2. The externally accessible hostedView property now has a custom get and set.
3. The get only returns the _hostedView value if that view is still a subview of the cell. If hostedView isn't a subview, the get sets _hostedView to nil (which will cause it to be removed from this cell's view hierarchy) and returns nil. Dropping the reference once something tries to access hostedView feels a little strange. However, as the superview property of UIView isn't KVO compliant (and we don't want to get involved in the dark-arts of method swizzling to make it so) there is no way for us to know that the hostedView has a new superview without querying the hostedView and there is no point in querying the hostedView until something tries to access it - so a little self-contained strangeness is the only viable option here.
4. The set takes the new value and assigns it to the backing-store property.

With a backing-store property it is essential that you practice good access hygiene - _hostedView should only be directly accessed in either hostedView or _hostedView everywhere else should use hostedView.

You can see that by truthfully returned hostedView we have added a lot of complexity to our cell, but none of that complexity leaks out and we can have confidence that hosting a view controller's view won't lead to unintended consequences.

Seeing the 🐛 for yourself

If you want to see this bug in the wild, you can download the example project from my repo. In that example project, I've added logging for when the view controller's view isn't a subview on that cell anymore so that you can easily see when that bug would have happened by watching the console. If you want to see the bug's impact on the UI, comment out the isDescendant(of:_) check, in HostedViewCollectionViewCell.

Core Data and Unit Testing haven't always been the best of friends. Like two members of the same friend group who don't really know each other but really like their UIKit friend, Core Data and Unit Testing have in fact discovered that they have a lot in common and have gradually got more and more friendly with each other.

But before we delve into how they can take it one step further and become firm friends, we need to understand what makes each of them tick.

Getting to know each other

Core Data

Core Data is an object graph manager that helps you create the model layer of your app. An object graph is a collection of objects connected to each other through relationships. Core Data abstracts away the lifecycle of the objects it controls, providing tools to allow CRUD operations to be performed on those objects in its graph. In one configuration, Core Data's object graph can be persisted between executions of the app, with those objects being persisted in either XML, binary, or SQLite stores. However in another configuration, the object graph exists purely in-memory and dies when the app is killed. In fact, the range of configuration options available to Core Data makes it very difficult to define what Core Data actually is - in one configuration Core Data can resemble a SQL wrapper while in a different configuration it looks more like an ORM. The truth is that Core Data sits somewhere between both, with the ability to intelligently load a subset of its store's data into memory as custom domain objects that can be manipulated like any other Swift object before being "saved" back to the store where those changes are then persistent (be that across app executions or just in that app execution).

A consequence of this flexibility is it's not uncommon to see two apps using Core Data in significantly different ways. This has led to Core Data gaining a reputation as a hard to use framework (not helped by the significant changes that the framework has gone through since inception). To avoid any confusion around what configuration of Core Data we are going to use, in this post our setup will look like:

• Persistent Container is a fairly new member of the Core Data family. It was introduced in iOS 10 to help simplify the creation of the managed object model, persistent store coordinator and any managed object contexts.
• Managed Object Context is a temporary, in-memory record of all NSManagedObject instances accessed, created or updated during its lifecycle. An app will typically have multiple contexts in existence at any one given time. These contexts will form a parent-child relationship. When a child context is saved it will push its changes to its parent's context which will then merge these changes into its own state. At the top of this parent-child hierarchy is the main context, this context upon being saved will push its changes into the persistent store.
• Persistent Store Coordinator acts as an aggregator between the various different contexts to ensure the integrity of the persistent store(s). It does this by serialising read/write operations from the contexts to its store(s). There is only one coordinator per Core Data stack.
• Managed Object Model is a set of entities that define each NSManagedObject subclass. An entity can be thought of as a table in a database.
• Persistent Object Store is an abstraction over the actual storage of our data. Handles communication with that storage e.g. with SQLite storage, converts fetch requests into SQL statements.
• Storage it's common for this to be a SQLite store but the store could also be: XML, binary and in-memory.

Unit Testing

Unit Testing is ensuring that the smallest part (unit) of testable code in your app behaves as expected in isolation (from the other parts of your app). In object-oriented programming, a unit is often a particular method, with the unit test testing one scenario (path) through that method. A unit test does this by providing a strict, written contract that the unit under test must satisfy in order to pass. If there are multiple paths through a unit i.e. an if and else branch, then more than one unit test would be required to cover each path.

Unit tests are then combined into a test suite within a test target/project, this suite can then be run to give an increased level of confidence in that the app classes are valid.

Each unit test should be executed in isolation from other unit tests to ensure that the failure of a previous test has no impact upon the next test. It is up to the unit test to ensure that any conditions (test data, user permissions, etc) that it depends on are present before the test is run and it has to tidy up after itself when it is finished. This helps to ensure that the unit test is repeatable and not dependent upon any state on the host environment. The unit test is also responsible for ensuring that the unit under test is isolated from other methods within the app and that any calls (relationship) it makes for information with other methods are mocked out. A mocked method will then return a known, preset value without performing any computation so that if a unit test fails we can have confidence that it has failed because of the code under test rather than having to hunt down the failure in its dependencies.

Each unit test should be as quick as possible to run to ensure that during development, the feedback loop between running the unit test and making code changes is as small as possible.

Building that friendship

From the above descriptions, we can see why Core Data and Unit Testing didn't instantly hit it off. Their differences centre on two issues:

Treatment of data

• Unit Testing treats data as ephemeral
• The main use case of Core Data is persisting data between app executions

Tolerance for delays

• Unit tests should be lightning quick to execute
• Core Data typically has to communicate with a SQLite file on disk which is slow (when compared with pure memory operations)

And like building any good relationship, all the changes will come from one side 😜 - Core Data.

(Ok ok, I'm sketching a little bit now so please don't take that as genuine relationship advice)

CoreDataManager

Let's build a typical Core Data stack together and unit test it as we go.

If you want to follow along, head over to my repo and download the completed project.

Let's start with the manager that will handle setting up our Core Data stack:

class CoreDataManager {

    // MARK: - Singleton

    static let shared = CoreDataManager()
}

We could add a unit test here and assert that the same instance of CoreDataManager was always returned when shared was called however when unit testing we should only test code that we control and the logic behind creating a singleton is handled by Swift itself - so no need to create that test class yet.

Unit tests while asserting the correctness of an implementation also act as a living form of documentation so if you did want to add a unit test to assert the same instance was returned, I won't put up too much of a fight.

As our project is being developed with an iOS deployment target of iOS 11 we can use the persistent container to simplify the Core Data stack's setup.

lazy var persistentContainer: NSPersistentContainer! = {
    let persistentContainer = NSPersistentContainer(name: "TestingWithCoreData_Example")

    return persistentContainer
}()

In the above code snippet, we added a lazy property to create an instance of NSPersistentContainer. Loading a Core Data stack can be a time-consuming task especially if migration is required. To handle this we need to add a dedicated asynchronous setup method that can handle any time-consuming tasks.

While the below method doesn't actually show the migration itself, I think it's important to create and test a realistic Core Data stack and data migrations are very much a common task in iOS apps.

// MARK: - SetUp

 func setup(completion: (() -> Void)?) {
     loadPersistentStore {
         completion?()
     }
 }

 // MARK: - Loading

 private func loadPersistentStore(completion: @escaping () -> Void) {
     //handle data migration on a different thread/queue here
     persistentContainer.loadPersistentStores { description, error in
         guard error == nil else {
             fatalError("was unable to load store \(error!)")
         }

         completion()
     }
 }

Ok so now we have something to test - does calling setup actually set up our stack. Let's create that unit test class:

class CoreDataManagerTests: XCTestCase {

    // MARK: Properties

    var sut: CoreDataManager!

    // MARK: - Lifecycle

    override func setUp() {
        super.setUp()

        sut = CoreDataManager()
    }

    // MARK: - Tests

    // MARK: Setup

    func test_setup_completionCalled() {
        let setupExpectation = expectation(description: "set up completion called")

        sut.setup() {
            setupExpectation.fulfill()
        }

        waitForExpectations(timeout: 1.0, handler: nil)
    }
}

In the above class, we declare a property sut (subject under test) which will hold a CoreDataManager instance that we will be testing. I prefer to use sut as it makes it immediately obvious which object we are testing and which objects are collaborators/dependencies. It's important to note that the sut property is an implicitly unwrapped optional - it's purely to make our unit tests more readable by avoiding having to handle it's optional nature elsewhere and is a technique that I would not recommend using too widely in production code. The test suite's setUp method is where the CoreDataManager instance is being created and assigned to sut.

Let's take a closer look at the unit test itself:

func test_setup_completionCalled() {
    let setupExpectation = expectation(description: "set up completion called")

    sut.setup() {
        setupExpectation.fulfill()
    }

    waitForExpectations(timeout: 1.0, handler: nil)
}

When it comes to naming I follow the naming convention:

test_[unit under test]_[condition]_[expected outcome]

[condition] is optional.

So the test method signature tells us that we are testing the setup method and that the completion closure should be triggered.

Now with this test, we are really jumping straight into the deeper end of unit testing by testing an asynchronous method but as we can see the code isn't actually that difficult to understand. The first thing we do is create an XCTestExpectation instance, it's important to note here that we are not directly creating an XCTestExpectation instance using XCTestExpectation's init method instead we are using the convenience method provided by XCTestCase. By creating it via XCTestCase we will tie both the XCTestExpectation and XCTestCase together which will allow us to use waitForExpectations and cut down on some of the boilerplate required with expectations. If you have never used expectations before, you can think of them as promising that an action will happen within a certain time frame. Sadly like actual promises, they can be broken and when they are - the test fails.

As I'm sure you have noted, test_setup_completionCalled doesn't actually contain any asserts, this is because we are using the expectation as an implicit assert.

So we've tested that the completion closure is called but we haven't actually checked that anything was set up. A successful set up should result in our persistent store being loaded so let's add a test to check that:

func test_setup_persistentStoreCreated() {
   let setupExpectation = expectation(description: "set up completion called")

    sut.setup() {
        setupExpectation.fulfill()
    }

    waitForExpectations(timeout: 1.0) { (_) in
        XCTAssertTrue(self.sut.persistentContainer.persistentStoreCoordinator.persistentStores.count > 0)
    }
}

As we can see test_setup_persistentStoreCreated contains an assert to check that the persistentStoreCoordinator has at least one persistentStore. It's important to note that as persistentContainer is lazy loaded merely checking that it's not nil wouldn't be a valid test as calling the property would result in creating the persistentContainer.

The two unit tests that we have added are very similar and as you can see it's possible for test_setup_persistentStoreCreated to fail for two reasons:

1. Completion closure not triggered
2. Persistent store not created

The first reason is actually being tested in test_setup_completionCalled so why have I created another test that's dependent on it here? The reason is that it's actually impossible not to check this condition as it's an implicit dependency on any test that uses this method. Now the argument could be made that these two tests should be one - effectively a test_setup_stackCreated test. I opted for two tests as I felt that it improved the readability in the event that one of those tests failed by providing a higher level of granularity for that happening. You sometimes hear people saying that a unit test should only ever have one assert and that any unit test that has more than one assert is wrong. IMHO, this is foolhardy. There are very few hard and fast rules in life, just about everything is context based - in this context having two asserts (one implicit, one explicit) in test_setup_persistentStoreCreated makes sense as both asserts are checking that the same unit of functionality is correct.

Now, the more eagled eyed 👀 among you will have spotted that we are using the default storage type for our Core Data stack (NSSQLiteStoreType) in the above tests - this creates a SQLite file on the disk and as we know any I/O operation is going to be much slower than a pure in-memory operation. It would be great if we could tell the Core Data stack which storage type to use - NSSQLiteStoreType for production and NSInMemoryStoreType for testing:

class CoreDataManager {

    private var storeType: String!

    lazy var persistentContainer: NSPersistentContainer! = {
        let persistentContainer = NSPersistentContainer(name: "TestingWithCoreData_Example")
        let description = persistentContainer.persistentStoreDescriptions.first
        description?.type = storeType

        return persistentContainer
    }()

    // MARK: - Singleton

    static let shared = CoreDataManager()

    // MARK: - SetUp

    func setup(storeType: String = NSSQLiteStoreType, completion: (() -> Void)?) {
        self.storeType = storeType

        loadPersistentStore {
            completion?()
        }
    }

    // MARK: - Loading

    private func loadPersistentStore(completion: @escaping () -> Void) {
        persistentContainer.loadPersistentStores { description, error in
            guard error == nil else {
                fatalError("was unable to load store \(error!)")
            }

            completion()
        }
    }
}

As a long-term Objective-C iOS developer, I still get a thrill about being able to provide a default value to a parameter (like we do with the storeType parameter above) without having to create a whole new method. This change will allow us to use the much quicker NSInMemoryStoreType storage type in our tests while keeping a simple interface in production. However, it's not free and because we have introduced a new way of setting up our Core Data stack we need to update our existing tests to test this new path:

class CoreDataManagerTests: XCTestCase {

    // MARK: Properties

    var sut: CoreDataManager!

    // MARK: - Lifecycle

    override func setUp() {
        super.setUp()

        sut = CoreDataManager()
    }

    // MARK: - Tests

    // MARK: Setup

    func test_setup_completionCalled() {
        let setupExpectation = expectation(description: "set up completion called")

        sut.setup(storeType: NSInMemoryStoreType) {
            setupExpectation.fulfill()
        }

        waitForExpectations(timeout: 1.0, handler: nil)
    }

    func test_setup_persistentStoreCreated() {
       let setupExpectation = expectation(description: "set up completion called")

        sut.setup(storeType: NSInMemoryStoreType) {
            setupExpectation.fulfill()
        }

        waitForExpectations(timeout: 1.0) { (_) in
            XCTAssertTrue(self.sut.persistentContainer.persistentStoreCoordinator.persistentStores.count > 0)
        }
    }

    func test_setup_persistentContainerLoadedOnDisk() {
        let setupExpectation = expectation(description: "set up completion called")
        
        sut.setup {
            XCTAssertEqual(self.sut.persistentContainer.persistentStoreDescriptions.first?.type, NSSQLiteStoreType)
            setupExpectation.fulfill()
        }
        
        waitForExpectations(timeout: 1.0) { (_) in
            self.sut.persistentContainer.destroyPersistentStore()
        }
    }

    func test_setup_persistentContainerLoadedInMemory() {
        let setupExpectation = expectation(description: "set up completion called")

        sut.setup(storeType: NSInMemoryStoreType) {
            XCTAssertEqual(self.sut.persistentContainer.persistentStoreDescriptions.first?.type, NSInMemoryStoreType)
            setupExpectation.fulfill()
        }

        waitForExpectations(timeout: 1.0, handler: nil)
    }
}

If we run the above tests we are able to see the difference in running times between test_setup_persistentContainerLoadedInMemory and test_setup_persistentContainerLoadedOnDisk:

As you can see on the above execution of both tests, loading the store on disk took 17 times longer than loading the store into memory - 0.001 vs 0.017 seconds.

In real terms, this speed increase isn't much on its own but once we start adding in tests that create, update and delete NSManagedObject instances dealing with an in-memory store will allow these tests to be executed faster than if we used an on-disk store.

So far, we have made great progress on producing a Core Data stack that is unit testable but a Core Data stack without a context (or two) isn't going to be very useful - let's add some:

lazy var backgroundContext: NSManagedObjectContext = {
    let context = self.persistentContainer.newBackgroundContext()
    context.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy

    return context
}()

lazy var mainContext: NSManagedObjectContext = {
    let context = self.persistentContainer.viewContext
    context.automaticallyMergesChangesFromParent = true

    return context
}()

And we need to add some unit tests for them:

func test_backgroundContext_concurrencyType() {
    let setupExpectation = expectation(description: "background context")

    sut.setup(storeType: NSInMemoryStoreType) {
        XCTAssertEqual(self.sut.backgroundContext.concurrencyType, .privateQueueConcurrencyType)
        setupExpectation.fulfill()
    }

    waitForExpectations(timeout: 1.0, handler: nil)
}

func test_mainContext_concurrencyType() {
    let setupExpectation = expectation(description: "main context")

    sut.setup(storeType: NSInMemoryStoreType) {
        XCTAssertEqual(self.sut.mainContext.concurrencyType, .mainQueueConcurrencyType)
        setupExpectation.fulfill()
    }

    waitForExpectations(timeout: 1.0, handler: nil)
}

Good news is that we have finished creating and testing our Core Data stack and I think it wasn't actually too difficult 🎉.

Introducing ColorsDataManager

There is no point in creating a Core Data stack if we don't actually use it. In the example project we populate a collectionview with instances of a subclass of NSManagedObject - Color. To help us deal with these Color objects we will be using a ColorsDataManager:

class ColorsDataManager {

    let backgroundContext: NSManagedObjectContext

    // MARK: - Init

    init(backgroundContext: NSManagedObjectContext = CoreDataManager.shared.backgroundContext) {
        self.backgroundContext = backgroundContext
    }

    // MARK: - Create

    func createColor() {
        backgroundContext.performAndWait {
            let color = NSEntityDescription.insertNewObject(forEntityName: Color.className, into: backgroundContext) as! Color
            color.hex = UIColor.random.hexString
            color.dateCreated = Date()

            try? backgroundContext.save()
        }
    }
    
    // MARK: - Deletion
    
    func deleteColor(color: Color) {
        let objectID = color.objectID
        backgroundContext.performAndWait {
            if let colorInContext = try? backgroundContext.existingObject(with: objectID) {
                backgroundContext.delete(colorInContext)
                
                try? backgroundContext.save()
            }
        }
    }
}

In the above class, we have a simple manager that handles creating and deleting Color instances. As a responsible member of the Core Data community, our example app treats the backgroundContext as a read-write context and the mainContext as a read-only context - this is to ensure that any time-consuming tasks don't block the main (UI) thread. You may have noticed that the above class doesn't actually contain any mention of CoreDataManager instead this class only knows about the background context. By injecting this context into the class, we are able to decouple ColorsDataManager from CoreDataManager which should allow us to more easily test ColorsDataManager 😉.

Let's look at implementing our first test for ColorsDataManager:

class ColorsDataManagerTests: XCTestCase {

    // MARK: Properties

    var sut: ColorsDataManager!

    var coreDataStack: CoreDataTestStack!

    // MARK: - Lifecycle

    override func setUp() {
        super.setUp()

        coreDataStack = CoreDataTestStack()

        sut = ColorsDataManager(backgroundContext: coreDataStack.backgroundContext)
    }

    // MARK: - Tests

    // MARK: Init

    func test_init_contexts() {
        XCTAssertEqual(sut.backgroundContext, coreDataStack.backgroundContext)
    }
}

As we can see, the majority of the above class is taken up with setting up the test suite. The test itself, merely checks that the context that we pass into ColorsDataManager is the same context that is assigned to the backgroundContext property.

You may also have noticed that coreDataStack isn't a CoreDataManager instance but instead a CoreDataTestStack instance.

Let's go on a slight detour and have a look at CoreDataTestStack:

class CoreDataTestStack {

    let persistentContainer: NSPersistentContainer
    let backgroundContext: NSManagedObjectContextSpy
    let mainContext: NSManagedObjectContextSpy

    init() {
        persistentContainer = NSPersistentContainer(name: "TestingWithCoreData_Example")
        let description = persistentContainer.persistentStoreDescriptions.first
        description?.type = NSInMemoryStoreType

        persistentContainer.loadPersistentStores { description, error in
            guard error == nil else {
                fatalError("was unable to load store \(error!)")
            }
        }

        mainContext = NSManagedObjectContextSpy(concurrencyType: .mainQueueConcurrencyType)
        mainContext.automaticallyMergesChangesFromParent = true
        mainContext.persistentStoreCoordinator = persistentContainer.persistentStoreCoordinator

        backgroundContext = NSManagedObjectContextSpy(concurrencyType: .privateQueueConcurrencyType)
        backgroundContext.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy
        backgroundContext.parent = self.mainContext
    }
}

CoreDataTestStack is very similar to CoreDataManager but with its asynchronous setup behaviour stripped out and the storeType always set to NSInMemoryStoreType. This class allows us to more easily set up and tear down the stack between tests without having to wait on any asynchronous tasks to complete. Another difference between CoreDataTestStack and CoreDataManager is that the two contexts that are being created are not in fact standard NSManagedObjectContext instances but are actually NSManagedObjectContextSpy instances.

If you are curious as to what a spy is, Martin Fowler has produced a very insightful article on naming test objects - it's in the section titled: The Difference Between Mocks and Stubs.

NSManagedObjectContextSpy is a subclass of NSManagedObjectContext that adds special state tracking properties. System classes occupy a grey area when it comes to if you should use them in your tests or if you need to replace them with mock/stub instances. In this case, I felt that mocking out a context's functionality would be too much work and would actually be counter-productive to what the test is attempting to achieve so I'm perfectly happy to use it directly.

class NSManagedObjectContextSpy: NSManagedObjectContext {
    var expectation: XCTestExpectation?

    var saveWasCalled = false

    // MARK: - Perform

    override func performAndWait(_ block: () -> Void) {
        super.performAndWait(block)

        expectation?.fulfill()
    }

    // MARK: - Save

    override func save() throws {
        save()

        saveWasCalled = true
    }
}

Ok, detour over. Let's get back to testing the ColorsDataManager class:

func test_createColor_colorCreated() {
    let performAndWaitExpectation = expectation(description: "background perform and wait")
    coreDataStack.backgroundContext.expectation = performAndWaitExpectation

    sut.createColor()

    waitForExpectations(timeout: 1) { (_) in
        let request = NSFetchRequest.init(entityName: Color.className)
        let colors = try! self.coreDataStack.backgroundContext.fetch(request)

        guard let color = colors.first else {
            XCTFail("color missing")
            return
        }

        XCTAssertEqual(colors.count, 1)
        XCTAssertNotNil(color.hex)
        XCTAssertEqual(color.dateCreated?.timeIntervalSinceNow ?? 0, Date().timeIntervalSinceNow, accuracy: 0.1)
        XCTAssertTrue(self.coreDataStack.backgroundContext.saveWasCalled)
    }
}

There is a bit more happening here than in the previous test that we saw. We create an XCTestExpectation instance that we then assign to the expectation property on the context. As we have seen above this expectation should be fulfilled when performAndWait is called. Once that expectation has been triggered, we then check that the Color instance was created and saved into our persistent store.

Testing the deletion of a Color follows a similar pattern:

func test_deleteColor_colorDeleted() {
    let performAndWaitExpectation = expectation(description: "background perform and wait")
    coreDataStack.backgroundContext.expectation = performAndWaitExpectation
    
    let colorA = NSEntityDescription.insertNewObject(forEntityName: Color.className, into: self.coreDataStack.backgroundContext) as! Color
    let colorB = NSEntityDescription.insertNewObject(forEntityName: Color.className, into: self.coreDataStack.backgroundContext) as! Color
    let colorC = NSEntityDescription.insertNewObject(forEntityName: Color.className, into: self.coreDataStack.backgroundContext) as! Color
    
    sut.deleteColor(color: colorB)
    
    waitForExpectations(timeout: 1) { (_) in
        let request = NSFetchRequest.init(entityName: Color.className)
        let backgroundContextColors = try! self.coreDataStack.backgroundContext.fetch(request)
        
        XCTAssertEqual(backgroundContextColors.count, 2)
        XCTAssertTrue(backgroundContextColors.contains(colorA))
        XCTAssertTrue(backgroundContextColors.contains(colorC))
        XCTAssertTrue(self.coreDataStack.backgroundContext.saveWasCalled)
    }
}

We first populate our persistent (in-memory) store, call the deleteColor method and then check that the correct Color has been deleted. There is one special case - because we read on the main context and delete on the background context, the color instance passed into this method may be from the main context, the above test is not covering this case so let's add another test that does:

func test_deleteColor_switchingContexts_colorDeleted() {
    let performAndWaitExpectation = expectation(description: "background perform and wait")
    coreDataStack.backgroundContext.expectation = performAndWaitExpectation
    
    let colorA = NSEntityDescription.insertNewObject(forEntityName: Color.className, into: self.coreDataStack.backgroundContext) as! Color
    let colorB = NSEntityDescription.insertNewObject(forEntityName: Color.className, into: self.coreDataStack.backgroundContext) as! Color
    let colorC = NSEntityDescription.insertNewObject(forEntityName: Color.className, into: self.coreDataStack.backgroundContext) as! Color
    
    let mainContextColor = coreDataStack.mainContext.object(with: colorB.objectID) as! Color
    
    sut.deleteColor(color: mainContextColor)
    
    waitForExpectations(timeout: 1) { (_) in
        let request = NSFetchRequest.init(entityName: Color.className)
        let backgroundContextColors = try! self.coreDataStack.backgroundContext.fetch(request)
        
        XCTAssertEqual(backgroundContextColors.count, 2)
        XCTAssertTrue(backgroundContextColors.contains(colorA))
        XCTAssertTrue(backgroundContextColors.contains(colorC))
        XCTAssertTrue(self.coreDataStack.backgroundContext.saveWasCalled)
    }
}

Pretty much the same as before with the only difference being that we retrieve the color to be deleted from the main context before passing that in.

An interesting point to note when adding those three tests is that we haven't had to add any code to clear our persistent store. This is because by using an NSInMemoryStoreType store and ensuring that we create a new stack before each test - we never actually persist data. Not only does this save us time having to write the tidy up code, it also removes a whole category of bugs where leftover state from one test affects the outcome of another due to faulty/missing clean up code.

To come back to the point about in-memory stores being quicker to use than on-disk stores, we can see a typical difference in running times for the above tests below:

In-memory store

SQLite store

Best Friends Forever?

In the above code snippets, we have seen that unit testing with Core Data doesn't need to be that much more difficult than unit testing in general, with most of that added difficulty coming in the set up of the Core Data stack. And while Core Data and Unit Testing may not become BFFs (let's be honest UIKit has that position sealed down), we've seen that they can become firm friends 👫. That friendship is built on small alterations to our Core Data stack which allows its data to be more easily thrown away and the use of special subclasses to allow us to better track state.

You can download the example project for this post here.

Once upon a time, an app was just a stripped-down version of a website. Those days are long gone. Our users now expect to be able to do everything through the app that they can do through the website. This change in expectation has meant that our apps have become increasingly media hungry. Despite network speeds increasing year-on-year, network requests are still the most likely source of bottlenecks, especially media requests. Every time a user has to wait for a network request to complete before they can get on with their task, we risk losing that user.

A common approach to try and alleviate these bottlenecks is to cancel network requests as soon as possible. While this approach is valid in some scenarios, I have too often seen it used naively for all network request. When we cancel a network request, we throw away any progress that request has made in downloading the requested asset. If the app then goes on to request that asset again, the download starts at 0% and then spend time redownloading data it previously had.

In this post, I want to look at how we can build a better downloading approach that doesn't throw away data and merges duplicate download requests. All without requiring any extra effort from the consumer of the media layer.

This post will build up to a working example however if you're just too excited to wait for that, then head on over to the completed example and take a look at AssetDownloadsSession, AssetDownloadItem and AssetDownloadItemDelegate to see how things end up. To run the example project, follow the instructions below.

Time for a small detour 🗺️

Just because we want to resume a cancelled download does not mean that we can. For resumption to be possible, we need the following to be true:

• The resource has not changed since it was first requested.
• The task is an HTTP or HTTPS GET request.
• The server provides either the ETag or Last-Modified header (or both) in its response.
• The server supports byte-range requests.

If all of the above is true then congratulations you are ready for the rest of this article; if the above isn't true, then you have some work to do before you can implement the below solution.

Now, before we start building our downloading layer, let's look (briefly) at how iOS handles supporting resuming downloads via URLSession. To resume a download, we need to use a special init'er on URLSessionDownloadTask - downloadTask(withResumeData:completionHandler:) (this example is only going to show the completion handler approach if you want to see a delegation based approach Alexander Grebenyuk has a great article on that approach here). Rather than taking a URL or URLRequest, downloadTask(withResumeData:completionHandler:) takes a Data instance. This Data instance, among other things, tells the URLSessionDownloadTask instance where the already downloaded media data should be on the file system and what parts of the media asset have already been downloaded. We get this Data instance by calling cancel(byProducingResumeData:) when cancelling the original URLSessionDownloadTask instance.

If the above introduction to resuming downloads feels too short, don't fret we will cover the topic in a lot more detail as we build up the media download layer below.

Let's get building 👷

Our media download layer has 5 primary responsibilities:

1. Scheduling downloads.
2. Pausing downloads.
3. Resuming paused downloads.
4. Removing unwanted paused downloads (freeing memory).
5. Downloading the requested media.

These responsibilities together produce the following class structure:

• AssetDataManager is the manager for accessing assets, e.g. downloading an asset or locally retrieving a cached asset.
• AssetDownloadsSession is the controller for scheduling, pausing, resuming, cancelling and deleting download requests.
• AssetDownloadItem is a wrapper around an URLSessionDownloadTask instance - adding easy-to-use coalescing and pausing functionality around it.
• AssetDownloadItemDelegate is a protocol to allow an AssetDownloadItem instance to communicate with the AssetDownloadsSession instance.

Both AssetDownloadItem and AssetDownloadItemDelegate are private and only visible to AssetDownloadsSession.

We won't see the implementation of AssetDataManager as it is the consumer of pausable downloads rather than a part of it. I included it in the above diagram to show how the media download layer can be used.

Before we start adding in the ability to download an asset, let's look at the possible lifecycle states an AssetDownloadItem instance can be in:

1. Ready indicates that our AssetDownloadItem instance hasn't started downloading yet.
2. Downloading indicates that our AssetDownloadItem instance is currently downloading.
3. Paused indicates that our AssetDownloadItem instance isn't actively downloading but has in the past and has kept the previously downloaded data.
4. Cancelled indicates that our AssetDownloadItem instance isn't actively downloading and has thrown away any previously downloaded data.
5. Completed indicates that our AssetDownloadItem instance download has finished (either successfully or due to a failure) and the work of this AssetDownloadItem instance is over.

The lifecycle of an AssetDownloadItem instance looks like:

So a download that starts and completes without pausing or cancelling would look like:

Ready -> Downloading -> Completed

Whereas another download that has been paused would look more like:

Ready -> Downloading -> Paused -> Downloading -> Completed

These states are best represented as an enum:

fileprivate enum State: String {
    case ready
    case downloading
    case paused
    case cancelled
    case completed
}

Now that we have the possible lifecycle states represented, let's add in the basic structure of AssetDownloadItem to move between those states:

fileprivate class AssetDownloadItem {

    //Omitted other properties

    private(set) var state: State = .ready

    //Omitted other methods

    func resume() {
        state = .downloading
    }

    func cancel() {
        state = .cancelled
    }

    func pause() {
        state = .paused
    }

    private func complete() {
        state = completed
    }
}

These methods don't do much yet - we will gradually fill them out as we build up the example.

Now that we have the lifecyle in, let's add in the ability to download an asset:

typealias DownloadCompletionHandler = ((_ result: Result<Data, Error&rt;) -> ())

fileprivate class AssetDownloadItem { // 1
    //Omitted other properties

    private let session: URLSession
    private var downloadTask: URLSessionDownloadTask?

    let url: URL
    var completionHandler: DownloadCompletionHandler? // 2

    // MARK: - Init

    init(session: URLSession, url: URL) {
        self.session = session
        self.url = url
    }

    // MARK: - Lifecycle

    // 3
    func resume() {
        state = .downloading

        os_log(.info, "Creating a new download task")
        downloadTask = session.downloadTask(with: url, completionHandler: downloadTaskCompletionHandler)

        downloadTask?.resume()
    }

    // 4
    private func downloadTaskCompletionHandler(_ fileLocationURL: URL?, _ response: URLResponse?, _ error: Error?) {
        var result: Result
        defer {
            downloadCompletionHandler?(result)

            if state != .paused {
                complete()
            }

            cleanup()
        }

        guard let fileLocationURL = fileLocationURL else {
            result = .failure(NetworkingError.retrieval(underlayingError: error))
            return
        }

        do {
            let data = try Data(contentsOf: fileLocationURL)
            result = .success(data)
        } catch let error {
            result = .failure(NetworkingError.invalidData(underlayingError: error))
        }
    }

    // 5
    private func cleanup() {
        downloadTask = nil
        completionHandler = nil
    }

    //Omitted other methods
}

If you have cloned the example project, you will notice that I make extensive use of protocols that are not shown above. The protocols are used to allow me to better unit test this solution; I've omitted them from this post to make it more readable.

Here’s what we did:

1. AssetDownloadItem is a simple wrapper around creating and resuming an URLSessionDownloadTask instance.
2. The download completion closure that will be used when the download completes.
3. Mirroring URLSessionDownloadTask, AssetDownloadItem has its own resume() method that when called causes the download to start.
4. The completion handler for the download task. Depending on how the download progresses, the completionHandler is triggered either with the requested asset as a Data instance or an error detailing what happened. Only non-paused downloads are considered complete when the closure is called.
5. Regardless of how a download is completed, we need to clean up this AssetDownloadItem instance by setting its downloadTask and completionHandler to nil so that they can't accidentally be reused.

The possible errors that can be returned from download request are:

enum NetworkingError: Error {
    case unknown
    case retrieval(underlayingError: Error?)
    case invalidData(underlayingError: Error?)
}

Now that we can start a download let's look at how we can cancel a download. URLSessionDownloadTask has two methods for cancellation:

1. cancel() - download is stopped and any data downloaded so far is discarded.
2. cancel(byProducingResumeData: @escaping (Data?) -> Void) - download is stopped and any data downloaded so far is stored in a temporary filesystem location. Details of the partially completed download are passed back as a Data instance. It's important to note that the Data instance returned in the completion closure of the cancel method, is not the data that has been downloaded so far but is instead data that will allow the download to be resumed from its partial downloaded state.

In the context of strengthening our media download system, we can think of these cancel methods, respectively as:

1. Cancel (cancel())
2. Pause (cancel(byProducingResumeData:))

Lets add the ability to cancel a download:

fileprivate class AssetDownloadItem {

    //Omitted properties and other methods

    // 3
    func cancel() {
        state = .cancelled

        os_log(.info, "Cancelling download")
        downloadTask?.cancel()

        cleanup()
    }
}

cancel() forwards the cancel instruction onto the wrapped URLSessionDownloadTask instance.

Now let's add the ability to pause a download:

fileprivate class AssetDownloadItem {

    //Omitted other properties

    // 1
    private var resumptionData: Data?

    //Omitted other methods

    // 2
    func pause() {
        state == .paused

        os_log(.info, "Pausing download")
        downloadTask?.cancel(byProducingResumeData: { [weak self] (data) in
            guard let data = data else {
                return
            }

            os_log(.info, "Cancelled download task has produced resumption data of: %{public}@ for %{public}@", data.description, self?.url.absoluteString ?? "unknown url")
            self?.resumptionData = data
        })

        cleanup()
    }
}

With the above changes here’s what we did:

1. Added a property to store any resumption Data instance.
2. pause() cancels the wrapped URLSessionDownloadTask instance and sets resumptionData with the Data instance returned in the closure.

You may be thinking "Why not just call suspend() on the download?". It's a good idea however calling suspend() on an active download doesn't actually stop that download (even though the URLSessionDownloadTask instance will report that it's stopped downloading). You can see this in action if you use Charles Proxy to snoop on a supposedly suspended download.

Now that we have some resumption data, let's use it by refactoring our resume() method:

fileprivate class AssetDownloadItem {

    //Omitted properties

    func resume() {
        //Omitted start of method

        if let resumptionData = resumptionData {
            os_log(.info, "Attempting to resume download task")
            downloadTask = session.downloadTask(withResumeData: resumptionData, completionHandler: downloadTaskCompletionHandler)
        } else {
            os_log(.info, "Creating a new download task")
            downloadTask = session.downloadTask(with: url, completionHandler: downloadTaskCompletionHandler)
        }

        downloadTask?.resume()
    }

    //Omitted other methods
}

With the above changes, we now have two ways to create a URLSessionDownloadTask instance: with and without resumption data.

It's not uncommon for the same asset to be requested for download multiple times. Making multiple requests for the same asset is wasteful. While caching assets is outside of the scope of the media download layer, coalescing (or merging) active download requests for the same asset isn't:

fileprivate class AssetDownloadItem {

    //Omitted properties and other methods

    //MARK: - Coalesce

    func coalesceDownloadCompletionHandler(_ otherDownloadCompletionHandler: @escaping DownloadCompletionHandler) {
        let initalDownloadCompletionHandler = downloadCompletionHandler

        downloadCompletionHandler = { result in
            initalDownloadCompletionHandler?(result)
            otherDownloadCompletionHandler(result)
        }
    }
}

In coalesceDownloadCompletionHandler(_:) the existing completion handler (initalDownloadCompletionHandler) and the completion handler (otherDownloadCompletionHandler) for the new download are wrapped together in a third completion handler (downloadCompletionHandler). This third completion handler is then set as this AssetDownloadItem instance's downloadCompletionHandler value. This technique means that when this download completes the completion handler for both download requests will be triggered.

It is possible to recursively wrap any number of DownloadCompletionHandler closures using this approach.

Lets add a few connivence properties for interpreting the state property:

fileprivate class AssetDownloadItem {
    //Omitted other properties

    // 1
    var isCoalescable: Bool {
        return (state == .ready) ||
            (state == .downloading) ||
            (state == .paused)
    }

    // 2
    var isResumable: Bool {
        return (state == . ready) ||
            (state == .paused)
    }

    // 3
    var isPaused: Bool {
        return state == .paused
    }

    // 4
    var isCompleted: Bool {
        return state == .completed
    }

    //Omitted methods
}

1. We only want to be able to coalesce ready, downloading or paused.
2. We can only resume AssetDownloadItem instances that are not currently downloading or have been completed.
3. Wrapper around a check for if the state is paused so that it reads better.
4. Wrapper around a check for if the state is completed so that it reads better.

We will see how these are used in AssetDownloadsSession.

Before we leave AssetDownloadItem lets add a description property to aid our debugging:

fileprivate class AssetDownloadItem {
    //Omitted other properties

    var description: String {
        return url.absoluteString
    }

    //Omitted other methods
}

Our AssetDownloadItem instances description will now show the URL of the asset that it is downloading.

So far we have been building up AssetDownloadItem and while we are not yet done with, AssetDownloadItem now has enough functionality to allow us to turn our attention to AssetDownloadsSession.

As mentioned above AssetDownloadsSession has 4 tasks with regards to download requests:

• Scheduling.
• Pausing.
• Resuming.
• Cancelling.

However, not all 4 need to be exposed. Only scheduling and cancelling need to be public. Resuming and pausing can private. Resuming a download is just a special case of scheduling and pausing is just a special case of cancellation. By hiding the ability to resume and pause a download, we can keep the interface of AssetDownloadsSession minimal.

First, lets look at how we can schedule a download:

class AssetDownloadsSession {

    // 1
    static let shared = AssetDownloadsSession()

    // 2
    private var assetDownloadItems = [AssetDownloadItem]()

    private var session: URLSession

    // MARK: - Init

    // 3
    init(urlSessionFactory: URLSessionFactory = URLSessionFactory()) {
        self.session = urlSessionFactory.defaultSession()
    }

    // MARK: - Schedule

    // 4
    func scheduleDownload(url: URL, completionHandler: @escaping DownloadCompletionHandler) {
        let assetDownloadItem = AssetDownloadItem(session: session, url: url)
        assetDownloadItem.downloadCompletionHandler = downloadCompletionHandler

        os_log(.info, "Adding new download: %{public}@", assetDownloadItem.description)

        assetDownloadItems.append(assetDownloadItem)

        assetDownloadItem.resume()
    }
}

Here’s what we did:

1. AssetDownloadsSession is a singleton as we want all asset downloads to go through the same component which will allow any duplicate download requests to be spotted and coalesced.
2. An array of the AssetDownloadItem instances that are either downloading, paused or cancelled.
3. URLSessionFactory is a factory that handles the creation of URLSession instances.
4. In scheduleDownload(url:completionHandler:) a new AssetDownloadItem instance is created, added to the assetDownloadItems array and the download is started.

URLSessionFactory looks like:

class URLSessionFactory {

    // MARK: - Default

    func defaultSession(delegate: URLSessionDelegate? = nil, delegateQueue queue: OperationQueue? = nil) -> URLSession {
        let configuration = URLSessionConfiguration.default

        //For demonstration purposes disable caching
        configuration.requestCachePolicy = .reloadIgnoringLocalCacheData
        configuration.urlCache = nil

        let session = URLSession(configuration: configuration, delegate: delegate, delegateQueue: queue)

        return session
    }
}

Now that we can schedule a download, let's look at how to pause a download.

As mentioned above, outside of AssetDownloadsSession there is no concept of pausing a download so our pause method will be presented as a cancel method.

class AssetDownloadsSession {
    //Omitted properties and other methods

    // MARK: - Cancel

    func cancelDownload(url: URL) {
        guard let assetDownloadItem = assetDownloadItems.first(where: { $0.url == url }) else {
            return
        }

        os_log(.info, "Download: %{public}@ going to be paused", assetDownloadItem.description)
        assetDownloadItem.pause()
    }
}

In the above method, we first determine whether an existing AssetDownloadItem instance exists for the URL passed in and if it does pause() is called on it.

The only time that we really want to be actually cancelling downloads is when the system is under strain, and we need to free up memory. When this happens iOS posts a UIApplication.didReceiveMemoryWarningNotification notification that we can listen for:

class AssetDownloadsSession {
    //Omitted properties

    // MARK: - Init

    // 1
    init(urlSessionFactory: URLSessionFactory = URLSessionFactory(), notificationCenter: NotificationCenter = NotificationCenter.default) {
        self.session = urlSessionFactory.defaultSession(delegate: self)
        registerForNotifications(on: notificationCenter)
    }

    // MARK: - Notification

    // 2
    private func registerForNotifications(on notificationCenter: NotificationCenter) {
        notificationCenter.addObserver(forName: UIApplication.didReceiveMemoryWarningNotification, object: nil, queue: .main) { [weak self] _ in
            self?.purgePausedDownloads()
        }
    }

    // 3
    private func purgePausedDownloads() {
        accessQueue.sync {
           os_log(.info, "Cancelling paused items")

           assetDownloadItems = assetDownloadItems.filter { (assetDownloadItem) -> Bool in
               let isPaused = assetDownloadItem.isPaused
               if isPaused {
                   assetDownloadItem.cancel()
               }

               return !isPaused
           }
        }
    }

    //Omitted other methods
}

With the above changes here’s what we did:

1. Inject the default NotificationCenter instance into the AssetDownloadsSession init'er.
2. Register for the UIApplication.didReceiveMemoryWarningNotification notification.
3. Loop through the AssetDownloadItem instances, find those that are paused and the cancel those instances. Finally, filter out those now-cancelled AssetDownloadItem instances to remove them from the assetDownloadItems array.

So far we are able to schedule, pause and cancel downloads so let's add in the ability to resume a download:

class AssetDownloadsSession {
    //Omitted properties and other methods

    func scheduleDownload(url: URL, completionHandler: @escaping DownloadCompletionHandler) {
        if let assetDownloadItem = assetDownloadItems.first(where: { $0.url == url && $0.isCoalescable }) {
            os_log(.info, "Found existing %{public}@ download so coalescing them for: %{public}@", assetDownloadItem.state.rawValue, assetDownloadItem.description)

            assetDownloadItem.coalesceDownloadCompletionHandler(completionHandler)

            if assetDownloadItem.isResumable {
                assetDownloadItem.resume()
            }
        } else {
            //Omitted
        }
    }
}

With the changes above when a URL is passed in, a check is made to see if there is an existing AssetDownloadItem instance with that URL that can be coalesced. If there is an existing AssetDownloadItem instance, then the new download request is coalesced with it. If that coalesced download was paused, it is resumed.

Lets add a delegate so that our AssetDownloadItem instances can inform AssetDownloadsSession that a download has been completed so that it can be removed from the assetDownloadItems array:

fileprivate protocol AssetDownloadItemDelegate {
    func assetDownloadItemCompleted(_ assetDownloadItem: AssetDownloadItem)
}

fileprivate class AssetDownloadItem: Equatable {
    //Omitted other properties

    var delegate: AssetDownloadItemDelegate?

    //Omitted other methods

    private func complete() {
        state = .completed

        delegate?.assetDownloadItemCompleted(self)
    }
}

AssetDownloadsSession needs to implement AssetDownloadItemDelegate so that the completed download can be removed:

class AssetDownloadsSession: AssetDownloadItemDelegate // 1 {
    //Omitted properties and other methods

    func scheduleDownload(url: URL, completionHandler: @escaping DownloadCompletionHandler) {
        if let existingCoalescableAssetDownloadItem = assetDownloadItems.first(where: { $0.url == url && $0.isCoalescable }) {
            //Omitted
        } else {
            let assetDownloadItem = AssetDownloadItem(session: session, url: url)
            assetDownloadItem.downloadCompletionHandler = completionHandler
            assetDownloadItem.delegate = self // 2

            os_log(.info, "Created a new download: %{public}@", assetDownloadItem.description)

            assetDownloadItems.append(assetDownloadItem)

            assetDownloadItem.resume()
        }
    }

    // MARK: - AssetDownloadItemDelegate

    // 3
    func assetDownloadItemCompleted(_ assetDownloadItem: AssetDownloadItem) {
        os_log(.info, "Completed download of: %{public}@", assetDownloadItem.description)

        if let index = assetDownloadItems.firstIndex(where: { $0.url == assetDownloadItem.url && $0.isCompleted }) {
            assetDownloadItems.remove(at: index)
        }
    }
}

Here’s what we did:

1. AssetDownloadsSession now conforms to AssetDownloadItemDelegate.
2. Added the current AssetDownloadsSession instance as delegate to the new AssetDownloadItem instance.
3. When the delegate assetDownloadItemCompleted(_:) method is triggered, we remove that completed download from assetDownloadItems.

We are almost there, but we need to fix a big gotcha in the solution. Most of what AssetDownloadsSession does is manipulate the assetDownloadItems array. This array is updated from multiple locations, and there is currently no guarantee that the same thread will be used in all updates potentially leading to a race condition where thread A has triggered an AssetDownloadItem instance to be removed from the assetDownloadItems array just as thread B is attempting to coalesce that same instance. We can avoid these types of scenarios by using a serial dispatch queue to wrap all assetDownloadItems array updates:

class AssetDownloadsSession: AssetDownloadItemDelegate {
    //Omitted other properties

    private let accessQueue = DispatchQueue(label: "com.williamboles.downloadssession")

    //Omitted other methods

    private func registerForNotifications(on notificationCenter: NotificationCenter) {
        notificationCenter.addObserver(forName: UIApplication.didReceiveMemoryWarningNotification, object: nil, queue: .main) { [weak self] _ in
            self?.accessQueue.sync {
                //Omitted rest of method
            }
        }
    }

    // MARK: - Schedule

    func scheduleDownload(url: URL, completionHandler: @escaping DownloadCompletionHandler) {
        accessQueue.sync {
            //Omitted rest of method
        }
    }

    // MARK: - Cancel

    func cancelDownload(url: URL) {
        accessQueue.sync {
            //Omitted rest of method
        }
    }

    // MARK: - AssetDownloadItemDelegate

    fileprivate func assetDownloadItemCompleted(_ assetDownloadItem: AssetDownloadItem) {
        accessQueue.sync {
            //Omitted rest of method
        }
    }
}

And that's the media download layer complete 🥳.

How do we know it actually works? 🕵️

If you've run the example project, you will notice that downloads start, pause and finish but how we know the download is actually resuming after it's paused and isn't just starting again from 0%?

Well we can add in some more logging to get that information. URLSessionDownloadDelegate has a special method for downloads that are resumed. Lets add it into AssetDownloadsSession:

class AssetDownloadsSession: NSObject, AssetDownloadItemDelegate, URLSessionDownloadDelegate  // 1 {
    //Omitted properties

    // MARK: - Init

    init(urlSessionFactory: URLSessionFactory = URLSessionFactory(), notificationCenter: NotificationCenter = NotificationCenter.default) {
        super.init() // 2

        self.session = urlSessionFactory.defaultSession(delegate: self) // 3
        registerForNotifications(on: notificationCenter)
    }

    //Omitted other methods

    // MARK: - URLSessionDownloadDelegate

    func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didFinishDownloadingTo location: URL) { /*no-op*/ }

    // 4
    func urlSession(_ session: URLSession, downloadTask: URLSessionDownloadTask, didResumeAtOffset fileOffset: Int64, expectedTotalBytes: Int64) {
        guard let url = downloadTask.currentRequest?.url else {
            return
        }
        let resumptionPercentage = (Double(fileOffset)/Double(expectedTotalBytes)) * 100
        os_log(.info, "Resuming download: %{public}@ from: %{public}.02f%%", url.absoluteString, resumptionPercentage)
    }
}

Here’s what we did:

1. AssetDownloadsSession now conforms to URLSessionDownloadDelegate. As URLSessionDownloadDelegate inherits from NSObjectProtocol, AssetDownloadsSession now needs to be an NSObject subclass.
2. As AssetDownloadsSession is now an NSObject subclass, a call to super must be made in its init'er.
3. When creating the URLSession instance, we need to set the AssetDownloadsSession instance as the delegate of that session.
4. Implemented the urlSession(_:downloadTask:didResumeAtOffset:expectedTotalBytes:) to log the percentage of when a download is resumed.

It's interesting to note that when creating URLSessionDownloadTask instances, this solution is now using both the completion handler and the delegate.

As well as logging at what percentage a download is resumed, it is also useful to know that downloads percentage when it was paused:

fileprivate class AssetDownloadItem {
    //Omitted other properties

    private var observation: NSKeyValueObservation? // 1

    //Omitted other

    deinit {
        observation?.invalidate()  // 2
    }

    //Omitted other methods

    func resume() {
        //Omitted rest of method

        // 3
        observation = downloadTask?.progress.observe(\.fractionCompleted, options: [.new]) { [weak self] (progress, change) in
            os_log(.info, "Downloaded %{public}.02f%% of %{public}@", (progress.fractionCompleted * 100), self?.url.absoluteString ?? "")
        }
    }

    //Omitted other methods

    // 4
    private func cleanup() {
        observation?.invalidate()
        downloadTask = nil
    }
}

Here’s what we did:

1. Added a NSKeyValueObservation property so that it can outlive the method it will be created in.
2. When this AssetDownloadItem instance is deinit'ed, the NSKeyValueObservation instance is invalidated so it will stop observing.
3. Add an observe onto the progress property of the URLSessionDownloadTask instance. As it changes, a log is made detailing what the new download percentage is.
4. During cleanup, we invalidate the NSKeyValueObservation instance. N.B. cleanup is called when pausing a download as well as when completing a download so we need both observation?.invalidate() in both cleanup() and deinit().

With this additional logging, it is now possible to get a really good understanding of what is happening with our downloads.

Only half the story 📖

This has been a fairly long post so if you've made it here, take a moment to breathe out and enjoy it 👏.

To recap, we built a download media layer that:

1. Allows for scheduling, cancelling, pausing and resuming of download requests.
2. Allows for coalescing multiple download requests for the same asset.

And does this without exposing the complexity of pausing, resuming or coalescing.

With this new media download layer, we have improved download performance by enhancing the mechanics of downloading, but we can improve download performance further by:

1. Downloading the smallest possible asset required for the UI.
2. Predicting where the user is going and prefetching those assets.
3. Caching offline previously downloaded assets.

In the end, our users are going to have to wait for media to download, all that we can do is ensure that they are waiting due to their desire for ever-increasingly media-rich apps rather than our choices over how downloads are handled.

You can download the example project for this post here.

Running the example project 🏃

In the example project I used Imgur as an online image database to populate the app. Imgur has a great JSON based API and an extensive library of freely available media. Imgur API, while being free, does require us to register our example project to get a client-id which needs to be sent with each request.

As the example project does not require access to a user's personal data, when registering select the Anonymous usage without user authorization option for Authorization type. At the time of writing, you had to provide a URL in the Authorization callback URL field even for anonymous authentication - I found any URL value would work for this.

After you register, you should be given a unique client-id which will allow you access to Imgur's content. You will need to add your client-id as the value of the clientID property in RequestConfig. After you've added your client-id, you should be able to run the app and see how our background-transfer solution works.

This post is now out-of-date, please instead see: "Progressive Core Data Migration".

People really care about their possessions. Nowhere do you see this more than on public transport. It's not unusual to see bags occupying seats while people stand. As a Brit, we have developed a powerful non-verbal based form of communication to indicate that you want someone to move their bag - maybe a slight shuffle, eye contact with other standing commuters and tutting. Even with these clear signs some people have the audacity to ignore them and force you into the doing the unthinkable - speaking to a stranger on public transport.

"Excuse me, could you please move your bag so that I can sit down"

Surprisingly often, you are met with disdain as the other person consigns their bag to the indignity of the floor. As you settle into your seat and (pretend to) read, you begin thinking how oddly connected we are to our possessions rather than the needs other humans.

But it turns out that it's not just physical items that we really care about, we also feel the same way about our data. Especially if the data has been earned somehow - think about the sense of betrayal you feel when a game crashes and takes your just unlocked thingamabob with it.

In our iOS apps we often store these thingamabobs in Core Data. The structure of which is defined by a model/schema - a set of entities with attributes and relationships. A common situation in development is that over time our model changes. In order to allow these changes to happen we need to migrate our user's data from the old structure to the new structure. In this post, we are going to build a simple system to handle Core Data migrations.

If you want to follow along at home you can download the example project we will be working through from my GitHub repo. We are going to work through 4 different Models changes and see how our migration approach can handle each unique change.

Green field projects 🌿

On a green field project we don't have to care about migrating because we don't have anything to migrate. So lets see our first Core Data stack:

class CoreDataManager {

    lazy var persistentContainer: NSPersistentContainer! = {
        let persistentContainer = NSPersistentContainer(name: "CoreDataMigration_Example")

        return persistentContainer
    }()

    lazy var backgroundContext: NSManagedObjectContext = {
        let context = self.persistentContainer.newBackgroundContext()
        context.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy

        return context
    }()

    lazy var mainContext: NSManagedObjectContext = {
        let context = self.persistentContainer.viewContext
        context.automaticallyMergesChangesFromParent = true

        return context
    }()

    // MARK: - Singleton

    static let shared = CoreDataManager()

    // MARK: - SetUp

    func setup(completion: @escaping () -> Void) {
        loadPersistentStore {
            completion()
        }
    }

    // MARK: - Loading

    private func loadPersistentStore(completion: @escaping () -> Void) {
        self.persistentContainer.loadPersistentStores { description, error in
            guard error == nil else {
                fatalError("was unable to load store \(error!)")
            }

            completion()
        }
    }
}

CoreDataManager is a singleton who's responsibility is to setup the Core Data stack and provide access to various different contexts. If you have ever seen a Core Data stack setup before, you will instantly notice how little code the above manager contains. Over the years Core Data has evolved and become more developer friendly. Above, we are taking advantage of a relatively new piece of the Core Data family - NSPersistentContainer which was introduced in iOS 10. The NSPersistentContainer simplifies the creation of the managed object model, persistent store coordinator and the managed object contexts:

lazy var persistentContainer: NSPersistentContainer! = {
    let persistentContainer = NSPersistentContainer(name: "CoreDataMigration_Example")

    return persistentContainer
}()

Our example project is called CoreDataMigration-Example - see Apple's documentation on why the - has become a _

As we will see later we can still get access to our NSManagedModel, NSPersistentStoreCoordinator and NSManagedObjectContext instances via this container but we no longer have to copy and paste in their set-up code.

With loading a persistent store(s) being is an asynchronous operation, our setup needs to also be asynchronous:

func setup(completion: @escaping () -> Void) {
    loadPersistentStore {
        completion()
    }
}

// MARK: - Loading

private func loadPersistentStore(completion: @escaping () -> Void) {
    persistentContainer.loadPersistentStores { description, error in
        guard error == nil else {
            fatalError("was unable to load store \(error!)")
        }

        completion()
    }
}

As Core Data is so central to what we do in our apps, if we are unable to load the store the above method throws a fatal exception - this will allow us to fail fast during development.

Our model consists of a single Entity Post that consists of 3 properties:

• postID
• color
• date

From v1 to v2

So we release the app (with the version1 of the model) and our users ❤️ it.

Of course - you did a great job!

Due to this success we hire a new developer. In their first week they mistake the information stored in the color property on Post to be a representation of a color as an RGB string (when in fact it is represented as a hex string) which leads to the app crashing 😞. To avoid this issue happening when we hire more developers we decide to rename that property to hexColor. Now this is a change to the model which means a new model version which will result in a migration.

Before we delve straight into the migration itself, let's look more in depth at the migration process works.

The Migration Process

Core Data supports evolving the model over time. It does this by allowing us to create new versions of the model so that we end up with something like:

(The green tick indicating which version is currently being developed against.)

To change the current version you would switch the Model Version value shown below:

With this ability to evolve the model over time, we also need to handle migrating the user's data from one model version to another another. This is handled by creating a mapping model between those two versions. By default in iOS the migration process is completed in 1 step from source to destination models so if we support 4 versions, mapping models would exist for 1 to 4, 2 to 4 and 3 to 4. While this approach is the most efficient (in terms of processing and migration time), it is also the most developer and tester heavy. With each new destination model we need to redo all the paths between those models. So if we introduce version 5 we now need to be handle 1 to 5, 2 to 5, 3 to 5 and 4 to 5 - as you can see there no reuse from the previous migration paths. For each new version you must add n-1 new mappings. This can lead to a lot of work (and potential for 🐞s) for each new version of the model we introduce or convince us to drop support for migrating from certain versions and so result in corrupted data for any users on those dropped versions 😞.

Instead of following the default migration process, we will look at a system that performs migration over multiple steps i.e 1 to 2, 2 to 3, 3 to 5 and 4 to 5 - this means that for each new version of the model we need only add 1 additional step.

But before we get into get into the technical details of our migration implementation, lets look at our migration options.

The first question to ask yourself before engaging in the migration process is:

"Do I need to bother with migration?"

Seriously, just because you use Core Data in your project does not mean that you need to care about migrating the data stored there. If you only use Core Data as a local cache and always override it with the content you get from an API response, you probably don't need to go through the effort of migrating from one version to another. Just delete the local .sqlite files and recreate your Core Data stack, populating that new model with calls to the API. If that applies to you, you can stop reading now if you want or continue on with a certain smugness knowing that the difficulties being described below do not relate to you 😜.

Now in iOS you have 2 migration options:

1. Lightweight
2. Standard

Lightweight

Lightweight migration is where Core Data is able to infer how to migrate from source model to the destination model.

Standard

Standard migration is where Core Data isn't able to infer how to migrate and we have to detail the path by providing a custom mapping model *.xcmappingmodel and/or NSEntityMigrationPolicy subclasses.

It's important to note that both Lightweight and Standard migration techniques will produce a mapping model, it's just that in Standard the mapping model is explicitly created and lives in the project as a *.xcmappingmodel file.

Both Lightweight and Standard migration techniques can be achieved automatically or manually. By default with NSPersistentContainer Core Data will attempt to perform Standard and then fall back to Lightweight if it can't find the needed mapping model. For the rest of this post will be focused on handling all migrations manually - this allows us to specify the step-by-step approach mentioned above (automatic migrations try to migrate in 1 step). To disable automatic migrations we need to set the shouldInferMappingModelAutomatically on the NSPersistentStoreDescription to false - we will see this happening later.

From v1 to v2 cont.

So back to the task at hand. As we seen above we need to introduce a new version of the Core Data model.

This can be achieved by highlighting the xcdatamodel (it may be called xcdatamodeld) and going to Editor->Add Model Version.... We then just need to name it, I would suggest keeping it the same name and adding a number to it so CoreDataMigration_Example 2.

So now we have two models and we can freely edit the model's structure within CoreDataMigration_Example 2.

When it comes to renaming color to hexColor we have two options:

• canonical name
• *.xcmappingmodel

A canonical name effectively acts a bridge between what that property used to be called and what it now called. You would set the canonical name in the renaming identifier of that property. However here we are going to use *.xcmappingmodel.

As you see in the above screenshot we have created a custom mapping between CoreDataMigration_Example (effectively version1) and CoreDataMigration_Example 2. As mentioned above we have created a custom mapping because we have renamed a property color to hexColor and Core Data isn't able to infer that these properties are the same. With this mapping we are telling Core Data that the new hexColor property should be mapped to old color property by: $source.color.

So if we were not doing the step-by-step migration approach but rather the standard Core Data migration approach, that would be our job finished. However as we discussed the standard Core Data migration approach does not scale well so we need to create our custom migrator.

Step-by-step Migrations

In order to build our stack, we have to answer a few questions:

1. What is a Step?
2. How do we group steps?
3. What's responsible for triggering the migration?

Let's start answering them.

1. What is a Step?

struct CoreDataMigrationStep {

    let source: NSManagedObjectModel
    let destination: NSManagedObjectModel
    let mapping: NSMappingModel
}

A CoreDataMigrationStep is a migration between two versions of the model: source and destination and the actual mapping model itself.

It's possible to have multiple mapping models between versions, (this can be especially useful when migrating large data sets) in this post in an attempt to keep things simple I assume only one mapping model but if you need to support multiple mappings you would transform the mapping property into an array.

2. How do we group steps?

First, lets create a representation of what a model version is:

enum CoreDataVersion: Int {
    case version1 = 1
    case version2

    // MARK: - Accessors

    var name: String {
        if rawValue == 1 {
            return "CoreDataMigration_Example"
        } else {
            return "CoreDataMigration_Example \(rawValue)"
        }
    }

    static var all: [CoreDataVersion] {
        var versions = [CoreDataVersion]()

        for rawVersionValue in 1...1000 { // A bit of a hack here to avoid manual mapping
            if let version = CoreDataVersion(rawValue: rawVersionValue) {
                versions.append(version)
                continue
            }

            break
        }

        return versions.reversed()
    }

    static var latest: CoreDataVersion {
        return all.first!
    }
}

CoreDataVersion is an enum backed by an Int that should mirror the versions available in *.xcdatamodeld package. It provides a nice abstraction for what a version is and allows us to ask questions on a type such as:

• "Which is the latest version?"
• "What's the name of that model?"

With both CoreDataMigrationStep and CoreDataVersion abstraction we can create a migration path from the source model to the destination model:

class CoreDataMigrationModel {

    let version: CoreDataVersion

    var modelBundle: Bundle {
        return Bundle.main
    }

    var modelDirectoryName: String {
        return "CoreDataMigration_Example.momd"
    }

    static var all: [CoreDataMigrationModel] {
        var migrationModels = [CoreDataMigrationModel]()

        for version in CoreDataVersion.all {
            migrationModels.append(CoreDataMigrationModel(version: version))
        }

        return migrationModels
    }

    static var current: CoreDataMigrationModel {
        return CoreDataMigrationModel(version: CoreDataVersion.latest)
    }

    /**
     Determines the next model version from the current model version.

     NB: the next version migration is not always the next actual version. With
     this solution we can skip "bad/corrupted" versions.
     */
    var successor: CoreDataMigrationModel? {
        switch self.version {
        case .version1:
            return CoreDataMigrationModel(version: .version2)
        case .version2:
            return nil
        }
    }

    // MARK: - Init

    init(version: CoreDataVersion) {
        self.version = version
    }

    // MARK: - Model

    func managedObjectModel() -> NSManagedObjectModel {
        let omoURL = modelBundle.url(forResource: version.name, withExtension: "omo", subdirectory: modelDirectoryName) // optimized model file
        let momURL = modelBundle.url(forResource: version.name, withExtension: "mom", subdirectory: modelDirectoryName)

        guard let url = omoURL ?? momURL else {
            fatalError("unable to find model in bundle")
        }

        guard let model = NSManagedObjectModel(contentsOf: url) else {
            fatalError("unable to load model in bundle")
        }

        return model
    }

    // MARK: - Mapping

    func mappingModelToSuccessor() -> NSMappingModel? {
        guard let nextVersion = successor else {
            return nil
        }

        switch version {
        case .version1: //manual mapped versions
            guard let mapping = customMappingModel(to: nextVersion) else {
                return nil
            }

            return mapping
        default:
            return inferredMappingModel(to: nextVersion)
        }
    }

    func inferredMappingModel(to nextVersion: CoreDataMigrationModel) -> NSMappingModel {
        do {
            let sourceModel = managedObjectModel()
            let destinationModel = nextVersion.managedObjectModel()
            return try NSMappingModel.inferredMappingModel(forSourceModel: sourceModel, destinationModel: destinationModel)
        } catch {
            fatalError("unable to generate inferred mapping model")
        }
    }

    func customMappingModel(to nextVersion: CoreDataMigrationModel) -> NSMappingModel? {
        let sourceModel = managedObjectModel()
        let destinationModel = nextVersion.managedObjectModel()
        guard let mapping = NSMappingModel(from: [modelBundle], forSourceModel: sourceModel, destinationModel: destinationModel) else {
            return nil
        }

        return mapping
    }

    // MARK: - MigrationSteps

    func migrationSteps(to version: CoreDataMigrationModel) -> [CoreDataMigrationStep] {
        guard self.version != version.version else {
            return []
        }

        guard let mapping = mappingModelToSuccessor(), let nextVersion = successor else {
            return []
        }

        let sourceModel = managedObjectModel()
        let destinationModel = nextVersion.managedObjectModel()

        let step = CoreDataMigrationStep(source: sourceModel, destination: destinationModel, mapping: mapping)
        let nextStep = nextVersion.migrationSteps(to: version)

        return [step] + nextStep
    }

    // MARK: - Metadata

    static func migrationModelCompatibleWithStoreMetadata(_ metadata: [String : Any]) -> CoreDataMigrationModel? {
        let compatibleMigrationModel = CoreDataMigrationModel.all.first {
            $0.managedObjectModel().isConfiguration(withName: nil, compatibleWithStoreMetadata: metadata)
        }

        return compatibleMigrationModel
    }
}

CoreDataMigrationModel this is where the real magic happens.

func migrationSteps(to version: CoreDataMigrationModel) -> [CoreDataMigrationStep] {
    guard self.version != version.version else {
        return []
    }

    guard let mapping = mappingModelToSuccessor(), let nextVersion = successor else {
        return []
    }

    let sourceModel = managedObjectModel()
    let destinationModel = nextVersion.managedObjectModel()

    let step = CoreDataMigrationStep(source: sourceModel, destination: destinationModel, mapping: mapping)
    let nextStep = nextVersion.migrationSteps(to: version)

    return [step] + nextStep
}

The above method recursively builds an array containing all the steps required to perform a migration from the current version (as defined by the version property) to the latest version.

For each of these steps, the model determines if the mapping should be inferred or needs to explicitly defined by the developer:

func mappingModelToSuccessor() -> NSMappingModel? {
    guard let nextVersion = successor else {
        return nil
    }

    switch version {
    case .version1: //custom mapped versions
        guard let mapping = customMappingModel(to: nextVersion) else {
            return nil
        }

        return mapping
    default:
        return inferredMappingModel(to: nextVersion)
    }
}

While strictly speaking we didn't need to include the inferred branch for migrating form version1 to version2 (and normally I try to avoid using default), we will need an inferred mapping branch for future migrations so I've included it here for completeness.

As you can see we determine whether to create a custom or inferred mapping model by using a switch statement to check which version is current being accessed. You will know if you need to custom a mapping model if you break the rules as defined in the Lightweight documentation or more likely by attempting to perform an inferred migration and having the app crash on you during development.

func inferredMappingModel(to nextVersion: CoreDataMigrationModel) -> NSMappingModel {
    do {
        let sourceModel = managedObjectModel()
        let destinationModel = nextVersion.managedObjectModel()
        return try NSMappingModel.inferredMappingModel(forSourceModel: sourceModel, destinationModel: destinationModel)
    } catch {
        fatalError("unable to generate inferred mapping model")
    }
}

func customMappingModel(to nextVersion: CoreDataMigrationModel) -> NSMappingModel? {
    let sourceModel = managedObjectModel()
    let destinationModel = nextVersion.managedObjectModel()
    guard let mapping = NSMappingModel(from: [modelBundle], forSourceModel: sourceModel, destinationModel: destinationModel) else {
        return nil
    }

    return mapping
}

With inferredMappingModel we ask NSMappingModel to produce the mapping model by figuring out the differences and how to map between them. Again we follow the fail fast approach by catching the exception and then throwing a more meaningful fatal error.

customMappingModel is very similar but instead of getting an NSMappingModel instance based on Core Data figuring out the mapping, we ask it to search the app bundle and find a *.xcmappingmodel which has a matching source and destination model.

The next important part of CoreDataMigrationModel is to look at how the successor version of the current model is determined.

var successor: CoreDataMigrationModel? {
    switch self.version {
    case .version1:
        return CoreDataMigrationModel(version: .version2)
    case .version2:
        return nil
    }
}

As we only have two models we only need to handle migrating from version1 to version2. version2 being the current model, doesn't require a mapping model. You may be thinking that this is overkill and we could simplify this by always getting the next version up as the successor but sadly real-life isn't always so perfect and it's possible that we released a model version that contains issues and want to skip migrating any unaffected users to that version. With the above approach it would be possible to define a custom path. So if we pretend that we actually have four versions, it would be possible to skip a version altogether (in this case version3) which would give us the following structure:

var successor: CoreDataMigrationModel? {
    switch self.version {
    case .version1:
        return CoreDataMigrationModel(version: .version2)
    case .version2:
        return CoreDataMigrationModel(version: .version4) //skipping version3
    case .version3:
        return CoreDataMigrationModel(version: .version4)
    case .version4:
        return nil
    }
}

In order to perform we need to create an initial CoreDataMigrationModel instance based on the currently installed model version:

class CoreDataMigrationSourceModel: CoreDataMigrationModel {

    // MARK: - Init

    init?(storeURL: URL) {
        guard let metadata = NSPersistentStoreCoordinator.metadata(at: storeURL) else {
            return nil
        }

        let migrationVersionModel = CoreDataMigrationModel.all.first {
            $0.managedObjectModel().isConfiguration(withName: nil, compatibleWithStoreMetadata: metadata)
        }

        guard migrationVersionModel != nil else {
            return nil
        }

        super.init(version: migrationVersionModel!.version)
    }
}

CoreDataMigrationSourceModel is a convenience subclass of CoreDataMigrationModel. We will see later how this is used.

3. What's responsible for triggering the migration?

Ok, so we've looked at the how the steps are created and how each step knows which mapping model will move it to it's successor, below we are going to look at how those steps are called and how we prepare the app for a migration to occur.

class CoreDataMigrator {

    // MARK: - Check

    func requiresMigration(at storeURL: URL, currentMigrationModel: CoreDataMigrationModel = CoreDataMigrationModel.current) -> Bool {
        guard let metadata = NSPersistentStoreCoordinator.metadata(at: storeURL) else {
            return false
        }

        return !currentMigrationModel.managedObjectModel().isConfiguration(withName: nil, compatibleWithStoreMetadata: metadata)
    }

    // MARK: - Migration

    func migrateStore(at storeURL: URL) {
        migrateStore(from: storeURL, to: storeURL, targetVersion: CoreDataMigrationModel.current)
    }

    func migrateStore(from sourceURL: URL, to targetURL: URL, targetVersion: CoreDataMigrationModel) {
        guard let sourceMigrationModel = CoreDataMigrationSourceModel(storeURL: sourceURL as URL) else {
            fatalError("unknown store version at URL \(sourceURL)")
        }

        forceWALCheckpointingForStore(at: sourceURL)

        var currentURL = sourceURL
        let migrationSteps = sourceMigrationModel.migrationSteps(to: targetVersion)

        for step in migrationSteps {
            let manager = NSMigrationManager(sourceModel: step.source, destinationModel: step.destination)
            let destinationURL = URL(fileURLWithPath: NSTemporaryDirectory(), isDirectory: true).appendingPathComponent(UUID().uuidString)

            do {
                try manager.migrateStore(from: currentURL, sourceType: NSSQLiteStoreType, options: nil, with: step.mapping, toDestinationURL: destinationURL, destinationType: NSSQLiteStoreType, destinationOptions: nil)
            } catch let error {
                fatalError("failed attempting to migrate from \(step.source) to \(step.destination), error: \(error)")
            }

            if currentURL != sourceURL {
                //Destroy intermediate step's store
                NSPersistentStoreCoordinator.destroyStore(at: currentURL)
            }

            currentURL = destinationURL
        }

        NSPersistentStoreCoordinator.replaceStore(at: targetURL, withStoreAt: currentURL)

        if (currentURL != sourceURL) {
            NSPersistentStoreCoordinator.destroyStore(at: currentURL)
        }
    }

    // MARK: - WAL

    func forceWALCheckpointingForStore(at storeURL: URL) {
        guard let metadata = NSPersistentStoreCoordinator.metadata(at: storeURL), let migrationModel = CoreDataMigrationModel.migrationModelCompatibleWithStoreMetadata(metadata)  else {
            return
        }

        do {
            let model = migrationModel.managedObjectModel()
            let persistentStoreCoordinator = NSPersistentStoreCoordinator(managedObjectModel: model)

            let options = [NSSQLitePragmasOption: ["journal_mode": "DELETE"]]
            let store = persistentStoreCoordinator.addPersistentStore(at: storeURL, options: options)
            try persistentStoreCoordinator.remove(store)
        } catch let error {
            fatalError("failed to force WAL checkpointing, error: \(error)")
        }
    }
}

CoreDataMigrator is undertaking 3 tasks:

1. Determining if migration is necessary
2. Setting up a consistent state
3. Performing the migration

func requiresMigration(at storeURL: URL, currentMigrationModel: CoreDataMigrationModel = CoreDataMigrationModel.current) -> Bool {
    guard let metadata = NSPersistentStoreCoordinator.metadata(at: storeURL) else {
        return false
    }

    return !currentMigrationModel.managedObjectModel().isConfiguration(withName: nil, compatibleWithStoreMetadata: metadata)
}

In the above method, we are loading the meta data for the persistent store that is currently on the device and determining if it is compatible with latest version's meta data - returning either true or false based on the outcome of that comparison.

func forceWALCheckpointingForStore(at storeURL: URL) {
    guard let metadata = NSPersistentStoreCoordinator.metadata(at: storeURL), let migrationModel = CoreDataMigrationModel.migrationModelCompatibleWithStoreMetadata(metadata)  else {
        return
    }

    do {
        let model = migrationModel.managedObjectModel()
        let persistentStoreCoordinator = NSPersistentStoreCoordinator(managedObjectModel: model)

        let options = [NSSQLitePragmasOption: ["journal_mode": "DELETE"]]
        let store = persistentStoreCoordinator.addPersistentStore(at: storeURL, options: options)
        try persistentStoreCoordinator.remove(store)
    } catch let error {
        fatalError("failed to force WAL checkpointing, error: \(error)")
    }
}

Since iOS 7 Core Data has used Write-Ahead Logging (WAL) journalling by default on it's SQLite database. In WAL mode Core Data appends transactions to a -wal file and uses a -shm shared memory file, both in the same location as the main sqlite file. While this results in improved performance I noticed when testing that sometimes during migrations, the changes contained in the -wal file were not migrated. This would then result in a crash when anything in the new model format was then written to the old model formatted -wal file. The above method is forcing the changes contained in the -wal file to be committed to the main sqlite file.

Please note that in order for the -wal commit to be successful, it is necessary to load the model of the sqlite file on disk rather than the latest model.

As we will see below this forced committing of the WAL transactions will happen just before we perform the migration.

func migrateStore(at storeURL: URL) {
    migrateStore(from: storeURL, to: storeURL, targetVersion: CoreDataMigrationModel.current)
}

func migrateStore(from sourceURL: URL, to targetURL: URL, targetVersion: CoreDataMigrationModel) {
    guard let sourceMigrationModel = CoreDataMigrationSourceModel(storeURL: sourceURL as URL) else {
        fatalError("unknown store version at URL \(sourceURL)")
    }

    forceWALCheckpointingForStore(at: sourceURL)

    var currentURL = sourceURL
    let migrationSteps = sourceMigrationModel.migrationSteps(to: targetVersion)

    for step in migrationSteps {
        let manager = NSMigrationManager(sourceModel: step.source, destinationModel: step.destination)
        let destinationURL = URL(fileURLWithPath: NSTemporaryDirectory(), isDirectory: true).appendingPathComponent(UUID().uuidString)

        do {
            try manager.migrateStore(from: currentURL, sourceType: NSSQLiteStoreType, options: nil, with: step.mapping, toDestinationURL: destinationURL, destinationType: NSSQLiteStoreType, destinationOptions: nil)
        } catch let error {
            fatalError("failed attempting to migrate from \(step.source) to \(step.destination), error: \(error)")
        }

        if currentURL != sourceURL {
            //Destroy intermediate step's store
            NSPersistentStoreCoordinator.destroyStore(at: currentURL)
        }

        currentURL = destinationURL
    }

    NSPersistentStoreCoordinator.replaceStore(at: targetURL, withStoreAt: currentURL)

    if (currentURL != sourceURL) {
        NSPersistentStoreCoordinator.destroyStore(at: currentURL)
    }
}

In the above method we iterate through each of the migration steps using an instance of NSMigrationManager.

The more alert among you will have noticed that we store the user's data into a temporary sqlite file rather than override the starting sqlite file. This is a safety precaution incase an error happens during migration. We only overwrite the starting sqlite file once we know that the migration has been a success - this can be extremely useful during development.

In the above class we've seen a number of methods used that are not part of the standard NSPersistentStoreCoordinator API so I've included the extension that contains these methods below. As with most extensions, the methods are used to reduce boilerplate code.

extension NSPersistentStoreCoordinator {

    // MARK: - Destroy

    static func destroyStore(at storeURL: URL) {
        do {
            let persistentStoreCoordinator = NSPersistentStoreCoordinator(managedObjectModel: NSManagedObjectModel())
            try persistentStoreCoordinator.destroyPersistentStore(at: storeURL, ofType: NSSQLiteStoreType, options: nil)
        } catch let error {
            fatalError("failed to destroy persistent store at \(storeURL), error: \(error)")
        }
    }

    // MARK: - Replace

    static func replaceStore(at targetURL: URL, withStoreAt sourceURL: URL) {
        do {
            let persistentStoreCoordinator = NSPersistentStoreCoordinator(managedObjectModel: NSManagedObjectModel())
            try persistentStoreCoordinator.replacePersistentStore(at: targetURL, destinationOptions: nil, withPersistentStoreFrom: sourceURL, sourceOptions: nil, ofType: NSSQLiteStoreType)
        } catch let error {
            fatalError("failed to replace persistent store at \(targetURL) with \(sourceURL), error: \(error)")
        }
    }

    // MARK: - Meta

    static func metadata(at storeURL: URL) -> [String : Any]?  {
        return try? NSPersistentStoreCoordinator.metadataForPersistentStore(ofType: NSSQLiteStoreType, at: storeURL, options: nil)
    }

    // MARK: - Add

    func addPersistentStore(at storeURL: URL, options: [AnyHashable : Any]) -> NSPersistentStore {
        do {
            return try addPersistentStore(ofType: NSSQLiteStoreType, configurationName: nil, at: storeURL, options: options)
        } catch let error {
            fatalError("failed to add persistent store to coordinator, error: \(error)")
        }

    }
}

I won't go into their purpose as I think that is self-evident.

Wrapping it all into the manager

At the start of this post we looked at our simple, migration free Core Data stack, it's now time to go back to that manager and look at how supporting migrations will affect it.

class CoreDataManager {

    let migrator: CoreDataMigrator

    lazy var persistentContainer: NSPersistentContainer! = {
        let persistentContainer = NSPersistentContainer(name: "CoreDataMigration_Example")
        let description = persistentContainer.persistentStoreDescriptions.first
        description?.shouldInferMappingModelAutomatically = false //inferred mapping will be handled else where

        return persistentContainer
    }()

    lazy var backgroundContext: NSManagedObjectContext = {
        let context = self.persistentContainer.newBackgroundContext()
        context.mergePolicy = NSMergeByPropertyObjectTrumpMergePolicy

        return context
    }()

    lazy var mainContext: NSManagedObjectContext = {
        let context = self.persistentContainer.viewContext
        context.automaticallyMergesChangesFromParent = true

        return context
    }()

    // MARK: - Singleton

    static let shared = CoreDataManager()

    // MARK: - Init

    init(migrator: CoreDataMigrator = CoreDataMigrator()) {
        self.migrator = migrator
    }

    // MARK: - SetUp

    func setup(completion: @escaping () -> Void) {
        loadPersistentStore {
            completion()
        }
    }

    // MARK: - Loading

    private func loadPersistentStore(completion: @escaping () -> Void) {
        migrateStoreIfNeeded {
            self.persistentContainer.loadPersistentStores { description, error in
                guard error == nil else {
                    fatalError("was unable to load store \(error!)")
                }

                completion()
            }
        }
    }

    private func migrateStoreIfNeeded(completion: @escaping () -> Void) {
        guard let storeURL = persistentContainer.persistentStoreDescriptions.first?.url else {
            fatalError("persistentContainer was not set up properly")
        }

        if migrator.requiresMigration(at: storeURL) {
            DispatchQueue.global(qos: .userInitiated).async {
                self.migrator.migrateStore(at: storeURL)

                DispatchQueue.main.async {
                    completion()
                }
            }
        } else {
            completion()
        }
    }
}

As you can see, it's not too different to what we seen before. But let's look at what has changed:

let migrator: CoreDataMigrator

We store an instance CoreDataMigrator as a property which we pass in during init'ing the CoreDataManager instance:

init(migrator: CoreDataMigrator = CoreDataMigrator()) {
    self.migrator = migrator
}

As we will see in our unit testing by injecting the migrator we will be able to more easily test different scenarios by overriding it's methods.

As we are going to override the default migration process we next need to take control of it:

lazy var persistentContainer: NSPersistentContainer! = {
    let persistentContainer = NSPersistentContainer(name: "CoreDataMigration_Example")
    let description = persistentContainer.persistentStoreDescriptions.first
    description?.shouldInferMappingModelAutomatically = false //inferred mapping will be handled else where

    return persistentContainer
}()

Here is set shouldInferMappingModelAutomatically to false as our CoreDataMigrationModel class will handle setting the correct mapping model approach on each CoreDataMigrationStep step.

private func loadPersistentStore(completion: @escaping () -> Void) {
    migrateStoreIfNeeded {
        self.persistentContainer.loadPersistentStores { description, error in
            guard error == nil else {
                fatalError("was unable to load store \(error!)")
            }

            completion()
        }
    }
}

private func migrateStoreIfNeeded(completion: @escaping () -> Void) {
    guard let storeURL = persistentContainer.persistentStoreDescriptions.first?.url else {
        fatalError("persistentContainer was not set up properly")
    }

    if migrator.requiresMigration(at: storeURL) {
        DispatchQueue.global(qos: .userInitiated).async {
            self.migrator.migrateStore(at: storeURL)

            DispatchQueue.main.async {
                completion()
            }
        }
    } else {
        completion()
    }
}

The migrateStoreIfNeeded method continues with our fail fast approach by throwing a fatal error if the persistent container doesn't meet our expectations. Next we ask the question:

"Do we need to perform a migration here?"

If the answer is yes - the migrator then attempts to perform the migration on a background queue/thread before triggering it's completion closure once the migration is complete and allowing the Core Data stack to contine being set up.

If the answer is no - the completion block is called and setting up the Core Data stack continues unabated.

Migration completed! 🎉 🎉 🎉

Woohoo! That's us at the end of migrating from version1 to version2 of our model. It was a lot to take in but trust me, it will get much easier to perform future migrations now that we implemented this step-by-step approach.

As a recap, let's see a lovely diagram of what we just created:

From v2 to v3

So far we have migrated from model version1 to version2 but the new hexColor is still causing issues so instead we decide to extract it out into it's own Entity: Color. This change will required us to create a new version CoreDataMigration_Example 3 which looks like:

Now this migration is slightly trickier than from version1 to version2 and will require us to create not only a xcmappingmodel but also a custom NSEntityMigrationPolicy subclass.

As you see in the above screenshot we have created a custom mapping between CoreDataMigration_Example 2 and CoreDataMigration_Example 3. In version3 we have introduced a new entity Color which has taken the place of the previous color property on the Post entity - you can see in the 'Custom Policy' field we are using Post2ToPost3MigrationPolicy to handle migrating from this property to an entity.

final class Post2ToPost3MigrationPolicy: NSEntityMigrationPolicy {

    override func createDestinationInstances(forSource sInstance: NSManagedObject, in mapping: NSEntityMapping, manager: NSMigrationManager) throws {
        try super.createDestinationInstances(forSource: sInstance, in: mapping, manager: manager)

        guard let dInstance = manager.destinationInstances(forEntityMappingName: mapping.name, sourceInstances: [sInstance]).first else {
            fatalError("was expecting a post")
        }

        let color = NSEntityDescription.insertNewObject(forEntityName: "Color", into: dInstance.managedObjectContext!)
        color.setValue(UUID().uuidString, forKey: "colorID")
        color.setValue(sInstance.value(forKey: "hexColor"), forKey: "hex")

        dInstance.setValue(color, forKey: "color")
    }
}

One interesting point to note, is that we are dealing with NSManagedObject rather than our custom NSManagedObject subclasses. This is because Core Data wouldn't know which version of Post to load. So to work with both representations of the data we need to use KVC to get and set properties.

Next we need to make some changes to our CoreDataMigrationModel class by introducing a version3 case and handling migrating from version2 to version3:

var successor: CoreDataMigrationModel? {
    switch self.version {
    case .version1:
        return CoreDataMigrationModel(version: .version2)
    case .version2:
        return CoreDataMigrationModel(version: .version3)
    case .version3:
        return nil
    }
}

and as migrating to version3 will require a custom mapping model:

func mappingModelToSuccessor() -> NSMappingModel? {
    guard let nextVersion = successor else {
        return nil
    }

    switch version {
    case .version1, .version2: //manual mapped versions
        guard let mapping = customMappingModel(to: nextVersion) else {
            return nil
        }

        return mapping
    default:
        return inferredMappingModel(to: nextVersion)
    }
}

From v3 to v4

The success of our app knows no bounds and we decide to add the ability to hide posts that have been seen. We want to store this hidden value in our model. So we need to add a new version CoreDataMigration_Example 4. The good news here is that as this change involves adding properties to our model rather than transforming an existing Entity we can use the inferred mapping approach. This means that we need to make even fewer code changes than when we migrated from version2 to version3:

var successor: CoreDataMigrationModel? {
    switch self.version {
    case .version1:
        return CoreDataMigrationModel(version: .version2)
    case .version2:
        return CoreDataMigrationModel(version: .version3)
    case .version3:
        return CoreDataMigrationModel(version: .version4)
    case .version4:
        return nil
    }
}

And that's it.

Making sure it actually migrates

With something as important as data migrations we need to be sure that it actually works. While I won't post all of the unit tests here, it's important to touch on those unit tests as creating a Core Data stack that was unit testable played a major role in helping to shape the above approach. So let's look at how we test that it's possible to migrate from one version of the model to another:

func test_individualStepMigration_3to4() {
    let sourceURL = CoreDataMigratorTests.moveFileFromBundleToTmpDirectory(fileName: "CoreDataMigration_Example_3.sqlite")
    let targetURL = sourceURL

    let modifiedDateBeforeMigration = try! FileManager.default.attributesOfItem(atPath: sourceURL.path)[FileAttributeKey.modificationDate] as! Date

    migrator.migrateStore(from: sourceURL, to: targetURL, targetVersion: CoreDataMigrationModel(version: .version4))

    let modifiedDateAfterMigration = try! FileManager.default.attributesOfItem(atPath: targetURL.path)[FileAttributeKey.modificationDate] as! Date

    XCTAssertTrue(FileManager.default.fileExists(atPath: targetURL.path))
    XCTAssertTrue(modifiedDateAfterMigration.timeIntervalSince(modifiedDateBeforeMigration) > 0)
}

So in the test target we have a number of sqlite files filled with data for each version of our model, we then load the appropriate sqlite file and attempt to migrate checking that such a migration has in fact occurred and been successful.

Please see the GitHub repo for more unit tests.

We got there 🏁

Core Data migration can often seem like a very dense and difficult to understand process but hopefully with the above example you can see that by breaking it down into small steps and getting creative with how you then connect those steps together, you can actually take a lot of the difficulty out migrations.

And remember if that's how someone on the bus reacts to having to move their bag, we should try and spare this person the trauma that losing their unlocked thingamabobs would surely cause 😉.

I would like to acknowledge that I leaned on the most excellent Core Data book by Florian Kugler and Daniel Eggert which you can get here. I would highly recommend that you give that book a read as it's a treasure trove of Core Data knowledge.

I've found that talking about tech debt is a favourite subject among developers (especially if the developer responsible for introducing that tech debt is no longer on the project 😉). However, often talk is all that happens. As with any potentially ambiguous and daunting task it is easier to talk about it than to actually do it.

In this post I want to explore some of the roadblocks that can come up during these tech debt discussions, what the motivations for those roadblocks could be and how we can overcome them to get on with actually reducing the tech debt in our projects.

In no particular order, let's look at some.

#1. We could be building features instead of refactoring

Motivation

Any time spent addressing tech debt won't directly benefit the end user.

Solution

TL;DR: Keep tech debt refactoring small.

This is a valid concern and one that actually shows that the person raising it cares deeply about the experience the end user has. However it is also flawed as a good tech debt repayment plan involves making frequent small refactoring improvements when the pressure is off rather than big payments when the debt is stopping you from implementing a new feature that the company needs implemented now. This refactoring tasks if undertaken frequently enough will become part of your team's culture and as such tech debt payments will lose some of their importance and begin to look like any other development task. Not only will doing smaller refactoring reduce the stress of the refactoring, it will/can also improve the quality of unwritten code. It does this by setting the overall quality level/bar of the project higher - this is perfectly described by the Broken Window Theory (please note that I'm using the programming version of the Broken Window Theory as my inspiration above - not the actual New York City Police policy). I find it much easier to maintain a certain quality in a project than to try and achieve that quality.

#2. We need to sneak paying back tech debt into feature development

Motivation

That tech debt is solely the responsibility of the developers.

Solution

TL;DR: Promote tech debt repayments to a wider company level so that everyone is aware of the issues and the benefits from refactoring work.

Discuss how tech debt affects the wider company and agree with the other stakeholders in the project that a percentage of the weekly development effort will be spent addressing tech debt (or generally anything the tech feels is important). We may often feel that we work in feature factories but we are also responsible for the quality of project that we work on and if something is important to us we need to ensure that the wider company knows this. It's important to note that when describing these concerns to the wider company that we phrase the problem in a way that affects them - discuss how tech debt slows down development of new features, slows down development of extending features, leads to more bugs and can result in higher developer turn over (compromise for too long and you stop taking pride in your work).

In the past on joining a new team I've insisted on having a Captain for each iteration - this Captain is responsible for answering general project questions, uploading the app, releasing the approved version and resolving general bugs (not related to a feature under development). But the most important aspect of the Captain is that they are not assigned any feature (or Product) work, that way when they are not addressing the tasks listed above the Captain is focused on tackling tech debt with the total agreement of the wider company.

#3. We should only be refactoring feature "X"

Motivation

That paying back tech debt on any other feature than "X" is a waste of time and that everyone should focus on feature '"X".

Of course someone else feels the same way around feature "Y".

Solution

TL;DR: Create objectives for an extended period of time (e.g. a quarter) and use those objectives to guide your tech debt refactoring work.

Since the start of 2017 as a team we've been experimenting with OKRs and have found that having team wide objectives (that are connected to wider company objectives) to be a really useful filter when deciding on which tech debt item to address. As a team we can discuss if refactoring feature "X" or feature "Y" is connected to one of our objectives. If the refactoring isn't connected to an objective we can instantly discard it - it's not that it's not important, it's just that it's in an area of the app that we have chosen not to focus on.

The more cynical among you dear readers will be thinking:

All that does is shift the debate from the tech debt to OKR setting

and my response to that is:

...kind of

Team OKRs should be set based on the wider company objectives which should act as a limiting factor on debate and provide clear boundaries which allow us to better detect when we go off topic. For example if our objective was "Increase revenue by 25%" and we began discussing refactoring our profile screen, it becomes clear that we have moved off topic.

Even if you use OKRs it's still possible to have multiple different possible tech debt payment options. Here each member of the team gets a certain number of votes that they can spend on the possibilities - so if we have 5 possible options each person gets 2 votes. The option with the smallest number of votes is eliminated until we end up with just one option. That way everyone is involved in the decision making process and while they may not agree with output they can't deny that they had a say on what it was we should be doing. Often I find people don't mind not getting their own way but they really want to feel involved in the process and that their views were taken into consideration so the voting system is always great for overall team morale.

#4. We can't refactor feature "X" because it works

Motivation

That re-opening a feature that is stable is unacceptable risk.

Solution

TL;DR: Don't give into the fear of bugs, instead use tech debt refactoring to introduce automated testing into your project.

This one is tricky as it's a very valid concern. However you need to discuss what happens if you don't refactor it and something breaks anyway - would you prefer to refactor a feature when it is impacting your end users and your manager is demanding hourly progress updates or when you can take your time and unit test those features to death? Fear of introducing bugs isn't a valid argument for not tackling tech debt as the environments that we work in don't stand still, the feature may work in one version of the OS but be broken in the version just released. As developers we are solely responsible for introducing bugs into our apps - the only perfect app is the one that exists 100% in your head. A good tech debt repayment should include a form of automated testing to make bugs harder to be introduced in future and to better detail why you chose the solution you did. I believe that a lot of tech debt comes about because the intention of the original developer wasn't adhered to by subsequent developers - unit tests help to better express that intention by acting as a form of living documentation. Having to fix an unexpected failing test forces us to really think if our change makes sense.

#5. We can't refactor feature "X" until we agree on the solution

Motivation

To not create future tech debt by utilising group thinking.

Solution

TL;DR: Accept that every solution we implement will be imperfect and empower the developer to make their own decisions.

It's important in any team situation to try and seek an agreed approach however you also need to move fast and know when to break rules. A good solution today is better than a perfect one tomorrow. So where that agreed common solution can not be found the developer who is actually going to undertake the work has the final say in which solution they implement. The caveat here is that autonomy is only used here where both approaches are better than the current solution but optimise on different principles e.g. solution A on testability and solution B on readability.

#6. We don't have time for that

Motivation

To focus on other development tasks.

Solution

TL;DR: Explain the hidden cost of tech debt and make space for repaying it on the product backlog sooner rather than later.

Committing to repayments

Each day we get better as developers and discover new solutions to problems we have already solved, tech debt is the name that we give to those past decisions. Tech debt is unavoidable in development, it's the cold reminders of our past choices which can lead to very heated debates around future choices. Hopefully the above list of possible issues and how to overcome them will allow you to more quickly move onto making new tech debt for the future rather than just discussing old tech debt.