Relationships

There's an extension for that.

The YapDatabaseRelationship extension allows you to create a "relationship" between any two objects in the database. If you're familiar with relationships in Core Data, it works very much the same way.

• Graph concepts
  • Example 1: Parent -> child relationship (one-to-one)
  • Example 2: Parent -> children relationships (one-to-many)
  • Example 3: Parent -> child relationship + Notify (weak)
  • Example 4: Reference counting relationship
• Relationships with files
  • Example 1: Parent -> file (one-to-one)
  • Example 2: Reference counting a file
• Getting Started
• Edge creation
  • Manual edge creation
  • YapDatabaseRelationshipNode protocol
• Querying for edges
• Notify
• Differences with Core Data
• Graph Processing

 

Graph concepts

In common graph parlance, each object is called a node, and the line between the two nodes is called an edge.

(nodeA) -----> (nodeB)
         edge

YapDatabaseRelationship allows you to specify the edges, along with a rule to specify what should happen if one of the 2 nodes gets deleted from the database.

The edges are directional, however the associated delete rules are bidirectional. Thus you can create the edge in either direction, and setup the delete rules to match whatever you need.

In terms of edge direction, the edge starts from the "source" node (nodeA in the example above). And the object at the other end of the edge is called the "destination" node (nodeB in the example above).

Every edge has a name. This is simply a string that you define, which can be anything you want (except nil). The name is useful when searching the graph for particular edges. And it also comes into play when you create one-to-many, or many-to-many relationships.

Every edge also has a property called the 'nodeDeleteRules'. This is a bitmask, which you can specify using the following constants:

struct YDB_NodeDeleteRules: OptionSet {

  let rawValue: Int

  // notify only
  static let notifyIfSourceDeleted      = YDB_NodeDeleteRules(rawValue: 1 << 0)
  static let notifyIfDestinationDeleted = YDB_NodeDeleteRules(rawValue: 1 << 1)
      
  // one-to-one
  static let deleteSourceIfDestinationDeleted = YDB_NodeDeleteRules(rawValue: 1 << 2)
  static let deleteDestinationIfSourceDeleted = YDB_NodeDeleteRules(rawValue: 1 << 3)
  
  // one-to-many & many-to-many
  static let deleteSourceIfAllDestinationsDeleted = YDB_NodeDeleteRules(rawValue: 1 << 4)
  static let deleteDestinationIfAllSourcesDeleted = YDB_NodeDeleteRules(rawValue: 1 << 5)
}

Note that because it's a bitmask, you can combine multiple rules together. But not all rule combinations are legal (because they don't make logical sense, as you'll see shortly).

Let's look at a few examples:

 

Example 1: Parent -> child relationship (one-to-one)

In this situation, the "parent" owns the "child", and the child object should be removed from the database if the parent is removed from the database. Usually the child object is a part of the parent, and only makes sense if the parent exists. For example, the parent object is a "Player", and the child object is an "Avatar".

So you could create the edge like this:

(player) ----------> (avatar)
          "avatar",[.deleteDestinationIfSourceDeleted]

The sourceNode is the "player" object, and the destination node is the "avatar" object. The edge name is "avatar", and the edge nodeDeleteRules specify that if the player object (source) is deleted, then the avatar object (destination) should be automatically deleted as well.

If it is more convenient for you, you could easily create the same edge in reverse, and get the exact same effect:

(avatar) ----------> (player)
          "player"[.deleteSourceIfDestinationDeleted]

It does not make any difference which way you do it. Whichever is easiest, or most convenient for you.

 

Example 2: Parent -> children relationships (one-to-many)

This is really the exact same as the example above. You just create multiple edges. For example, the parent object is a "Contact", and the child objects are "PhoneNumber"s. Because a contact may have multiple phone numbers.

So you could create the edges like this:

(contact) ----------> (phoneNumber1)
           "number"[.deleteDestinationIfSourceDeleted]
(contact) ----------> (phoneNumber2)
           "number"[.deleteDestinationIfSourceDeleted]
(contact) ----------> (phoneNumber3)
           "number"[.deleteDestinationIfSourceDeleted]

So basically, creating a "one-to-many" relationship is easy. You just create an edge for each relationship.

And again, you could alternatively do the inverse:

(phoneNumber1) ----------> (contact)
                "contact"[.deleteSourceIfDestinationDeleted]
(phoneNumber2) ----------> (contact)
                "contact"[.deleteSourceIfDestinationDeleted]
(phoneNumber3) ----------> (contact)
                "contact"[.deleteSourceIfDestinationDeleted]

 

Example 3: Parent -> child relationship + Notify (weak)

In the first example (player -> avatar), we setup an edge that would automatically delete the "avatar" if the "player" was deleted. But what happens if we delete the "avatar"? This might be important if there is a person.avatarKey property, and we want to automatically set this property to nil if the avatar is deleted.

This is typically referred to as a "weak" relationship. And we can do something similar using the YapDatabaseRelationship extension.

(player) ----------> (avatar)
          "avatar",[.deleteDestinationIfSourceDeleted, .otifyIfDestinationDeleted]

This is an example of combining rules to meet your needs. Here we've specified that if the player is deleted, the avatar should automatically be deleted as well. (Same as before.) But if only the avatar is deleted, a "notify" method is invoked on the player object which allows us to set the player.avatarKey property to nil.

The Notify section describes the notify method, how it works, and how to implement it.

 

Example 4: Reference counting relationship

There may be times when you place items in the database which have multiple parents. Here are a few examples:

• You're making a real estate application, and you download neighborhood information on demand. (Think crime statistics, school districts, etc.) You may want to keep this information around until the properties in the neighborhood get removed. And there may be multiple properties in a single neighborhood that reference the info.

• You download a photo that belongs to a post. But a bunch of other people forwarded/reposted/retweeted the same post, and their post has the same photoId. And you want to automatically delete the photo when all the associated posts expire from the client side database.

These situations are easily handled with YapDatabaseRelationship, using a simple reference counting type scheme. Here's how it works:

(property1) ----------------> (neighborhoodInfo)
             "neighborhood"[.deleteDestinationIfAllSourcesDeleted]
(property2) ----------------> (neighborhoodInfo)
             "neighborhood"[.deleteDestinationIfAllSourcesDeleted]
(property3) ----------------> (neighborhoodInfo)
             "neighborhood"[.deleteDestinationIfAllSourcesDeleted]

Now let's say that property1 gets deleted (maybe because it was sold, and thus no longer on the market). The extension will look at the edges in the database, and see if there are any others with the same name and destination node. In this case, it will find the edges from property2 & property3. And so the neighborhoodInfo will remain in the database, and only the first edge is deleted. But when property2 & property3 are eventually deleted (and if no other edges were added with the same name, pointing at the same neighborhoodInfo), then the neighborhoodInfo gets automatically deleted.

 

Relationships with files

Not all your data will be stored in the database. You'll likely choose to store some of your data directly on the file system. And there are very good reasons to do so.

Generally speaking, storing large files into the database isn't recommended. Apple's own recommendations for Core Data generally states:

If fileSize > 1mb, then store the file on disk and reference the filePath from your database object

And these recommendations basically come directly from sqlite, because the sqlite database stores its data in pages. Which means storing a really big file into the database will require breaking it up into a bunch of pages, and then writing these pages to the database file. The pages might end up a bit scattered... And, long story short, for larger files it's pretty much guaranteed you're going to get better performance from directly using the file system. Both in terms of writing the file, and reading it back.

The most common example is images. Once images reach a certain size, you're going to get better performance from storing these images directly on the filesystem versus storing them in the database.

For more detailed information on this topic (and statistics!), see the FAQ: Should I store images in YapDatabase?.

Fortunately this kind of stuff is really easy to accomplish with YapDatabase. Because you can create an edge from an object-in-the-database to a file-on-the-filesystem ! You need only use a fileURL property in your objects:

(object in the database) ----------> (file on the filesystem)
          "avatar",[.deleteDestinationIfSourceDeleted]

For example, we may represent all the information about a Player in our database object. But the player's avatar may be a rather big image. So we store that directly on the file system, and reference it's location within the player object.

The YapDatabaseRelationship extension supports this architecture, and can be used to automatically delete the external file for you automatically!

Let's look at a few examples:

 

Example 1: Parent -> file (one-to-one)

In this situation, the "parent" object "owns" the file, and the file should be deleted from the file system if the parent is removed from the database. For example, the parent object is a "Player", and the file is an image associated with the player.

So you could create the edge like this:

(player) ----------> (avatar image stored on the filesystem)
          "avatar",[.deleteDestinationIfSourceDeleted]

You'll notice this looks no different than a relationship between two objects in the database. And that's by design. Internally, the relationship extension has some extra code to handle files, and it simply performs a file deletion (rather than an object deletion) at the appropriate time.

So if the player object (source) is deleted, then the avatar file (destination) will be automatically deleted as well.

 

Example 2: Reference counting a file

As you'd expect, this works exactly the same as a reference counting relationship between 2 objects in the database. (Like example 4 above.) But in this situation, one of the nodes is a file.

(city) --------------> (stateFlag image on the filesystem)
         "stateFlag"[.deleteDestinationIfAllSourcesDeleted]
(city) --------------> (stateFlag image on the filesystem)
         "stateFlag"[.deleteDestinationIfAllSourcesDeleted]
(city) --------------> (stateFlag image on the filesystem)
         "stateFlag"[.deleteDestinationIfAllSourcesDeleted]

This allows you to share a file between a bunch of different objects in the database. And if all the objects that reference the file get deleted, then the database automatically deletes the file for you.

IMPORTANT: The relationship extension does NOT monitor the filesystem. So you CANNOT, for example, use a nodeDeleteRule that specifies that if the file gets deleted, then the corresponding database object gets deleted. This won't work. If you need such functionality, then you'll need to write it yourself. Most likely, you're the one deleting the file(s) anyway. And if not... this kind of functionality isn't really feasible, because the file deletion is occurring outside the scope of a transaction, and thus there would be edge cases.

 

Getting Started

Just register the relationship extension after initializing your database:

let ext = YapDatabaseRelationship()

let extName = "relationships"
database.asyncRegister(ext, withName: extName) {(ready) in
  if !ready {
    print("Error registering extension: \(extName)")
  }
}

 

Edge creation

There are two ways in which you can create an edge:

1. Manually add the edge by invoking a method of YapDatabaseRelationship
2. Implement the YapDatabaseRelationshipNode protocol for your classes/structs

Multiple ways exist for convenience. You can use either. Or you can use both.

 

Manual edge creation

There are methods to add & remove edges in YapDatabaseRelationshipTransaction:

class YapDatabaseRelationshipTransaction {

  func add(_ edge: YapDatabaseRelationshipEdge)

  func remove(_ edge: YapDatabaseRelationshipEdge,
              withProcessing reason: YDB_NotifyReason)

  func removeEdge(withName edgeName: String,
                  sourceKey: String,
                  collection sourceCollection: String?,
                  destinationKey: String,
                  collection destinationCollection: String?,
                  withProcessing reason: YDB_NotifyReason)

  // ...
}

For manual edge creation, you generally just add the edge at the same time you add the objects. For example:

dbConnection.asyncReadWrite {(transaction) in

  transaction.setObject(newPlayer, forKey: newPlayer.playerId inCollection: "players")
  transaction.setObject(newPlayerAvatar, forKey: newPlayer.playerId inCollection: "avatars")

  let edge = 
    YapDatabaseRelationshipEdge(name: "avatar"
                           sourceKey: newPlayer.playerId
                          collection: "players"
                      destinationKey: newPlayer.playerId
                          collection: "avatars"
                     nodeDeleteRules: [.deleteDestinationIfSourceDeleted])

  if let relationshipTransaction = transaction.ext("relationships") as? YapDatabaseRelationshipTransaction {
    relationshipTransaction.add(edge)
  }
}

 

YapDatabaseRelationshipNode protocol

Most "relationships" are built into your data model:

class Song: Codable {
  // ...
  let albumId: String
  let artistId: String
  // ...
}

Thus, given a song identifier, you can easily fetch the associated album & artist too:

dbConnection.read {(transaction) in
  
  if let song = transaction.object(forKey: songId, inCollection: "songs") as? Song {
    let album = transaction.object(forKey: song.albumId, inCollection: "albums") as? Album
    let artist = transaction.object(forKey: song.artistId, inCollection: "artists")
  }
}

The YapDatabaseRelationshipNode protocol builds atop this standard pattern. It allows you to keep your data structured in this standard style, and provides a hook for your objects to report back what relationship edges they want. For example, if we were implementing the Song object (from above):

class Song: Codable, YapDatabaseRelationshipNode // <= Add this
  //...
  let albumId: String
  let artistId: String
  // ...

  // This function gets automatically called when
  // the song is inserted/updated in the database.
  func yapDatabaseRelationshipEdges() -> [YapDatabaseRelationshipEdge]? {
    
    let albumDeleteRules: YDB_NodeDeleteRules = [
      // automatically delete the album if all associated songs are deleted
      .deleteDestinationIfAllSourcesDeleted,
      
      // automatically delete this song if the album is deleted
      .deleteSourceIfDestinationDeleted
    ]
    let albumEdge =
      YapDatabaseRelationshipEdge(name: "album"
                        destinationKey: albumId
                            collection: "albums"
                       nodeDeleteRules: albumDeleteRules)
    
    let artistDeleteRules: YDB_NodeDeleteRules = [
      // automatically delete the artist if all associated songs are deleted
      .deleteDestinationIfAllSourcesDeleted,
      
      // automatically delete this song if the artist is deleted
      .deleteSourceIfDestinationDeleted
    ]
    let artistEdge =
      YapDatabaseRelationshipEdge(name: "artist"
                        destinationKey: artistId
                            collection: "artists"
                       nodeDeleteRules: artistDeleteRules)

    return [albumEdge, artistEdge]
  }
}

The extension will invoke yapDatabaseRelationshipEdges() automatically:

• when the object is first inserted into the database
• and anytime the object is updated in the database

(Ensure that your class/struct conforms to the YapDatabaseRelationshipNode protocol.)

 

Querying for edges

You can query the database to find particular edges using any combination of of the following:

• edge name
• source node
• destination node

For example, if you wanted to find all phone numbers for a particular contact:

var numbers: [PhoneNumber] = [];
dbConnection.read {(transaction) in
  
  guard
    let contact = transaction.object(forKey: contactId, inCollection: "contacts") as? Contact,
    let relationshipTransaction = transaction.ext("relationships") as? YapDatabaseRelationshipTransaction
  else {
    return
  }
    
  relationshipTransaction.enumerateEdges(withName: "number"
                                        sourceKey: contactId
                                       collection: "contacts") {
    (edge, stop) in
    
    if let number = transaction.object(forKey: edge.destinationKey, inCollection: edge.destinationCollection) as? PhoneNumber {
      numbers.append(number)
    }
  }
}

See YapDatabaseRelationshipTransaction.h for more enumerate methods. There are also similar versions of these methods if you just want to grab the edge count.

 

Notify

The YapDatabaseRelationshipNode protocol defines an optional method for being notified of a deleted edge:

optional func yapDatabaseRelationshipEdgeDeleted(_ edge: YapDatabaseRelationshipEdge, with reason: YDB_NotifyReason) -> Any?

This is designed to work with the .notifyIfSourceDeleted & .notifyIfDestinationDeleted flags, and can be used to implement weak relationships. (i.e. setting a property to nil when the item it references is deleted.)

Let's look at some example code (taken from Example #3 above):

class Player: Codable, NSCopying, YapDatabaseRelationshipNode {

  var avatarID: String?

  func yapDatabaseRelationshipEdges() -> [YapDatabaseRelationshipEdge]? {
    
    guard let avatarID = self.avatarID else {
      return nil
    }

    let edgeDeleteRules: YDB_NodeDeleteRules = [
      // automatically delete the avatar if the player is deleted
      .deleteDestinationIfSourceDeleted,
      
      // automatically invoke "notify method" (below) if only avatar is deleted
      .notifyIfDestinationDeleted
    ]
    let edge = // (player) --> (avatar)
      YapDatabaseRelationshipEdge(name: "avatar"
                        destinationKey: avatarID
                            collection: "avatars"
                       nodeDeleteRules: edgeDeleteRules)   

    return [edge]
  }
  
  func yapDatabaseRelationshipEdgeDeleted(_ edge: YapDatabaseRelationshipEdge, with reason: YDB_NotifyReason) -> Any? {
    
    if edge.name == "avatar" {
        // The "avatar" has been deleted.
        // Let's set the avatarID property to nil.
        
        let copy = self.copy() as! Player
        copy.avatarID = nil
        return copy
    }
    
    return nil
  }

}

If you return an object from the yapDatabaseRelationshipEdgeDeleted:withReason: method, then the returned object automatically replaces the previous object in the database. Specifically, the code invokes replaceObject:forKey:inCollection:. I.E. the object is replaced, but any existing metadata remains as is.

If you return nil, then your previous object remains as-is in the database.

 

Differences with Core Data

Since many developers are familiar with Core Data, it's instructive to point out some of the differences. (We will not be describing all the differences between Core Data and YapDatabase. Only those differences that apply to relationships in Core Data versus relationships in YapDatabase.)

In Core Data, you use the model editor to add a relationship to an entity. In doing so you specify what type of entity the relationship points to (it's class). And you are highly encouraged to create an inverse relationship. Which means editing the destination entity to point back to the source entity.

Some people, when migrating from Core Data to YapDatabase, decide they need 2 edges in YapDB. One from the source to the destination. And another from the destination back to the source. This is a mistake.

With YapDatabaseRelationship there's no need to create inverse edges.

Yes, the edge is "directional". But that doesn't mean anything. You can "traverse" the edge in either direction. So if you have the destination node, you can get the edge, and find the source node.

Edges are directional to make them easier to use. It makes it easier to run queries when edges are directional, and you can specify either the source or destination. And I think you'll find that directional edges fit nicely into your mental model of the data graph.

But if I don't specify an inverse edge, then how do I specify the inverse delete rule?

The nodeDeleteRules of an edge are bi-directional.

So even though an edge is "directional", you can specify what should happen if:

• The source node is deleted
• The destination node is deleted

To illustrate this concept, let's revisit some code from an example above. We were creating a database of songs, and we stored both Song objects & Album objects. The song object created an edge to the album object. However, the node delete rules specified the following:

• If the song is deleted, then delete the album IF there are no other songs pointing to the album
• If the album is deleted, then delete the song(s)

The code looked like this:

func yapDatabaseRelationshipEdges() -> [YapDatabaseRelationshipEdge]? {
  
  let albumDeleteRules: YDB_NodeDeleteRules = [
    // automatically delete the album if all associated songs are deleted
    .deleteDestinationIfAllSourcesDeleted,
    
    // automatically delete this song if the album is deleted
    .deleteSourceIfDestinationDeleted
  ]
  let albumEdge =
    YapDatabaseRelationshipEdge(name: "album"
                      destinationKey: albumId
                          collection: "albums"
                     nodeDeleteRules: albumDeleteRules)
  
  // ...
  
  return [albumEdge, artistEdge]
}

Let's continue with this Song & Album example to illustrate another difference. If we were modeling the Album entity in Core Data, we would specify a one-to-many relationship. Thus we might end up with some goofy convoluted ugliness like this:

class Album: NSManagedObject {
  
  @NSManaged var songs: Set<Song>? // Barf 🤮
}

With YapDatabase, you don't have to do any of this silliness.

With YapDatabase, there would be no need to store the list of songs within the Album object. In fact, doing so is likely just useless overhead (both in terms of storage space, and additional work required on your part to always keep both the Song & Album objects in-sync).

Instead you have multiple options to achieve the same thing:

Probably the easiest would be to use Views. This would give you a pre-sorted song list for any album in the database. And you could use it to act as the datasource for a tableView or collectionView.

I used Core Data extensively for many years before I created YapDatabase. One of the things I learned is that relationships in Core Data can create a significant amount of overhead. But it's difficult to do without them because simple "objectForKey" lookups aren't so simple in Core Data. A collection/key/value database changes this dramatically. One of the key concepts to understand when switching to YapDatabase is that relationships can be implicit or explicit.

By "implicit relationship" I mean that you don't create an edge (with the relationships extension) between two objects. You just have properties that point to each other. Like this:

class Car: Codable
  var makeId: String // e.g. points to Ford
  // ...
}

So a Car object might have an "implicit relationship" to the Ford object. Most of the time there is no reason to create an explicit graph edge from the Car object to the Ford object. An "explicit relationship" would have a corresponding graph edge.

With YapDatabase, explicit relationships are often unneeded. Implicit relationships handle the majority of use cases (and without any additional overhead). The primary reason to create explicit relationships is to make use of the nodeDeleteRules. If you don't need the nodeDeleteRules, there's usually little reason to create a graph edge. (The query capabilities can always be handled more efficiently by a view or a secondary index.)

So if you're coming to YapDatabase with a strong background in Core Data, keep in mind that YapDatabase has both implicit and explicit relationships. So for every relationship in your data model, just ask yourself the following question:

Does this relationship require nodeDeleteRules?

If the answer is NO (either because they're not needed, or because you already have code to handle it), then an explicit relationship (graph edge) is likely unnecessary.

 

Graph Processing

In order to optimize database access, the YapDatabaseRelationship extension performs its node deletion processing just before the commit. That is, after all the code in your transaction block has finished. For example:

dbConnection.readWrite {(transaction) in
    
  guard let player = transaction.object(forKey: playerId, inCollection: "players") as? Player else {
    return
  }
  let avatarId = player.avatarKey

  // (player) ----------> (avatar)
  //           "avatar",[.deleteDestinationIfSourceDeleted]
  
  transaction.removeObject(forKey: playerId, inCollection: "players")

  let exists = transaction.hasObject(forKey: avatarId, inCollection: "avatars")
  // exists == true !!!
  // But why?
  // Because the YapDatabaseRelationship extension runs the node deletion
  // processing at the very end of this transaction, after our code has run.
}

dbConnection.readWrite {(transaction) in

  let exists = transaction.hasObject(forKey: avatarId, inCollection: "avatars")
  // exists == false
  // Because the avatar was deleted in accordance with the\
  // nodeDeleteRules of the edge.
}

This optimization allows the extension to optimize cache access, and also allows it to skip deletion attempts on objects that you manually delete. For example, if you manually delete both the player and related avatar, the extension will see this, and then knows it only needs to remove the edge from the database.

However, you can manually invoke the 'flush' method to force the extension to run all its processing code. This may be useful when testing to ensure everything is getting deleted properly.

dbConnection.readWrite {(transaction) in
  
  guard let player = transaction.object(forKey: playerId, inCollection: "players") as? Player else {
    return
  }
  let avatarId = player.avatarKey

  // (player) ----------> (avatar)
  //           "avatar",[.deleteSourceIfDestinationDeleted]
  
  transaction.removeObject(forKey: playerId, inCollection: "players")

  if let relationshipTransaction = transaction.ext("relationships") as? YapDatabaseRelationshipTransaction {
    relationshipTransaction.flush() // run node deletion processing now
  }

  let exists = transaction.hasObject(forKey: avatarId, inCollection: "avatars")
  // exists == false
  // Because the avatar was deleted in accordance with the
  // nodeDeleteRules of the edge.
}

One other thing to note about delayed graph processing: It allows to you create edges to nodes that don't exist yet. For example:

dbConnection.readWrite {(transaction) in

  let edge = 
    YapDatabaseRelationshipEdge(name: "avatar"
                           sourceKey: player.playerId
                          collection: "players"
                      destinationKey: player.playerId
                          collection: "avatars"
                     nodeDeleteRules: [.deleteDestinationIfSourceDeleted])

  if let relationshipTransaction = transaction.ext("relationships") as? YapDatabaseRelationshipTransaction {
    relationshipTransaction.add(edge)
  }
  
  // We're adding the edge BEFORE we're adding the dstNode...
  // Is that OK?
  // The answer is YES. Because the extension will wait until our transaction
  // completes before requiring the edge node actually exists.
  // So as long as we also add the avatar during this transaction,
  // everything works fine.
  
  transaction.setObject(playerAvatar, forKey: player.playerId inCollection: "avatars")
}