JP Simard on Realm & Open Source on the Consult Podcast

Consult is a monthly interview podcast hosted by David Kopec. It focuses on software developers who work on Apple platforms to create client products.

JP Simard, iOS lead at Realm, joins the show to discuss the Realm Mobile Database and the new Realm Mobile Platform. JP digs into the technical details that make Realm a great choice for developers. He also discusses his prodigious open source work and his prior life as lead of an iOS consulting firm.

Listen to the full interview here, or check out the episode’s transcript below.


Introduction (0:00)

Hi everyone. Welcome to the November 2016 episode of Consult. I do want to reach out to the community, so if you know somebody who you think would be great on Consult, let me know. I’m @davekopec on Twitter.

My interview this month is with JP Simard of Realm. You all know Realm, the popular, open source mobile data base that’s really starting to take on Core Data. We really dig into the details of Realm and we also get into JP’s previous career as an iOS consultant.

A lot of you probably know JP from the open source community. He’s also the iOS lead at Realm.

Background (1:58)

Q: Tell us a little bit about your background. Go way back. How did you first get into computing? I know you had a consulting company for a number of years, tell us about that too.

JP: Going way back for me doesn’t actually go back all that far. My first intro into programming was in university. I was taking electrical engineering and I was honestly pretty fed up with the very long project cycles. You send something over to fab, or you do simulations. Essentially, every project either takes something on the scale of months or years, if you look at commercial projects. That just wasn’t very exciting for me.

I ended up building this technology that was linked to electrical engineering. Then I wanted to build an app around it. I stared programming, trying to figure out how you actually do that and then I realized that I had to pay the bills. I went around a few different festivals in my hometown in Ottawa, Canada. I tried to pitch them on making an app, even though I had never done it before. Eventually one of them stuck, and ended up having to build an app for them. This was in 2011, right when I was getting started.

I got by right by the skin of my teeth. I managed to patch together just enough of an app and interface builder to satisfy the contract and move on. I continued with that process, going door to door, asking friends, acquaintances, trying to figure out who needed an app, saying that I could build it without ever having anything close to it. Over time I ended up actually fulfilling those contracts, then having more contracts and growing that team.

Essentially, that became a business that was initially spawned by this idea of just scratching my own itch, making my own app. Ultimately, I never ended up building that app, but did grow that agency, something called Magnetic Bear Studios into a small team. It consisted of a few developers, a few designers and making apps for all sorts of companies across North America. At that point I was really hooked into programming. There was no escape.

iOS in Canada (4:29)

Q: Very interesting. Tell us a little bit about the iOS scene in Canada, in Ottawa more specifically. I actually lived for Ottawa for a year when I was a little kid, but I know nothing about the iOS scene there. What’s it like?

JP: I love it because it’s a very tight knit community. Now living in Silicon Valley, in San Francisco, there are definitely a lot of people in the community here. You tend to go to meet-ups. There are tons of them to start off with, and they’re actually competing in some sense; competing for the market share of local developers. Not only that, you tend to have such large groups or groups with members that pop in and out every now and again. There’s a really big difference between that, and say specifically, the Apple developer community in a small-ish city like Ottawa where you have people who are really ingrained in the community.

It’s a very small, tight knit group, a group of maybe 20 or 30 really hardcore people who have been doing Mac programming since the 90’s, for some of them. People who have really been ingrained in the industry. There are so few of them there, relatively speaking, compared to a major hub like New York City, London, or San Francisco, that you really do have to stick together. It’s very regular meetings for groups like Cocoa Heads, for example, and it’s people who are keen on sharing ideas and sharing what they learn on a very regular basis and keeping in touch.

It’s an interesting dynamic, and I certainly miss it in a lot of ways. I like being nostalgic every now and again, thinking back to those days. In the last few years, there’s this conference called NSNorth that’s run by Philippe Casgrain and Dan Byers. They’re able to attract people from all around the world to come to this conference because they’re so passionate about it and because they’ve been doing it for decades at this point, and the community is really small but strong.

Clients (7:00)

Q: I’ve always heard tremendous things about NSNorth and what a great experience it is for the attendees. When you were at Magnetic Bear Studios and running that, did you have mostly local clients or were the clients from all across Canada and the United States?

JP: For the most part they were local clients. We did have a handful of clients when we were growing and making a name for ourselves that were everywhere from Silicon Valley to both coasts of Canada. Ultimately, what lead to the end of the Magnetic Bear Studios agency was really my own limitations. At that point, I was running this business that I had no idea how to run. It was mostly focused on hustling and doing sales, doing business development, doing everything from accounting to everything that falls into managing small business.

Ultimately, not only was I not very good at that, I also didn’t really like it. All of that lead to the conclusion that towards the end of that three and a half year stint I was really looking for something a little bit more technical to sink my teeth into and that’s when I joined Realm.

Building a Team (8:24)

Q: I want to get more into Realm, but still talking about Magnetic Bear Studios, what was it like building out that team in Canada? Was there enough iOS local talent there? What’s the supply and demand like for iOS labor in Canada?

JP: Answering that in the present tense is really different than the landscape as it was in 2011-2013 when I was really involved. Back then, people were still riding the mobile wave as the hot new thing. Today you look at buzz words and they’ve shifted. Even though mobile is now a concrete part of a lot of businesses’ strategies, back in 2011 and 2012 people knew it was going to be big, it was already established at that point. The iPhone had been out for three or four years at that point.

Get more development news like this

It was already established, but companies still hadn’t fully caught up and created their own in-house teams. We were really at a good point, and got really lucky to ride the wave, honestly. That’s probably why the company survived at all.

If I was trying to do the same thing today, starting from scratch with no knowledge of the industry, it would be a lot harder to replicate that; at this point, a lot of the companies that used to hire agencies like Magnetic Bear Studios, or small agencies in general, now tend to have their own in-house group of developers and designers who can lead their mobile effort.

It’s not longer a fad, it’s no longer hype, it’s just like the web was five years ago. It’s expected that if you have some sort of electronic presence that you definitely need a mobile strategy of some sorts. The landscape has certainly shifted.

Consultancy Challenges (10:27)

Q: Right, that makes absolute sense. What was your key takeaway in terms of challenges of running an iOS consultancy during those three and a half years?

JP: Trying to put myself back in those shoes from a few years ago, which is becoming increasingly harder to do, I recall some of the challenges or at least some of the things that I used to complain about, at least internally. Even in 2011 and 2012, I thought I was too late to ride the hype and wave for mobile.

There were a handful of agencies that had popped up around the world, and I was finding myself competing with the more established smaller sized agencies that had done what I was doing. That was pretty tough, and I can only imagine someone trying to do the same thing today would have a much harder job of actually trying to pull that off. In retrospect, although I was doing that not at the very earliest time and maybe not at the perfect opportunistic time, the conditions were still really quite good to pull that off.

You look at the landscape now and not just in agencies but just in mobile development in general and the race to the bottom is really won. Very few people can make a living off of apps on the App Store and the strategies haves completely shifted towards either subscription based or in app purchases. Ultimately, we were already starting to see that trend three or four years ago, and that’s one of the reasons why I essentially grew a company as a contractor and as an agency rather than as a company developing apps directly for the App Store.

In a sense, a lot of these companies that were hiring us knew they needed a mobile presence, but didn’t know exactly how to monetize it and probably wouldn’t have been able to in a very good way anyway. Really the only way to bootstrap a company from the ground up was to get people who were willing to invest because end users of applications weren’t necessarily ready even at that point.

Starting at Realm (13:09)

Q: Tell us how you ended up at Realm and tell us a little bit about your role there.

JP: In 2013 and 2014, we were starting to see the limits of Magnetic Bear Studios and growing that as an agency. I was really lucky to have some very talented people, but ultimately, all the people in my team still had somewhat of an entrepreneurial spirit and building apps as an agency tends to wear you down. It’s not exactly for everyone, even though I do look back fondly on those days where we could have a wide variety of projects on an ongoing basis. It really kept you on your toes and kept you interested.

Ultimately, towards the end like I mentioned, I was doing more of the business side of things and I A) wasn’t very good at it and B) I didn’t enjoy it. I was looking to get more into a technical role and that’s where I saw that this stealth company in Silicon Valley was hiring. If you know me at all, especially in those days, those kinds of words were really a big turn off: “VC funded stealth company.” Back then, I was really all about bootstrapping a company, creating value right from day one rather than the promise of value for an investor.

Ultimately, the closer I looked at this company who didn’t even broadcast their name at that point, the more I appreciated that different types of companies and products require different types of business models and even bootstrapping processes. You look at a database that needs to be built from bottom up, and your minimum viable product there is a fully ACID compliant database. You can’t really just build half a database and release it out to the world.

If you’re going to disrupt a technological space like that that needs several years and a small team at least of upfront investment before you can actually start to monetize a product out of it, then venture capital or investment based strategies is really the only way unless you take more of a side project approach to building something like that. To be honest, I don’t really think that that would have worked with such a core fundamental technology as a database. I really appreciated and respected the approach that Realm took at the time.

I was chatting with the founders and at that point they had been building Realm’s underlying storage engine for the past three years. Starting summer 2011 and going through Y Combinator, that’s where they were really starting out. They had this concept of modernizing database engines, especially for embedded databases, in a way that really didn’t make sense 15 years ago when SQLite was developed. If anyone’s been in the mobile development industry here, even in the embedded space in the last decade and a half, they know that SQLite is a very robust type of software, but it didn’t always start off that way.

It needed to grow into that level of stability. Also, it was designed at a time when we didn’t have the smartphone revolution as we know it today. We didn’t exactly have iPhones or Android phones or even this concept of model oriented or object oriented, object relational mapping libraries. You look at Core Data, I mean you had web objects and stuff, but not so much in the embedded space.

You look at Core Data coming along and the genesis there really had the right ideas, where it was like, “Let’s build something that’s object oriented because that’s how we want people to write software.” Not this kind of table and row, kind of snapshot based system where you essentially query some external thing to ask what its state is and then you go and translate that. If you actually had a system like an ORM that can keep that in sync with the models and the objects that are really integral to object oriented programming, then you’re really burring the lines and at that point persistence and more database type concepts, like ACID compliance, then becomes more of an implementation detail.

You look at the progression of technology in the 90’s, the early 2000’s, even the late 2000’s and people really built amazing things on top of the fundamental technologies that they had. Then you get to 2010, 2011 and you really reach a tipping point where you’re starting to stretch the useful space in which those technologies designed ten years ago really start to apply and some of these concepts start to break down. You start to have arm based chips that are really prolific. You have SSD’s, you have vectorized operations and CPU’s, and these are all things that if you were designing a low level database today, you would really want to take advantage of those things.

It’s hard to take this older system that’s been battle tested, made very robust over the course of a long set of years, five, ten years, and then say, “Okay, well let’s re-architect this.” No matter what, at that point, anything you touch has an impact on the memory layout, has an impact on small things. That when you consider the scope of how wide it’s distribution is, you really start to realize that it’ll have very strong knockout effects. It’s very likely that it’ll break something somewhere.

You’re put in this impossible situation where you can’t innovate without breaking things, but people rely on these foundational technologies to be robust before anything else. Robustness over feature completeness, really. That’s where throwing out that concept, basically starting from scratch, really comes in handy. Where you can consider some of the research that was done in modern times about multi-version concurrency control, about copy on write semantics, about making the most of SSD’s and modern components, and modern computing architecture and really build something from scratch.

That’s what Realm did and that’s what ultimately drew me over to them. This promise of starting fresh. If we were to build something, a database for the smartphone revolution era. That’s what really got me in.

Working at Realm today (20:41)

Q: What’s your role at Realm today?

JP: My role today is leading the Apple software teams. We have two core products and these are user facing products. Those are Realm Objective-C and Realm Swift. I lead the team that’s responsible for building those, but ultimately the kinds of work that we do, that our team does, which is about seven or eight people at this point, spans everything from writing cross platform low level functionality, even at the Core Database level that can be used for other products, like Realm Java or Reactnative.net.

It spans from that to doing specifically Objective-C and Swift stuff, or even the Realm browser which is a Mac app that we have, to doing documentation tests, sample apps, tutorials, and community outreach as well. We don’t have any pure Realm evangelists in the company. We just have the engineers that are working on the product.

Why work at Realm? (22:03)

Q: Give us the elevator pitch for Realm. If I’m an iOS developer and I’m looking at my options in terms of data store, why should I pick Realm?

JP: The main thing that we strive for there is simplicity. Simplicity on so many levels. You look at a system, like a traditional ORM, and there’s so much intermediate state that needs to be maintained to keep the object side and the relational side in sync. That’s where you start to introduce faults, where you’re accessing something on one thread and another thread was mutating it, but because of the shared global mutable state that’s in common there, halfway through an operation you hit a fault and say, “Well, another thread deleted an object in a completely different context and I’m suffering the consequences there.”

Simplicity at that level. Everything from the most foundational layers up. Where if you lose this object relational mapping it’s just objects the whole way day. That simplicity translates into an easier way to reason about what’s happening under the hood. It translates to performance as well because you’re doing less work. It translates to safety because when the very lowest levels are built with that concurrency in mind, then you don’t need to have multiple layers where you need to ensure that that safety is preserved if it’s kept as simple as possible.

There’s simplicity on that end of the foundational technologies, and there’s also simplicity in terms of usage and the API. One of the things that we really strive for, more than feature completeness or even performance, is ease of use and maintainability. Initially when we released Realm Objective-C, which was our first public product, we had one idea of what the developer community was looking for and then they responded with something, not exactly completely different, but it did surprised us.

Initially we thought that speed was a very important factor and after talking to a lot of developers, a lot of them, if it’s fast enough, that’s all that matters. As long as they can have their UI table view that swirls at 60 frames a second, they’re happy. It doesn’t really matter if you can perform this complex object graph type query where you’re traversing many object graphs in a fraction of the time that it would take you with an ORM, that doesn’t matter as much for the vast majority of developers.

Another thing that we found is that a lot of the ping points like performance that people tend to experience with traditional solutions, people tend to internalize that. Where it’s part of normal development. You don’t exactly remember exactly how much effort it was to get things working within a reasonable time frame. We found out that performance as such wasn’t an end goal for most people, that it was actually more simplicity, maintainability, and the ability to have code that lasts for a long time, that doesn’t need to be constantly updated.

Core Data is complicated (25:41)

Q: That’s very interesting. A complaint of many new iOS devs is, “Why is Core Data so complicated?” Is it fair to say that Realm is competing with Core Data?

JP: Absolutely. It depends from what angle you look at it. Realm is definitely an object data layer and you can consider Core Data to be very similar in that sense. The types of things that you would do with Core Data, Realm is really a suitable replacement. There are design decisions that we’ve done to make sure that people migrating from Core Data to Realm have an easy path there. I’d say, especially in recent weeks, ever since Realm launched the Realm Mobile Platform, Core Data starts to really be more of kind of a subset in competition to Realm rather than an exact kind of competitor, if you will.

The Realm Mobile Platform, you can look at it as sync. It’s data synchronization between devices, between servers, but really if you peal back the layers, it’s a lot more than that. You look at apps today and a lot of the complexity goes into things like parsing JSON, doing network requests, doing caching, knowing when to pull, when to push, when to cache, when to invalidate a cache, what parts of that cache to invalidate, et cetera.

You look at all of that cruft and if there was a way to cut down on a lot of that complexity, app developers would really be able to focus a lot more on the value that they want to add in their app, on the kinds of functionality that really sets them apartment from the competition, compared to doing all this boiler plate stuff that has such a high investment cost and also so many ways in which things can go wrong.

You consider a traditional application that uses rest API’s and JSON serialization, caching layer, and you look at a round trip between keeping data that’s on the server, which is typically the canonical source of truth, in sync with at least what the application presents the user. There’s so much cruft that needs to go into that there if there was a way to cut down on that, you would really drastically simplify the way that people build apps today.

In a sense, Realm isn’t exactly competing with any one thing, we’re really trying to upset the status quo so that you can drastically simplify your apps, make them a lot more robust, while adding things like real time functionality with very minimal effort. That’s really kind of the approach that we’re taking.

Realm Performance Relative to Core Data (28:42)

Q: While performance is not your number one goal, simplicity is, how does Realm perform relative to Core Data? Are there any areas where Core Data outperforms it?

JP: There certainly are. There are fewer and fewer. Our mantra is to be within the same ball park as SQLite, for the most part. When it comes to benchmarks, one of the reasons why we’ve stopped producing benchmarks, at least externally, is because you can always make them look good for you, or for your use case. If you set the right size, the right transaction, or the right amount of concurrency competition that’s happening, you can make one product look much better than the other. We can tweak those numbers to make any product clip better than the rest of it’s competition.

We really do take performance very seriously and we have a number of performance tests that we will at least make sure that we don’t regress on a lot of key metrics or key operations. I’d really leave that to the end user to determine for themselves if for their use case Core Data, SQLite, or Realm is much faster than anything else. What I will say is that we really do strive to hit a sweet spot where we can be as performant, if not more than SQLite or Core Data, for the vast majority of types of architectures that mobile developers tend to use these days.

But I’d stop short from saying that any one product is considerably faster than anything else. You even look at Core Data and some of the things that it does is actually faster than Ross SQLite, depending on the type of operation you’re doing. I’d really leave that up to end users to determine for themselves.

Back end Realm usage (30:46)

Q: Realm is on several different platforms. Is Realm ever used as a web back end as well?

JP: It certainly is in several ways. One is in the same way as MySQL, or PostgreSQL, even SQLite can be used in server circumstances, where you need a transactional ACID database. Realm fits the bill there. There’s another angle to this, and this is probably the more common one, and that’s with the Realm Mobile Platform. Having a database on the server side that can communicate and coordinate with remote clients to facilitate things like synchronization and even things like event handling, that’s a much stronger use case.

That being said, Realm on the server is still in it’s early days, at least from a product standpoint. I wouldn’t say from a technological standpoint, but from a product standpoint, for sure. The only official language SDK that we provide really for the service is JS, is JavaScript. You look at the rest of our product line and you’ve got C sharp for .NET, you’re got Java for Android, you have Object C and Swift for Apple platforms, so that’s iOS, Mac OS, TV OS and Watch OS. None of those are designed to run say in a Linux infrastructure. The one product that we’ve specifically honed for that is the no JS SDK.

Q: The reason I asked the question is because Realm started as an embedded database for mobile devices. It seems like over time the database is spreading it’s wings and becoming a more general purpose database, is that a fair characterization?

JP: You could say that. Another way to put it is that Realm really started with the fundamental building blocks of building a very robust distributed database. That’s where the whole transaction log comes into play where we preserve the semantic intent of all the operations. This comes in nicely when you have distributed systems, especially when you need really high partitioning tolerance where devices can be offline at any time and you don’t want the performance of the database to take a hit for that.

If you’re in an intermittent connectivity area, which is basically all of mobile devices, all of mobile networks, because even as you’re walking through a building, even as you’re walking down the street switching between cell towers, you have intermittent connection, the last thing that you want is for your on device performance to suffer. Since the very beginning, since the genesis of the project, we’ve built everything in preparation for having a really robust distributed system. I’m really excited that we can finally offer that as a product now.

Realm Mobile Platform (34:13)

Q: Tell us more about that product. The big announcement from Realm in the last couple months was the Realm Mobile Platform. What is the Realm Mobile Platform?

JP: The Realm Mobile Platform is essentially exactly what you said. It’s taking Realm on device and making it spread it’s wings. The mobile platform lets you do things like device-to-device synchronization, it let’s you do things like real time collaboration, data back up and collaboration.

If you wanted to build something similar to Google Docs where you have multiple users that have different permission levels that are concurrently accessing a document making changes and having it all resolve automatically, those are really some of the main use cases that we’ve honed the Realm Mobile Platform to support.

There’s a number of other things that you can do. Earlier I was saying if you wanted to cut down on the complexity of your local stack, you remove the necessity to do JSON serialization, network requests, cache logic for invalidation and cache checking, if you wanted to do all of that, you could move a lot of the functionality from your app into the server and then at that point all the server really needs to do is to write objects to it’s local realm. Because of the way the Realm Mobile Platform works, that data is automatically available on the devices that have access to that realm.

Imagine you had an app that took a picture of a wine label and automatically gave you information about that wine bottle. What you could do is you could write a local object in your local embedded database, local embedded realm on your device that has the picture for that. Because of the way sync works, the server would then very quickly have access to that information, it could listen to changes that are happening, see, “Oh, this device just added a picture. Let me do all of the back end processing that I need to do, talk to several different servers, some of them maybe online, offline, some of them may have high latency,” whatever.

For the most part because it’s doing this work on a back end infrastructure, it typically has a much better internet connection so it can do a lot of this work and then once the response comes back from whatever this server that was listening to this change did, it can just write the response to it’s local realm object and then the device would then see that change locally in it’s local data store. You can really cut down on a lot of the complexity by sharing, say you have a cross platform app, you can share all of this business logic on the server where you don’t need to duplicate it and rewrite it in Java or Swift, or whatever, while still maintaining all of the functionality.

Q: Is it fair to say that the Realm Mobile Platform is in the same category as something like CloudKit, or what Parse was, or Firebase?

JP: There’s a lot of overlap when you mention products like those. There’s a lot of really key differences. You’re looking at CloudKit and for the bulk of what it offers, it really is for the most part a key value store. You have things like iCloud document syncing, but that’s really done more of on a very course grain document level. Then you look at something more like Firebase and it doesn’t really have much in terms of persistence or database type operations. You still need a solution for that locally on device.

Then you get into other differences, such as the merge algorithm for one, how do conflicting changes get resolved? This is something that Core Data with Core Data Sync tried to do with it’s ubiquity store. Ultimately there were a few architectural challenges with getting that to work where by building that at an ORM level rather than at a database level, you lose the transaction log of the database. You lose all of the fine grain operations that are done.

Whereas with Realm, if you actually build synchronization and merge conflict resolution from the very lowest levels up, you maintain that intent, that semantic intent of operation. When you have things like a move or a re-name or sub-string operations, if you were doing that at a high level without access to a transaction log, you have no idea if a change happened because of a move, because of a modification.

You can apply heuristics and try to guess what actually happened and what the intent of the user was, but as soon as you have a moderately complex use case, and you really don’t need to look very far to get that, that system starts to break down. That’s where you get things like incorrect merges. That’s really why Realm went and essentially reinvented the concept of an end-to-end database was to provide the fundamental tools to be able to do this efficiently and easily.

Q: Would it be fair to say that Realm is more robust than those other two solutions because it’s an end-to-end solution?

JP: I would certainly say so, but like you said, that’s generalizing certainly. Depending on your use case, you may not necessarily think so. I’m hard pressed to think of anything specific right now, but the last thing that I want to say is that this hammer that we’ve built, which is Realm, is a tool for all problems out there. The old adage, “When you have a hammer, everything looks like a nail.”

Sure you can do a lot with Realm. One of the reasons why we focused on a fundamental low level side of things was that it’s easier to build flexible solutions on top of robust primitives, rather than say have a really easy to use but super high level wrapper that’s a black box and that we don’t expose anything below the very surface. That’s what I say.

Q: Can developers install the Realm Mobile Platform back end server on their own servers or do they need to go through Realm?

JP: No, they actually have to install it on their own servers. We don’t offer a hosted solution right now. This is both a good thing and a bad thing because if you’re looking to own your data, if you’re looking to have full flexibility in terms of how it’s deployed, how it’s exposed, how it’s maintained, then the way that we currently offer the Realm Mobile Platform is really ideal for you.

This tends to be what most larges businesses tend to look for. That’s really who we’ve aimed to target from the very beginning. If you’re looking for a hosted solutions, we don’t offer anything right now and we are keeping our finger on the pulse in terms of how enterprises are using us in their own deployments to be able to figure out, “Okay, well is there a way that we can learn from this to maybe offer a fully hosted solution?” If that comes, it’ll come later.

Q: It sounds like you’re doing it in the right order. A lot of people felt very burned by what happened with Parse and that lead to a lot of consternation about having the data store itself be on someone else’s hardware. It sounds like you’re going about this in the right direction.

JP: We think so too, but obviously that’s more of a matter of opinion. If you’re a single person developer, managing your own infrastructure on top of your app might not necessarily be the answer that you’re looking for. We completely understand that, but on the other hand that’s exactly why we chose to fully open source the local embedded database. If you’re running Realm on your device, if you’re not using Sync, you’re using an entirely open source solution.

At that point there are no shackles, there is no lock in. You can do absolutely whatever you like. You can fork it and modify it to your heart’s content. We really hope that individual developers aren’t taken aback by our lack of hosted solution offering. On the other hand, it also helps us really focus on delivering the absolute best experience for the the people are willing to run this on their own infrastructure for now. We can then apply those lessons maybe later on when we’re ready to offer maybe a different type of product. The future will tell.

Q: Okay, so a bit of a cliff hanger. Maybe there will be a hosted solution in the future but that’s not yet determined, is that what we’re hearing?

JP: We’re not offering anything right now and we are focusing on self-hosted solution at this point.

Realm and the Open Source Community (43:57)

Q: Realm, the main database, is open source. Realm is very involved in the open source community. Tell us a little bit about that.

JP: One of the reasons why I wanted to join Realm in the first is really this ethos of contributing and being a participating member of the community. You’ll see a lot of companies that are in the developer tools space who might sponsor a lot of conferences, or they might have booths at trade shows and things like that. Although we do that from time to time, we aim to be a contributing member of the community. Not only does that help us to better keep in touch with our user base, but it also helps users because they can interact with us in a more fluid way.

That’s why we’ve been running the Swift Language User Group and hosting those videos on our site. Whenever possible with conferences, we’ll provide video recording, hosting, and transcription services for free, rather than just asking them to put our logo on a banner and giving them money for it. We want to be an active participant. That’s more on the community engagement side.

On the open source side, the high level SDKs have been open sourced since day one. Realm Objective-C, Realm Swift, Realm Java, JavaScript, .NET; all of that’s been open sourced. The core engine, however, was closed source for the first two years. There’s a handful of reasons there, but ultimately we want to be very transparent in terms of what our business model is and where Realm fits in, not just as a community member, but as a company.

You mentioned some other companies earlier and how people got burned. If you’re not fully clear about what your business model is, even though some people might not exactly be fully aligned with those priorities, then you’re really not being transparent. That really hurts people who are trying to build a business of off our products and our offerings. In that sense, keeping the core closed sources was an important message there, since we certainly did need to clean it up before it was available to the public, but also that’s where our intellectual property was. Being transparent about that was way more important than anything else.

Q: That makes complete sense. I want to also talk a little bit about some of your open source projects, like PeerKit and SourceKit. Tell us a little bit about them.

JP: PeerKit and SourceKit are really personal projects of mine. There are a handful of non-database related company projects that we have open sourced, like Jazzy, SwiftLint, and the Realm Browser.

Jazzy is an Objective-C and Swift documentation generator. That really was birthed out of a necessity for the company to have documentation for our own product.

Two years ago, at WWDC 2014, Swift was announced. This was funny timing because this was right around the time that we were gearing up to release Realm Objective-C. We said, “You know what, let’s just wait a week and see what happens at WWDC.”

Sure enough, Apple announced this new language and blindsided everyone. We essentially gave ourselves a week to support the language, and then released our product. Building a product that’s birthed into this new language, you need documentation for that language. That’s where Jazzy came about’ it was for our own business need to document our product, and it was a really fun project.

I loved hacking on that, reverse engineering Xcode, reverse engineering SourceKit, back when it was closed source and trying to figure out how to build a robust documentation generator. Another option would have been to build a parser for Swift, but if you look at Swift’s tumultuous history in the last two years, it would not have been fun at all.

Swift is a much more complex language than Objective-C, where in Objective-C you have very simple syntax. I’m not saying that it’s easier to learn at all, but from a parsing standpoint it’s a lot easier to par Objective-C than Swift. For one, you don’t have things like struts and language, sure there’s C++ struts and things like that. Another point is that you only ever have one level of nesting in Objective-C. You have classes and they can have properties, but you can’t have a class within class within property, or a closure within a function within a class.

Whereas with Swift, you have so much flexibility and so much complexity that follows that building a parser would have been a nightmare. The investment of reverse engineering Xcode and SourceKit seemed like a smarter choice and in retrospect I think it was.

There’s Jazzy and then there’s SwiftLint which is more of a linter for Swift that was really done more out of my own frustration that people hadn’t built one yet. I built SwiftLint in a weekend in May 2015 because I was giving a talk on fixing the Swift tooling problem at UIKonf in 2015. My point at that talk was really to say, “Sure, Swift is a new language and therefore has a lot of room for improvement, especially around tooling,” and also that, “We shouldn’t necessarily have to rely on Apple to build that tooling.”

I’ve been saying for a while that using something like SourceKit that you could pretty easily built a linter because Apple has gone through the efforts of exposing a way for people to build tooling for Swift. I’d been saying that for over a year at that point saying, “Hey, someone should really just built a linter on top of SourceKit.” As I was writing the talk, as anyone who’s given a talk before, you really want to procrastinate and do anything but. I ended up writing that on the plan just to show, to kind of make my point from the talk that, “Hey, we don’t have to wait for Apple to build tooling, we can do it ourselves.”

I built it then and it really exploded from there, mostly because no one else had really done it. Since then, there’s some terrific formatters and linters that have come out, but I think the community really latched on to SwiftLint and now it’s pretty actively used project.

Conclusion (52:04)

Q: Jazzy and SwiftLint are really fantastic projects and I recommend everyone listing check them out. Is there anything about Realm that we didn’t cover that you really want developers to know about it?

JP: That’s a good question. I think we really touched on a lot, especially around the Mobile Platform. This is really on us at Realm to do a better job of talking about this, but there are a handful of limitations in Realm that albeit annoying, are actually there for a very good reason. It’s not to say that they can’t be lifted, eventually, just that there’s a reason why they’re there.

One thing that comes to mind is thread confinement. Earlier you may have heard me say that Realm was a threadsafe database. What I mean by that is that transactions are isolated and any changes that are being done by one thread, or a different process, or even a different device that’s being synced, won’t effect your current consistent view of the database. We do this via a number of different ways: multi-version concurrency control, snapshots and copy-on-write.

One of the side effects of this is because Realm is an object graph database and that one thread may be accessing a different snapshot of the database than another, passing an object across a thread is actually very dangerous. What could happen is that as you’re passing this object, for a person that has a child property, a relationship or a link, what happens when the two different snapshots between those two threads have different values for that relationship? Or even different values between scalar primitive properties, like a String or an Integer?

You get into a situation where you don’t want to have a later snapshot try to access an object from an earlier snapshot. You get into this dependency resolution state in which it’s impossible to have a general solution. That’s why for safety we say, “Every object is confined to it’s own thread.” That’s not necessarily done out of our inability to let you access the object from another thread. It’s done so that people don’t accidentally write unsafe code that expects thing to be a certain way and then there data just changes from under them.

That being said, there are ways that we can improve ways to work around this, but you’re never going to be able to really pass data between threads or freely write data outside of a write transaction and that’s do to wanting transactional mutations. So if you need to change two things that if your app crashes half way through, then you don’t end up in this inconsistent state.

There’s a bunch of these limitations that people love to complain about and they are annoying (I’m the first to admit that), but the solution isn’t as simple as giving people the first solution that would come to mind. We’re working on solutions for a lot of these things and I really just appreciate everyone’s patience in understanding the complexity of the issues.

Q: I really recommend everyone check out Realm. How can our listeners get in touch with you, and where can they follow you?

JP: People can get in touch with Realm via Twitter, @realm or Realm.io on the interwebs. Email us [email protected] if you have any questions or issues. For me personally, I’m @SimJP on Twitter and then “JPSim” on everything else, like GitHub.

Q: JP, it’s been fantastic having you on the show. Thank you so much for taking the time, and I hope that all of our listeners learned a lot about Realm.

JP: Dave, thank you for having me, and I hope this was helpful!

Next Up: A Look Into Realm's Core DB Engine

General link arrow white

JP Simard

JP works at Realm on the Objective-C & Swift bindings, creator of jazzy (the documentation tool Apple forgot to release) and enjoys hacking on Swift tooling.

Edited by Billy Leet