// I need to

ModernizeUpgrade without breaking it RescueFix what someone else started Build NewTurn an idea into a product Reach More PeopleGet found, get used, include everyone

// Company

Case StudiesExpertiseAbout + CareersResourcesContact Book a call →

Retraction: Large Data Imports with AFIncrementalStore Are Painful

A retraction: the faster AFIncrementalStore large import technique from an earlier post broke Core Data relationships. What went wrong and what was learned.

Art+Logic Oct 9, 20133 min read

From the Art+Logic archive · written in 2013. The tools may have moved on; the thinking still holds.

Photo of neon sign by Franki Chamaki on Unsplash

Last time I wrote about doing imports of large data sets when using AFIncrementalStore to handle server synchronization I thought I had made an end-run around it, finding a technique which was compatible but faster. I was wrong.

Before I begin, this is not a knock on AFIncrementalStore which does its job well overall. It was my first use of this library, and I have no trouble recommending it for certain uses cases.

After my last post, when I started to test more thoroughly and use the imported data, I saw strange things. Mostly that meant my objects relationships were not what I expected.  I’m fairly sure I had seen these relationships created as they should be, but in the fog of the battle which was my work on this feature, I can’t say for certain. My bad.

A quick refresher on what I was doing in an effort to speed up a large first-time sync with the server. I requested the import data in batches and created NSManagedObjects from that JSON. In a second pass over the objects I set up the relationships. All that was left was to call save: on the managed object context. Easy and fast. Then I started using the data and saw, or rather didn’t see, the relationships I was expecting.

So what went wrong?

When NSManagedObjects are created Core Data gives them a temporary ID. Your object’s model might include a unique ID, but these are different. AFIncrementalStore cares about them (and it cares about your model’s unique ID). When the objects are saved they will be given a non-temporary ID.

While saving, AFIncrementalStore has the responsibility of transferring the objects to its private backing store. For each object saved, it updates its backing object with the new attributes and relationships. Except that with relationships, if the object being saved has a temporary ID it just ignores the relationship. Silently.

Core Data provides a method on NSManagedObjectContext to assign an object a permanent ID. This would make everything work nicely. Again, AFIncrementalStore is given the opportunity to manage permanent IDs. This part is more complicated, but it boils down to AFIncrementalStore doesn’t have the information it wants in order to assign the permanent ID, and it leaves the temporary ID as is. Silently, as is its way.

The extra information AFIncrementalStore wants is the unique ID as the server sees it. Since it didn’t make the server request, it doesn’t have that information. If it did, it would store that value in its special property it adds to your model, and stash some private information which doesn’t persist. There is no public method to tell it which unique property to use or what its value is.

I tried a few more things but ultimately I have settled on using batched NSFetchRequests which let AFIncrementalStore do what it does.

This was a tough battle, and hopefully you’ll be spared the trouble. In the end I wonder if Core Data is really the right choice when dealing with a few 50,000+ row tables of data. Some think not. I am sure that doing straight calls to a database or some more direct framework would have given the performance I was seeking. But I had to weigh that against the benefit I was getting from Core Data and AFIncrementalStore in other areas, and I chose to keep using it.

keep going

More to explore in Build New.

All Build New resources →
let's talk

Got a problem like this one?

Tell us what's stuck. A real person replies within one business day.

Talk to our team →

Talk to people who know software, not a sales script.

Your first call is with Aja, our VP of Sales. He'll help make sure you're solving the right problem. Here's what you'll cover together:

  1. What you're trying to accomplish

    The business goal first, then the technology.

  2. Possible approaches

    And the tradeoffs of each, in plain language.

  3. Risks you may not have considered

    Technical and business, before they cost you.

  4. Whether AI changes the math

    What's practical, and what's just hype for your project.

  5. Sensible next steps

    Even if the right next step is not us.

book-a-call.ics

Loading the scheduler…

Open the booking page →

Some problems are easier to explain in writing. Send us a few lines about what you're working on, and we'll reply within one business day. Helpful to include:

  1. What the software does, or should do
  2. What's stuck, or what's worrying you
  3. Any deadline or budget you have in mind
note.md

Loading the form…