Search Posts in my Blog

Monday 16 April 2012

Core Data Tutorial: How To Preload/Import Existing Data


This is the second part of a three-part series to help get you up to speed with the basics of Core Data quickly.
In the first part of the series, we created a visual data model for our objects, ran a quick and dirty test to make sure it works, and hooked it up to a table view so we could see a list of our objects.
In this part of the series, we’re going to discuss how to import or preload existing data into Core Data so that we have some good default data when our app starts up.
In this final part of the series, we’re going to discuss how we can optimize our app by using NSFetchedResultsController, to reduce memory overhead and improve response time.

Preloading / Importing Existing Data

So how do we preload data into a Core Data store, anyway? Well, there are two popular solutions to this problem:
  • Fill in Core Data on startup from external source. For this the app can start up, notice that the database hasn’t been imported yet, and start reading in data from an external source (such as an SQLite database or XML file) and then start inserting the data into Core Data.
  • Provide pre-filled in SQLite database. For this we’ll let Core Data create the database structure for us based on the model, and then we populate the database with a utility app. The utility app could be a Mac or iPhone app that uses Core Data to populate the database via Core Data APIs, or some kind of program that fills in the SQLite database directly. Once the database is populated, just include it with the app and make the app use it as the default database if no database already exists.
We’re going to go with option 2 because it’s simple and more efficient. To populate the database, we’ll just extend our Python script a bit since we have that working already.
Note that by using a Python script to import the data rather than working with a utility app that uses Core Data APIs, it’s more likely to break in the future because we’re kind of going under the hood here… but for this tutorial I thought a) it’s a better learning experience since we just covered SQLite and it shows how things are working more clearly, and b) it is simpler!
So, let’s grab a copy of the sqlite database that was generated by our project. The easiest way to find the file is to set a breakpoint inside the application delegate, inside the persistentStoreCoordinator function, below the storeUrl line. You can examine the storeUrl variable to see a full path to where the sqlite backing file resides. So find this and copy it to the directory your Python script is in.
Once you have the database, use sqlite3 to take a peek at how the database looks:
sqlite3 FailedBanksCD.sqlite

sqlite3> .schema
CREATE TABLE ZFAILEDBANKDETAILS ( Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER,
    Z_OPT INTEGER, ZZIP INTEGER, ZINFO INTEGER, ZUPDATEDDATE TIMESTAMP,
    ZCLOSEDATE TIMESTAMP );
CREATE TABLE ZFAILEDBANKINFO ( Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER,
    Z_OPT INTEGER, ZDETAILS INTEGER, ZNAME VARCHAR, ZSTATE VARCHAR,
    ZCITY VARCHAR );
CREATE TABLE Z_METADATA (Z_VERSION INTEGER PRIMARY KEY,
    Z_UUID VARCHAR(255), Z_PLIST BLOB);
CREATE TABLE Z_PRIMARYKEY (Z_ENT INTEGER PRIMARY KEY, Z_NAME VARCHAR,
    Z_SUPER INTEGER, Z_MAX INTEGER);
CREATE INDEX ZFAILEDBANKDETAILS_ZINFO_INDEX ON ZFAILEDBANKDETAILS (ZINFO);
CREATE INDEX ZFAILEDBANKINFO_ZDETAILS_INDEX ON ZFAILEDBANKINFO (ZDETAILS);

sqlite> select * from ZFAILEDBANKINFO;
1|2|1|1|Test Bank|Testland|Testville
2|2|1|2|Test Bank|Testland|Testville
3|2|1|3|Test Bank|Testland|Testville

sqlite> select * from ZFAILEDBANKDETAILS;
1|1|1|12345|1|292794835.855615|292794835.679693
2|1|1|12345|2|292794875.943392|292794875.768675
3|1|1|12345|3|292795809.375025|292795809.215297

sqlite> select * from Z_PRIMARYKEY;
1|FailedBankDetails|0|3
2|FailedBankInfo|0|3
Here’s a quick description of what’s in here. Z_METADATA contains some information about the Model that Core Data needs to function. Z_PRIMARYKEY contains (among other things) information on the max key that is currently used by each entity.
As for ZFAILEDBANKINFO and ZFAILEDBANKDETAILS, those are our main data tables! Z_PK is the unique id for each, Z_ENT is their entity id (same as what’s listed in the Z_PRIMARYKEY table), and finally there are our normal fields.
Now, let’s create a Python script to populate this database by reading in the contents of our old database, and creating the appropriate rows in the new database. Create a Python script like the following:
import sqlite3;
from datetime import datetime, date;
import time
 
inConn = sqlite3.connect('banklist.sqlite3')
outConn = sqlite3.connect('FailedBanksCD.sqlite')
 
inCursor = inConn.cursor()
outCursor = outConn.cursor()
 
outConn.execute("DELETE FROM ZFAILEDBANKINFO")
outConn.execute("DELETE FROM ZFAILEDBANKDETAILS")
 
maxId = 0
inCursor.execute("select * from failed_banks")
for row in inCursor:
    closeDate = datetime.strptime(row[5], "%Y-%m-%d %H:%M:%S")
    updatedDate = datetime.strptime(row[6], "%Y-%m-%d %H:%M:%S")
    closeDateSecs = time.mktime(closeDate.timetuple())
    updatedDateSecs = time.mktime(updatedDate.timetuple())
 
    # Convert time references secs to NSDate reference
    deltaSecs = time.mktime((2001, 1, 1, 0, 0, 0, 0, 0, 0))
    closeDateSecs = closeDateSecs - deltaSecs
    updatedDateSecs = updatedDateSecs - deltaSecs
 
    if row[0] > maxId:
        maxId = row[0]
 
    # Create ZFAILEDBANKINFO entry
    vals = []
    vals.append(row[0]) # Z_PK
    vals.append(2) # Z_ENT
    vals.append(1) # Z_OPT
    vals.append(row[0]) # ZDETAILS
    vals.append(row[1]) # ZNAME
    vals.append(row[3]) # ZSTATE
    vals.append(row[2]) # ZCITY
    outConn.execute("insert into ZFAILEDBANKINFO values(?, ?, ?, ?, ?, ?, ?)", vals)
 
    # Create ZFAILEDBANKDETAILS entry
    vals = []
    vals.append(row[0]) # Z_PK
    vals.append(1) # Z_ENT
    vals.append(1) # Z_OPT
    vals.append(row[4]) # ZZIP
    vals.append(row[0]) # ZINFO
    vals.append(closeDateSecs) # ZUPDATEDATE
    vals.append(updatedDateSecs) # ZCLOSEDATE
    outConn.execute("insert into ZFAILEDBANKDETAILS values(?, ?, ?, ?, ?, ?, ?)", vals)
 
outConn.execute("update Z_PRIMARYKEY set Z_MAX=?", [maxId])
 
outConn.commit()
This should be pretty straightforward. The only tricky thing we needed to do here was to convert the dates from the format they were stored in our old database (a string) to the format they are stored in the new databaes (epoch seconds). 
Give the Python script a whirl and if all works well you should be able to see your DB populated with data:
python coreData.py

sqlite3 FailedBanksCD.sqlite

sqlite> select * from ZFAILEDBANKINFO limit 3;
1|2|1|1|Desert Hills Bank|AZ|Phoenix
2|2|1|2|Unity National Bank|GA|Cartersville
3|2|1|3|Key West Bank|FL|Key West

sqlite> select * from ZFAILEDBANKDETAILS limit 3;
1|1|1|57060|1|1269576000|1269576000
2|1|1|34678|2|1269576000|1269576000
3|1|1|34684|3|1269576000|1269576000
Once you’re satisfied the data is in the database correctly, drag the populated database file into your XCode project in the Resources folder. Then open up FailedBanksCDAppDelegate.m, navigate to the persistentStoreCoordinator function, and replace the storeUrl line with the following:
NSString *storePath = [[self applicationDocumentsDirectory] 
    stringByAppendingPathComponent: @"FailedBanksCD.sqlite"];
NSURL *storeUrl = [NSURL fileURLWithPath:storePath];
 
// Put down default db if it doesn't already exist
NSFileManager *fileManager = [NSFileManager defaultManager];
if (![fileManager fileExistsAtPath:storePath]) {
    NSString *defaultStorePath = [[NSBundle mainBundle] 
        pathForResource:@"FailedBanksCD" ofType:@"sqlite"];
    if (defaultStorePath) {
        [fileManager copyItemAtPath:defaultStorePath toPath:storePath error:NULL];
    }
}
All this does is check to see if the sqlite file already exists in the documents directory, and if it doesn’t (i.e. it’s the first time the app has run) it copies the included database from the bundle into the documents directory. Then Core Data will start using the preloaded data and we’re good to go!
Delete the old SQLITE3 file from the iPhone simulator directory (or click iPhone Simulator\Reset Contents and Settings to clear everything), and re-run your app. If all works, you should now see the prepopulated list of banks!
Failed Banks App With Core Data

So next in this tutorial series we’ll cover 

Core Data Tutorial: How To Use NSFetchedResultsController

No comments:

Post a Comment