When I moved the default CSS rules into a separate file I was facing the old dilemma: how can I embed this in the static library but still be able to easily add contents to it via Xcode. I previously explained how you can turn any file into a c-Byte-array. But this still required manual work.

I did a bit of researching and found that on regular Linux systems people seem to have a tool named objcopy which can copy files as their are into an object file (.o) which can be linked together to the final binary by the linker. But unfortunately this tool does not come with Xcode. So it is out of the question because I want everybody to be able to build DTCoreText.

Xcode Build Rules come to the rescue. They can automate any kind of preprocessing you like and it turns out they are an easy solution for this very problem.

There are essentially three kinds of source file that Xcode knows how to deal with (greatly simplified):

• c, c++ or Objective-C source code, or any code that can be somehow compiled
• header files
• other files

“Other files”, if you add them to a target, are treated as resources that simply get copied into your app bundle.

And now I’ll teach you how you can make any kind of file compilable. Be it some textures that you are compressing for building your game, or be it some resource that you want to embed into your binary.

Make a Build Rule

When Xcode is asked to compile a file it has a set of rules how it does that. So what we are going to do is to add our own rule how a css file is turned into a .c file. And for .c files are something that Xcode has a built-in rule for. So it can happen that you have two or more daisy chained rules that hand off their results to each other.

It is somewhat inconvenient that you cannot define rules globally. Instead you have to repeat your custom rules for each target.

To add your own rule, go into the projects info where you see your targets and on the target you want the rule for add it like shown.

This rule matches all files that end with .css (asterisk is a wildcard) and executes a custom script:

cd "$INPUT_FILE_DIR"  # move into file dir, otherwise xxd takes the full path for the symbol
/usr/bin/xxd -i "$INPUT_FILE_NAME" "$DERIVED_SOURCES_DIR/$INPUT_FILE_BASE.css.c" # builds a c file with a hex array

I’m moving into the folder of the input file first because xxd will always take the full passed file path and turn it into the name of the c-array. This way it will just be named default_c and the length will be in default_c_len.

The above rule is all it takes to teach Xcode how to turn a .css file into a .c file. Now the next step is to make sure that we are actually compiling the default.css file as opposed to copying it together with the other resources.

Compile not Copy

You can see that default.css is compiled by having it in the Compile Sources section of the Build Phases.

If you build the project you see two hints about what happens now. Towards the top of the build log you see the preprocessing of the file. A little further down you see how the intermediate c file is being compiled.

Further down the library tool puts all object files into the library archive. If you don’t believe me, you can employ the nm tool to inspect the symbols in the final product.

nm DemoApp | grep default_c
00045474 D _default_css
00045fe0 D _default_css_len

These are the two symbols from the default.css.c file that made it into the final product.

This guide wouldn’t be complete if I didn’t show you how to get to these symbols.

Accessing the Objectified File

I told you above that xxd uses the passed file name to make two variables, one for the c-style array, one for the length. Only change is that it has to turn the file name characters into a legal symbol, so dots are turned into underscores.

In DTCSSStylesheet.m I am accessing this array like so. I define these two variables as external which prompts the linker to insert their correct address.

+ (DTCSSStylesheet *)defaultStyleSheet
// external symbols generated via custom build rule and xxd
extern unsigned char default_css[];
extern unsigned int default_css_len;
}

And to get the string out of them is just as easy:

+ (DTCSSStylesheet *)defaultStyleSheet
{
   // get the data from the external symbol
   NSData *data = [NSData dataWithBytes:default_css length:default_css_len];
   NSString *cssString = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding];
 
   return [[DTCSSStylesheet alloc] initWithStyleBlock:cssString];
}

This works because I know that all my source files are UTF8 encoded, i.e. one byte per (normal) character. It is quite handy to have the default_css_len integer telling us the length of the array, because in c an array is just a pointer to the first byte.

Conclusion

If you know how to employ custom build rules to your advantage many tasks that you would have previously run external scripts for now become part of the build process. What you can do with this knowledge is only limited by your imagination … and your scripting skills.

But the general advantage of build rules versus external programs is that they are part of the project. So if they use standard commands they are portable to other developer’s machines. Perfect for Open Source projects.

In this post I will to demonstrate how you can contribute to an Open Source project hosted on GitHub. I previously blogged about how you can make and apply patches, which is certainly one way. But on GitHub there is an even cooler method, one that is also less work on the project maintainer.

For the sake of this tutorial I shall pretend that there is an issue with a project by Matt Galloway, then I’ll fork his project. After fixing the issue I will push my changes online to my fork and finally notify Matt about these changes via a so-called “Pull Request”.

A prerequisite for this tutorial is to have your GitHub account and local git set up previously.

Step 1 – We have an Issue

Your first thought should be to document that in GitHub’s excellent bug tracking system. Each Open Source project there has a list of Issues (which are the bugs and feature requests). I found two things to improve in the project and so I added two issues.

We could hope for the project’s owner to fix that for us, but – being smart developers – we can also proceed to fix the issues ourselves. From the get go we won’t have writing access to the GitHub repository (unless the owner gives it to us, which is rare). You can see that this is the case if you only see the HTTP and Git Read-Only options on the repo URL:

The usual method is to make a copy of this project in our own area on GitHub where we CAN write to. This copy is what they call a “Fork”.

Step 2 – Stick a Fork in it

To make a fork you klick the “Fork” button at the top right of the project main page. After some “hardcore forking action” you end up on your own copy of the project. And there is now an extra option “SSH” telling you that you have read and write access to that.

A fork is a complete copy of the original project including all previous history. The reason for this that with Git every copy of a repository is always a full clone, so that each clone can also serve as a backup. This illustrates the distributed nature of Git, as opposed to centralized Source Code Management (SCM) systems like Subversion which has a central server.

Step 3 – Clone Wars

To work on our copy of the project we need to clone it to our hard disk. You can use the excellent official GitHub app but personally I prefer to work in terminal. I open terminal, go to a place where I want the copy of the project to be and issue the command:

git clone git@github.com:Cocoanetics/ios-library-with-resources.git

The URL used is the SSH one I copied from the web page. This results in a ios-library-with-resources folder appearing in the current working directory.

Step 4 – Fix-Tours

Now we can do our changes. So I open the Xcode project.

The first thing I am fixing for this example is to add the two sub-project products as dependency so that they get automatically built if somebody builds the containing app.

You see an M (for Modified) appear next to the root of the project tree which means that the project file has been modified. Each such fix should become one commit. So we right-click on MyLibraryTest, Source Control, Commit Selected Files. This opens a dialog that shows which files were modified and prompts for a commit message.

Since we fixed issue number #001 with that we also add this tag, because later we will see that GitHub conveniently links this commit with the issue via this number.

This committing basically saves the current state in your local Git repository (your clone) and adds your message to that. Actually it did a bit more than that, because Git has a an extra step called “staging” before committing.

In staging you select the modifications you want to be part of the commit by Adding them to the staging area. The second part then does this saving. If you do this via the Xcode interface you don’t need this staging phase because you specifically selected the files to commit already via the project navigator.

The second issue to fix is to change a build setting on the sub-project.

This time the M appears next to the MyLibrary.xcodeproj which really means the project.pbxproj file contained in the MyLibrary.xcodeproj bundle. (Remember that a bundle is a folder that looks like a file)

This time let’s do the committing on the command like, like a real pro. First let’s see the status.

git status
# On branch master
# Your branch is ahead of 'origin/master' by 1 commit.
#
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#	modified:   MyLibrary/MyLibrary.xcodeproj/project.pbxproj
#
no changes added to commit (use "git add" and/or "git commit -a")

This shows that we are ahead of our fork (origin/master) by 1 commit, which is the previous one we did. And it shows that the project file for the sub-project was modified. We issue a “git commit -a” which stages all modified files for the commit and immediately prompts us for a commit message in the currently set up text editor.

There is a second shortcut that GitHub recognizes besides the #002 tag. If you write “closes” or “fixes” in front of the hashtag it not only references the issue, but also marks it as closed. How convenient!

“Set Skip Install YES for resource bundle to prevent copying that into Archive. closes #002″

I type my message into the VIM that opened and save/exit via ESC, :wq, ENTER.

Now we have two local commits (git log to see them), next we need to push them to our online repository. We just had modifications to the project file in these two examples, but – trust me – it works just as well for code changes.

Step 5 – Push Up

When we cloned the project to our local hard disk Git was nice enough to automatically configure the location where it came from as so called “Remote”, the default name is “origin”. You can have multiple remote places set up and choose which to push.

The command for us is:

git push origin master

“origin” is the remote I told you about, “master” means that you are pushing this to the master branch, which is the default branch.

To check if our changes have really arrived online I like to check the commits view on GitHub. And indeed they have.

We could end here and many people do. They just make a copy of an Open Source project for the case that the original repository disappears. But since we are playing nice we want to offer our improvements to the original project’s owner.

Step 6 – I just called to say “Please Pull Me!”

Just like for the fork we find a “Pull Request” button at the top right of the GitHub website of our fork. Push it and you get a nice overview of what is contained in the request.

There are some advanced options regarding milestones and assignments, but we’ll ignore these for now. You’ll notice that the pull request automatically generates a new number, #3 in our case. So you can not only reference issues with your commit messages, but the same thing is possible for pull requests.

The final step is not for us, but for the original project’s owner to perform.

Step 7 – Accepted!

As owner of a GitHub repo it is up to you whether you want to accept the changes into your repository. If you kept your commits small and – for code changes – added some good comments to aid understanding then the owner will be happy to accept them. It saves him from doing the work that you did and all he needs is to push the green merge button.

The button to accept the changes is only green if the merge of the code can be performed automatically. If the author did some changes on his own that conflict with yours then the button is gray stating that the changes cannot be merged automatically. In this unfortunate situation either one of you needs to pull the other’s changes and manually resolve the merging conflicts. But this is a topic for another day.

Conclusion

I suggest you find yourself a a GitHub project to contribute to and try out this process to shed the timid feeling. It really is quite idiot proof.

People put their projects on GitHub not only so that you get free source code to use in your apps. They also hope that other developers might find and fix bugs. And every once in a while you see a pull request that makes you think “OMG I’m not worthy!” because those are the gems that really teach you something.

And this is why they call it collaborative coding.

The parsing of HTML necessary for my DTCoreText open source project is done entirely with the sophisticated use of NSScanner. But it has been long on my list to rewrite all of this parsing with a the industry standard libxml2 which comes preinstalled on all iOS devices. Not only is this potentially much faster in dealing with large chunks of HTML. It probably is also more intelligent in correcting structural errors of the HTML code you throw at it.

In part 1 of this series I showed you how to add the libxml2 library to your project and explained the basic concepts of using libxml2 for parsing any odd HTML snippet into a DOM. In this here part 2 we will create an Objective-C based wrapper around libml2 so that we can use it just like NSXMLParser, only for HTML.

There are basically two methods – approaches – to handling XML/HTML: DOM or SAX. DOM (short for Document Object Model) refers to building a tree of nodes and leaves which you then can query for information based on their path (with XPath). SAX (short for Simple API for XML) basically walks through the the document and sends you events as it discovers things. For example the begin of an element, the end of it, when a comment was found, etc.

NSXMLParser is a SAX parser for well-formed XML. That means you can use it only on XML, it cannot deal with the nature of HTML where some elements might not require that you close them. But if you look at the delegate protocol and compare it to the SAX interface of libxml2 you find an eerie resemblance. It might as well be that NSXMLParser is just a wrapper around the XML SAX interface of libxml2.

As we have previously learned we know that libxml2 also possesses a HTML parser which reuses most of the same internal data structures as its XML cousin. So we shall endeavor to create DTHTMLParser as a SAX parser for HTML.

Don’t worry about piecing together all the source code from this article, I will put that online in my DTFoundation project. Instead I want you to follow my lead and understand the pieces.

The libxml2 SAX Interface

The SAX interface works such that you register a handler function – in C – for each SAX event that you are interested in. This is done via a C-structure of type htmlSAXHandler which is nothing more than an aggregate of multiple C-function pointers.

In the libxml headers you often see some “html…” data structure typedef’d into one with “xml” as prefix. The reason for this is that it reuses whatever it can from the XML parser. But this also adds confusion – as I found through my experimenting – because some of the events that make sense for parsing XML don’t fire for HTML, like for example the one that reports CDATA elements. HTML does not have those, but the function pointer is still present in the htmlSAXHandler struct.

By CMD+Clicking on a type we can swim upstream to find the original definition of htmlSAXHandler:

htmlSAXHandler = xmlSAXHandler = _xmlSAXHandler =

struct _xmlSAXHandler {
    internalSubsetSAXFunc internalSubset;
    isStandaloneSAXFunc isStandalone;
    hasInternalSubsetSAXFunc hasInternalSubset;
    hasExternalSubsetSAXFunc hasExternalSubset;
    resolveEntitySAXFunc resolveEntity;
    getEntitySAXFunc getEntity;
    entityDeclSAXFunc entityDecl;
    notationDeclSAXFunc notationDecl;
    attributeDeclSAXFunc attributeDecl;
    elementDeclSAXFunc elementDecl;
    unparsedEntityDeclSAXFunc unparsedEntityDecl;
    setDocumentLocatorSAXFunc setDocumentLocator;
    startDocumentSAXFunc startDocument;
    endDocumentSAXFunc endDocument;
    startElementSAXFunc startElement;
    endElementSAXFunc endElement;
    referenceSAXFunc reference;
    charactersSAXFunc characters;
    ignorableWhitespaceSAXFunc ignorableWhitespace;
    processingInstructionSAXFunc processingInstruction;
    commentSAXFunc comment;
    warningSAXFunc warning;
    errorSAXFunc error;
    fatalErrorSAXFunc fatalError; /* unused error() get all the errors */
    getParameterEntitySAXFunc getParameterEntity;
    cdataBlockSAXFunc cdataBlock;
    externalSubsetSAXFunc externalSubset;
    unsigned int initialized;
    /* The following fields are extensions available only on version 2 */
    void *_private;
    startElementNsSAX2Func startElementNs;
    endElementNsSAX2Func endElementNs;
    xmlStructuredErrorFunc serror;
};

All these function pointer types are aptly named SomethingFunc and we’ll get to how these need to look like in a minute. The problem for us to solve how to bridge from our Objective-C paradigm (using self, properties, IVARs) into the C-function paradigm (no access to our parser object).

This problem is compounded by the fact that we are building this with ARC so that we don’t have to deal with memory management any more. Our resulting class will run on iOS 4 and above. Somehow we need to pass enough information to each event-function so that we can communicate with the parser object.

There are two reasonable possibilities: either we pass a reference to the parser instance or we pass a pointer to a struct that contains multiple values that we might need. I decided for the first because it seems cleaner to me.

libxml2 provides a concept of a user_data pointer that is passed to the event-functions together with other relevant information. We can use (abuse?) this for passing a reference to self. ARC requires this to be be cast to (__bridge void *) which in plain English means: “please treat this as just some typeless memory pointer and also don’t worry about retaining or releasing. I’ll take care of that, don’t panic.”

Like any self-respecting class we’ll start out with a good block of IVAR definitions and and init method.

@implementation DTHTMLParser
{
    htmlSAXHandler _handler;
 
    NSStringEncoding _encoding;
    NSData *_data;
 
    __unsafe_unretained id <DTHTMLParserDelegate> _delegate;
    htmlParserCtxtPtr _parserContext;
}

Notice that I’m defining the IVARs in the implementation, not the interface (in the header). This is possible as of using the new Apple LLVM compiler and provides for a nice way to hide the IVARs that we don’t want to be visible.

- (id)initWithData:(NSData *)data encoding:(NSStringEncoding)encoding
{
	self = [super init];
	if (self)
	{
		_data = data;
		_encoding = encoding;
 
		xmlSAX2InitHtmlDefaultSAXHandler(&_handler);
	}
 
	return self;
}

We save the data and the encoding into our IVARs, _data being a strong reference to it. That means that it retains it during the lifetime of our parser and will only release it when the parser goes away.

We also initialize the _handler structure via the xmlSAX2InitHtmlDefaultSAXHandler function. This sets up some initial internals and also chooses the SAX2 method for our work. There’s also an older SAX1 method, but we obviously prefer the newer one. This setup has to happen inside the init method because right after such an instance of a DTHTMLParser is create we want to be able to set the delegate which should take care of wiring up the needed functions.

Each event-function should have a corresponding delegate method in our DTHTMLParserDelegate protocol, but to avoid unnecessary function-calling we only add the functions to the handler for which the delegate actually implements the function.

#pragma mark Properties
 
- (void)setDelegate:(__unsafe_unretained id<DTHTMLParserDelegate>)delegate;
{
	if (delegate != _delegate)
	{
		_delegate = delegate;
 
		if ([_delegate respondsToSelector:@selector(parserDidStartDocument:)])
		{
			_handler.startDocument = _startDocument;
		}
		else
		{
			_handler.startDocument = NULL;
		}
 
		if ([_delegate respondsToSelector:@selector(parserDidEndDocument:)])
		{
			_handler.endDocument = _endDocument;
		}
		else
		{
			_handler.endDocument = NULL;
		}
// ...

This way when you set the delegate it is inspected one by one whether it responds to certain methods of the protocol. And whenever it does the corresponding function pointer is set in the _handler IVAR.

At the top of the file we need to define these functions. If we do just that then the compiler will warn us for all of these implementations that there was “no previous prototype”. In C a you call the first line of a C-function it’s prototype. So we just have an extra block for these.

#pragma mark Event function prototypes
 
void _startDocument(void *context);
void _endDocument(void *context);
void _startElement(void *context, const xmlChar *name,const xmlChar **atts);
void _endElement(void *context, const xmlChar *name);
void _characters(void *context, const xmlChar *ch, int len);
void _comment(void *context, const xmlChar *value);
void _error(void *context, const char *msg, ...);

Wherever you see a context parameter this will be the user_data pointer we will provide. And for example the _startElement function gets a name and an array of attributes as well. The xmlChar type is just a char. This is actually always provided as UTF8-encoded string regardless of what the data is encoded as.

Let’s look first at an easy event-function and then at a more complex one.

void _startDocument(void *context)
{
    DTHTMLParser *myself = (__bridge DTHTMLParser *)context;
 
    [myself.delegate parserDidStartDocument:myself];
}

You see how I cast the void pointer back into a pointer of type DTHTMLParser. I have to name the variable different than “self” because that is a reserved word. Then I can simply call the delegate method that corresponds to this event function.

By the way, these are the delegate methods we want to implement:

@class DTHTMLParser;
 
@protocol DTHTMLParserDelegate <NSObject>
 
@optional
- (void)parserDidStartDocument:(DTHTMLParser *)parser;
- (void)parserDidEndDocument:(DTHTMLParser *)parser;
- (void)parser:(DTHTMLParser *)parser didStartElement:(NSString *)elementName attributes:(NSDictionary *)attributeDict;
- (void)parser:(DTHTMLParser *)parser didEndElement:(NSString *)elementName;
- (void)parser:(DTHTMLParser *)parser foundCharacters:(NSString *)string;
- (void)parser:(DTHTMLParser *)parser foundComment:(NSString *)comment;
- (void)parser:(DTHTMLParser *)parser parseErrorOccurred:(NSError *)parseError;
 
@end

The @class is necessary before the protocol definition because a good delegate protocol also passes a reference to the caller. This also enables the casual reader of your code to know at a glance that this method belongs to a protocol connected to this class.

Now for something more complex, let’s look at didStartElement. The complexity of this method results from the arts parameter being alternating key, value, key, value etc.

void _startElement(void *context, const xmlChar *name,const xmlChar **atts)
{
	DTHTMLParser *myself = (__bridge DTHTMLParser *)context;
 
	NSString *nameStr = [NSString stringWithUTF8String:(char *)name];
 
	NSMutableDictionary *attributes = nil;
 
	if (atts)
	{
		NSString *key = nil;
		NSString *value = nil;
 
		attributes = [[NSMutableDictionary alloc] init];
 
		int i=0;
		while (1)
		{
			char *att = (char *)atts[i++];
 
			if (!key)
			{
				if (!att)
				{
					// we're done
					break;
				}
 
				key = [NSString stringWithUTF8String:att];
			}
			else
			{
				if (att)
				{
					value = [NSString stringWithUTF8String:att];
				}
				else
				{
					// solo attribute
					value = key;
				}
 
				[attributes setObject:value forKey:key];
 
				value = nil;
				key = nil;
			}
		}
	}
 
	[myself.delegate parser:myself didStartElement:nameStr attributes:attributes];
}

The while loop walks through the attributes and and when a NULL is encountered where a key would be then it breaks out of the loop. It is also necessary to deal with a situation where there is only an attribute name, but no value because HTML also allows that. Theoretically one attribute could also be occurring multiple times, but we ignore this. For later convenience we want the key/values for the attributes in an NSDictionary.

The final piece is how the parsing actually is started. For this we have a parse method, just like NSXMLParser.

- (BOOL)parse
{
	void *dataBytes = (char *)[_data bytes];
	int dataSize = [_data length];
 
	// detect encoding if necessary
	xmlCharEncoding charEnc = 0;
 
	if (!_encoding)
	{
		charEnc = xmlDetectCharEncoding(dataBytes, dataSize);
	}
	else
	{
		// convert the encoding
		// TODO: proper mapping from _encoding to xmlCharEncoding
		CFStringEncoding cfenc = CFStringConvertNSStringEncodingToEncoding(_encoding);
		CFStringRef cfencstr = CFStringConvertEncodingToIANACharSetName(cfenc);
		const char *enc = CFStringGetCStringPtr(cfencstr, 0);
 
		charEnc = xmlParseCharEncoding(enc);
	}
 
	// create a parse context
	_parserContext = htmlCreatePushParserCtxt(&_handler, (__bridge void *)self, 
		dataBytes, dataSize, NULL, charEnc);
 
	// set some options
	htmlCtxtUseOptions(_parserContext, HTML_PARSE_RECOVER | HTML_PARSE_NONET |
		HTML_PARSE_COMPACT);
 
	// parse!
	int result = htmlParseDocument(_parserContext);
 
	return (result==0);
}

There is a xmlDetectCharEncoding function in libxml to try and detect the encoding of the data which we invoke if the passed encoding is 0. The way to convert the NSStringEncoding into the xmlCharEncoding is only work shift, that needs some improving, possibly by simply having a big switch statement. Maybe later.

There are multiple ways how you can parse a document, some of which take individual parameters, some take a html parsing context. I settled on using the one for the push interface because there the htmlParseDocument function does not return a pointer to an html document tree (DOM), but instead a result that is 0 for success or -1 for error.

It is my understanding that this variant does not actually build the entire tree but keeps just enough state information to track the current hierarchy level inside the HTML tree. The push parser variant also has the ability to parse chunks of HTML data as they become available, e.g. while downloading. But exploring that is outside our scope for now.

Conclusion

Once your eyes have gotten used to the coding style of libxml2 you can start to appreciate the plethora of functions that it provides. You can also peruse the official reference manual online.

Finally I invite your collaboration on using and helping to polish DTHTMLParser if you have any need for parsing HTML text from within your iOS apps. Look for it on the GitHub DTFoundation project as soon as I can add a bit of AppleDoc-style documentation to it.

In my previous article about Sub-Projects in Xcode I showed you how you can have an Xcode project as a dependency. The advantage of this is that you can update your library from within the same project and debug into the sub-project’s code.

One thing that was not immediately obvious was how you deal with the problem that Xcode cannot find your sub-project’s library headers. Hence this writeup.

A header file generally informs the compiler what c-functions, global variables, Object-C methods and classes exist in an object file. All these are generally called “symbols”.

Build Process Review

The build process consists of these three steps:

1. Preprocessor – this takes care of including/importing header files into your source code, essentially pasting the content of the specified file right into your source code for building. This step also replaces all instances of #defined values with what you defined them to be.
2. Compiler – this goes through all the .m files and builds an object file with .o extension for each. Each item that is not coming from the same .m file is marked as an external symbol.
3. Linker – this merges all the object files into one big binary while at the same time linking the symbols with the actual implementations.

Header files do not actually contain executable code, but instead they tell the compiler which names and functions it should accept. It is the linker then that resolves these references. In short: If you want to use something which is implemented outside the current .m file than you need the appropriate import for that.

There are two kinds of imports:

• <path/header.h> with angle brackets – these are meant for system or “global” includes.
• “path/header.h” with double quotes – these are meant for items that are limited in scope to your current project

You have probably used both already, the one with the angle brackets for including headers that Apple provided. The one with the double quotes for accessing your own classes. Paths are usually relative paths because everybody has a different folder structure on his hard disk.

Get the Headers into Your App

The easiest way to get the headers into your project is to just add them. But you should generally refrain from duplicating your code for convenience, because then you have to perform an update on every copy of the file. A variation of this theme would be to not add the file itself, but just a reference. But this approach would also make your project dependent on your folder structure. Don’t do it, just don’t.

If you have followed the Sub-Projects guide you are already perfectly set up to get to the header while still keeping the project self-contained and independent from the file system layout. You only need to tell the outer Xcode project – probably your app that includes sub-projects for some static libraries – where it can find the necessary headers from the sub-project. Again you have multiple options: you can point it to the source code folder where the .h files are located. Or you can point it to the build output folder. Both are equally viable but if you work with static universal frameworks – as I do – then you can only use option two without losing your mind.

On my projects that contain reusable code I usually have multiple targets. One for a universal static framework – which is very convenient to add to a project because it automatically sets up the library and header paths for you when you drag it into your project. I have a second target for a static library because this allows for using the project as a sub-project and have the outer project link to the static library product. Because the framework conveniently bundles together multiple .h files you usually have the general name of the framework as path, i.e. you find <MobFox/MobFox.h> as the import of choice. For for example an Apple framework <Foundation/Foundation.h>.

Contrasting the MobFox/ path the actual location of the header file is PROJECT_ROOT/Core, so no MobFox path component there. This is the reason why we cannot point the containing app to the source folder because there is no header file MobFox.h inside a folder named MobFox. So via approach number 1 we would never find the header, we want to take door number two which is to point Xcode to the static library build output folder. But for this to work some setup is necessary.

Specifying Public Header Output Location

After building the static library Xcode will also copy certain headers to the output folder. A header can be public, project or private. Public means that it will be distributed together with the library. Project and Private won’t. You can set this for every header file individually. Headers for objects that you want to be able to see from your app should be Public.

If you now build the static library and look at the build log you will see that at the very end the public headers are copied somewhere:

We are astonished to find a path ending in Build/Products/Debug-iphoneos/usr/local/include for the two public header files. The reason for this being the default setting is that traditionally people would build binary files for the system they are working on. But that’s not what we want. So we go into the build settings for the static library target and adjust this. Don’t put this entry into either the Debug or Release lines, but into the line above these so that it may override both.

When we now build we see the output go to Build/Products/Debug-iphoneos/MobFox – as we want it. It is this extra MobFox path component that enables us to use the <MobFox/MobFox.h> style of including later on.

Now back to our app that is including this sub-project’s static library. There we now need to tell the compiler where it can find the above folder. It so happens that when building Xcode usually uses the same output folder for all products it needs to build, namely whatever is in the $(CONFIGURATION_BUILD_DIR) variable at the time. This in turn is set by default to $(BUILD_DIR)/$(CONFIGURATION)$(EFFECTIVE_PLATFORM_NAME).

The final piece of the puzzle to know is where to put that. This depends on the include style that I explained above. Double quoted includes need the path in “User Header Search Paths”. Angle-bracketed includes need the regular “Header Search Paths”. We are forced to use the latter due to the framework. We don’t need to check the Recursive box because at the given location there is a MobFox folder.

The <Multiple values> might cause you to pause for a second, but remember what I told you above. The configuration (Debug/Release) is part of this path and so even though you put the same variable in there, the actual result is a different one. Note that these values are bold because the target settings override the settings specified by the project. You can revert to the project settings by selecting the line and hitting CMD+Backspace.

There is an alternate option to put this path in the “User Header Search Paths” instead, but then you need to set “Always Search User Paths” to Yes. This will cause the compiler to also search the user header paths for angle bracket includes. Though personally I prefer the other option that needs less settings.

The other steps for building are explained in the Sub-Projects posts. If you carry them out (link with target), you will find that now your app finds the headers and successfully builds.

Conclusion

If Xcode complains that it cannot find a header then you should make sure that it is indeed in a place where it can find it. There is an abundance of default settings that may seem daunting, but if you know what to look for you can quickly find why your header cannot be found.

I wrote this article because I myself was stumped by not being able to explain why a project couldn’t find the MobFox header. So another learning of this episode is that you should not assume that the owner of your sub-project has set everything up correctly. In my case I too was missing the custom header output path setting. Errare humanum est.

Sometimes you want to tell somebody how you fixed a problem in their code, but for some reason the code is not on github so you cannot send them a pull request. If you felt really smart then you might put the changes you made into an e-mail, like “in file1.m:102 you change it to x, in file2.m:54 you make change y”.

Thought this doesn’t really help the developer you are trying to help. Even when following your change instructions to the letter it is a tedious and error-prone method of applying your changes.

Thanks to M. Douglas McIlroy, Adjunct Professor at Dartmouth Colleague – one of the early Unix pioneers – there is a better way. He invented the form of diff that we are using today to get the Difference between two versions of a file. And Larry Wall who invented the patch command which can take a diff file and effectively apply it to files.

Honestly I was afraid of using patch before, never having played with it. What you don’t know you fear…. but since you are reading this article I now finally did, so that I can explain its usage.

Making a Patch

Let’s try with a simple text file first.

file1.txt

Here's some text.
And a second line.
And a third line.
A fourth line.

file1_changed.txt

Here's some text.
And a third line.
A fourth line changed.

To get the difference between these files we just diff them:

diff file1.txt file1_changed.txt

This produces this output:

2d1
< And a second line.
4c3
< A fourth line. --- > A fourth line changed.

We see that diff features each changed block with a sort of address. We removed line two, so we see that the d in 2d1 probably means deleted. And the c in 4c3 probably means changed. If a line was changed you first see the original line prefixed by a less-than, then three dashes and then the changed line prefixed by a greater-than. diff also has a -y flag that puts the changes side-by-side, which more clearly visualizes the changes.

This kind of output is called the “normal” format. To be able to use the diff for patching this is missing some vital information, namely which file the changes belong to. For this purpose you have to use the “unified” format which you get with the -u flag. We also pipe the output into a new file.

diff -u file1.txt file1_changed.txt > patch.diff

This results in the patch file to have this content:

--- file1.txt	2011-12-25 12:17:30.000000000 +0100
+++ file1_changed.txt	2011-12-25 12:01:30.000000000 +0100
@@ -1,4 +1,3 @@
 Here's some text.
-And a second line.
 And a third line.
-A fourth line.
+A fourth line changed.

You can see that has information about which files are involved. Also you have minus and plus prefixes. A minus with a plus following means that the line was changed. Just a minus means that the line was only present in the original, i.e. deleted. Just a plus means that line was only present in the modified version, i.e. inserted.

Let’s Patch

Appling the patch we just created is very simple. If the current working directory contains both the patch and the to-be-patched file all we need is:

patch < patch.diff

You get an output telling you while files where touched. As soon as you have applied a patch, calling the patch command again with the same patch will get you a message about this giving you the option to ignore individual “hunks” which is what these blocks of changes are called.

Multiple Files

For multiple files the process is only slightly more complex. Say you have a a folder “original” and a folder “changed” and the latter contains multiple changed versions of the files in the former.

If you call diff passing a folder name instead of a file name then this causes diff to compare file1 in one folder against file1 in the second folder, exactly what we need. There are a couple of additional flags for diff that are of use in this case:

• -r to recurse through all subfolders of the passed folders
• -u to get the “unified format” output
• -p prints the c-function a change was contained in
• -N treats absent files as empty

So we go into the folder where our “original” and “changed” folders are located in and do:

diff -rupN original changed > original.patch

With this patch file it is easiest if you just move into the root of the original and copy the patch there. Then you can apply it without any additional parameters. Even though the diff contains the original and changed path prefixes it figures out that you must have meant the file contained in the current working directory.

cd original
cp ../original.patch .
patch < original.patch

There is one parameter of the patch command that you might need if the paths mentioned in the diff refer to a different folder structure than you have on your own machine. The man page of patch has this to say about -p:

-pnum or –strip=num
Strip the smallest prefix containing num leading slashes from each file name found in the patch file. A sequence of one or more adjacent slashes is counted as a single slash. This controls how file names found in the patch file are treated, in case you keep your files in a different directory than the person who sent out the patch. For example, supposing the file name in the patch file was

/u/howard/src/blurfl/blurfl.c

setting -p0 gives the entire file name unmodified, -p1 gives

u/howard/src/blurfl/blurfl.c

without the leading slash, -p4 gives

blurfl/blurfl.c

and not specifying -p at all just gives you blurfl.c. Whatever you end up with is looked for either in the current directory, or the directory specified by the -d option.

When I saw someone use patch for the first time he totally confused me by claiming that -p is short for the “patch level” which – as you can see above – is utter nonsense. It is probably short for prefix and the long form of it –strip actually means something entirely different.

It is essentially the number of slashes that are to be ignored, including the part of the path to the left of each ignored slash. A strip level of 0 does not modify the paths at all. A level of 1 ignores the path up to and including the first slash. Omitting the p parameter causes patch to ignore the complete paths and only look at the file names.

If a patch without specified strip level doesn’t work then you will have to take a look at the patch file and figure out how much of the paths you need to ignore so that these paths fit with the situation on your own file system. In our example above (with the original and changed folders) we could have used -p1 to have patch ignore the relative paths, but it seems to do that by itself.

Creating Patches from Repositories

Above we discussed one way to create diffs that can be used for patches by having two versions of your files in two folders and diff these two. This typically suites those people who have not yet warmed to the use of source code management systems (SCM) like git or Subversion.

In an SCM you don’t have multiple copies of each file, instead you have only one current version of the source code in the project directory and all previous versions are somehow hidden from view, but can be retrieved. In git this versions are called “commits” in Subversion “revisions”.

For the sake of trying it out, let’s create an empty Xcode project with a git repo. On the dialog where you specify the folder for the project you can set a checkbox to have Xcode also put it under version control.

It is probably an oversight in Xcode but if you create the project like this the first commit is being done for you with comment “Initial Commit”, however there are changes to the project file adding a “lastKnownFileType” to the project.pbxproj – which is contained in the xcodeproj bundle – that causes the project file to show as modified if you do “git status”.

To get to a clean starting state for the project I go to the folder where I created the project and do an additional commit.

cd ~/scmtest
git status
git commit -a
git log

I have vi set up as my editor so on the file that opens from the commit I enter some comment and then ESC : w q to save and finish the commit. The git log now shows no uncommitted changes, two commits and two untracked files containing some user workspace settings which you probably don’t need or want in your repo.

Now let’s modify some code. For this test I just added an NSLog into the app delegate file. What’s really interesting now is how to create a patch that contains the changes from our second commit to the current modified state of our working directory. Simple, really:

git diff --no-prefix > diff.patch

We need the option to omit the prefix because otherwise git diff adds an “a” and a “b” prefix in the paths which never correspond to actual paths. Of course you can also apply patches that still contain the prefixes if you use the -p1 strip level.

Our patch now looks like this:

diff --git scmtest/DTAppDelegate.m scmtest/DTAppDelegate.m
index 7730864..795fa6d 100644
--- scmtest/DTAppDelegate.m
+++ scmtest/DTAppDelegate.m
@@ -20,6 +20,8 @@

 - (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
 {
+    NSLog(@"Hello World");
+
     self.window = [[[UIWindow alloc] initWithFrame:[[UIScreen mainScreen] bounds]] autorelease];
     // Override point for customization after application launch.
     self.window.backgroundColor = [UIColor whiteColor];

You can see the inserted lines prefixed with plus, also there are a couple of lines of code before and after the insertion so that patch can find the place even when there have been other modifications to the file that changed the line numbers.

A simple git diff always contains all modifications made on the latest commit. To test this patch we discard our changes and see that we ended up on the second commit.

git reset --hard
git log

Then we apply the patch, we need a strip level of zero so that the relative paths are used as they are.

patch -p0 < diff.patch
git status

After this patch we see that the changes appeared again.

You can also create a patch between any two commits by specifying the commit identifiers as parameters. You can get the identifiers from git log.

git log
git diff 92cef602e685970e6d4316cd383217d39d169b89 ec24c6d3435a2ae9ddaabd76f958561e86f9f02e > b.patch

Finally let’s also see how the same can be gotten from an SVN repo. Same technique:

svn status
svn diff > diff.patch

Between two specific revisions the same is possible. Subversion numbers revisions incrementally instead of with identifiers as git does. So the command for the changes from a specific version to the next is this:

svn diff -r 126:127 > c.patch

Contrary to git the diff command for svn does not prefix the file paths, instead the revision info is contained in brackets after the files which is ignored by patch.

Conclusion

Knowing how to create and apply patches is a great skill to have as a programmer. So the next time you find that you want to precisely communicate what changes you made in somebody else’s code you should send them a patch instead of an email containing line numbers and code fragments.

And if they don’t know what to do with that, also send them a link to this here tutorial.

PS: AlBlue points out that on git your can also directly send patches via e-mail. Those even include your commit messages and identity.

Honestly I was very much excited when I found that I can use my current knowledge of Objective-C and Foundation classes like NSString to also build nifty little tools. Previously I had to resort to bash script to perform one-off operations on files that where too tedious to do manually. But knowing what I am going to show you in this article will enable you to also write these littler helpers.

I believe that beginners should rather start with writing a couple of command line tools before diving into building user interface driven apps. Commend line tools are linear and thus their function is easier to grasp. They are more akin to “functional programming” then “object oriented programming” if you will.

I am going to show you what goes into building a simple command line tool and you be the judge whether you agree with my assessment.

First we need a purpose for our tool. It just so happens that I have a great idea!

Purpose: Join Several Files in a Structure

Somebody handed me a set of all the Localizable.strings files from Apple’s built-in apps.  Those are divided by app and there you have one lproj sub-folder per language. I want to join all of these together in a big file so that I want to look up one translation I get all languages at the same time.

The strings files themselves are binary property lists, basically a big NSDictionary each. That’s the same format that they get converted to if you build your app. Xcode takes each text strings file, omits the comment and packs them into the bplist format for faster loading.

Before you set out to coding anything it is a good idea to reflect on which steps we think we require to achieve our goal:

1. Produce a command line utility that can be called from the terminal and passed one directory as parameters.
2. Iterate through the directories and get the contained files
3. Open each file ending in .strings as dictionary
4. Assemble in memory the tokens contained in each dictionary in a way that will facilitate generating the output we require

Ok, let’s get to it.

Step 1: Command Line Tool with Parameters

We create a new project with the “Command Line Tool” from the Mac OS X/Application category. Under “Type” we see a couple of different templates. We want to use the one that says “Foundation” because this gives us all the Objective-C classes we know and love, like NSArray, NSString etc. Let’s also make it an ARC project so we don’t have to worry about memory management. I called my tool stringsextract.

As with all C-based applications the starting point for the program is in the main function which you can find in main.m.

int main (int argc, const char * argv[])
{
    @autoreleasepool {
        // insert code here...
        NSLog(@"Hello, World!");
    }
    return 0;
}

If it weren’t for the autorelease pool this would look like a very normal C-function. In objc methods generally are either class (+) or instance methods (-). In c functions are standing alone.

This function main has an int as return value. If the program runs without error you want it to return 0, otherwise you want it to return something else, like 1.

It has two parameters argc and argv. The first (argument count) is the number of parameters contained in the second (argument values). Int is just a number, but the second looks a bit more completed to the untrained eye. It is an array of char pointers. Since in c arrays are same as pointers you sometimes see this also written as char **argv, because in essence it is a pointer to a pointer to a char.

Arrays in C are not secured against being overrun which is why we need the argc. Otherwise we would have no way of knowing how many entries this array has. C-style arrays are just some memory where the elements are in sequential order. To access the nth item, you use array[n-1] and the compiler will convert that into the correct memory address. (Note: in c indexes begin with 0)

Another thing to know about argc/argv is that there is always at least one parameter at index 0: the name and path of the executable. So even if you call this program without any parameters on the command line argc will be 1 and there will be a value in argv[0].

The next tricky thing besides having to deal with a c-style array is that we also have to know c-style strings to make sense of the argument values. I told you above that an array is just a bunch of items. C-strings are just a bunch of characters where the last one is a binary zero, aka ‘\0′, to delimit the string. So whenever you see char * this could either be a pointer to one character or it could be a pointer to the first character of a string.

Since we want to use Foundation constructs for our program we want to convert all c-stuff to the more intelligent variants like NSString, by means of stringWithUTF8String.

As a first experiment let’s output all the arguments. We do a normal for loop beginning with 0, convert each argument to an NSString and log it.

for (int i=0; i<argc; i++)
{
    NSString *str = [NSString stringWithUTF8String:argv[i]];
    NSLog(@"argv[%d] = '%@'", i, str);
}

If you hit Run now you will see the following output on the console:

2011-12-11 09:21:58.636 stringsextract[18003:707] argv[0] = '/Users/oliver/Library/Developer/Xcode/DerivedData/stringsextract-dlkdghopgfoghoanjhmahhrlotsp/Build/Products/Debug/stringsextract'

So really what I told you is true, the parameter at index 0 has the path of our binary.

How can we test our program while we are still working on it? Of course you could open a terminal and run it by hand:

iair:~ oliver$ cd /Users/oliver/Library/Developer/Xcode/DerivedData/stringsextract-dlkdghopgfoghoanjhmahhrlotsp/Build/Products/Debug/
iair:Debug oliver$ ./stringsextract bla
2011-12-11 09:25:03.357 stringsextract[18017:707] argv[0] = './stringsextract'
2011-12-11 09:25:03.363 stringsextract[18017:707] argv[1] = 'bla'

You see how I changed into the Debug output directory and executed the program there passing one extra parameter. The dot slash is necessary to tell the shell that it should take the stringsextract binary right there in the current directory. Otherwise it would go off searching the set up system paths. You also see that argv[0] does not actually contain the absolute path, but always how it was called.

Now this works, but it might be a bit tedious to work with while we are still testing and debugging our tool. Fortunately Xcode allows us to specify command line arguments in the schema. If one argument contains whitespace you need to put it in quotes.

If we hit Run like this then we can also debug and step through our code as if we had passed this parameter on the command line ourselves.

As a final step in this chapter let us add a usage text which is to be output if somebody calls this program of ours without the required one parameter.

// one parameter is required
if (argc!=2)
{
    // output usage
    printf("Usage: stringsextract <dir>\n");
 
    // leave with code 1
    exit(1);
}

Note the use of the c-function printf which unlike NSLog just outputs the stuff we want and no extra stuff.

Now we’re set to move to Step 2.

Step 2: Getting the Files

Foundation gives us some awesome utilities when working with the file system, most notably NSFileManager which groups together most of them. We have our parameter specifying an absolute path. So next we need to list the .app bundles contained in there.

Usually you would use [NSFileManager defaultManager] to get a file manager instance. Only if you need to set a call-back delegate you would go about alloc/init’ing it.

So following our check for the existence of at least one parameter we add:

// convert path to NSString
NSString *path = [NSString stringWithUTF8String:argv[1]];
 
// get file manager
NSFileManager *fileManager = [NSFileManager defaultManager];
 
// get directory contents
NSError *error;
NSArray *contents = [fileManager contentsOfDirectoryAtPath:path
                                                     error:&error];
 
// handle error
if (!contents)
{
    printf("%s\n",[[error localizedDescription] UTF8String]);
    exit(1);
}
 
NSLog(@"%@", contents);

This converts the parameter to an NSString, sets up a file manager and then gets the contents of the specified path. If this is successful then there is a non-nil value in contents. If not then error will point to a new NSError instance which gives us a nice localized description to tell the user. Again we use printf because we want the direct output. UTF8String is the method to get from an objective-C NSString to back to a regular C-string which is required by the %s print format.

The NSLog at the bottom is just temporary so that we can see if we indeed got a result. And so we did.

2011-12-11 10:08:47.023 stringsextract[18231:707] (
    ".DS_Store",
    "AppStore.app",
    "Calculator.app",
    "Camera.app",
    "Compass.app",
    "Contacts~iphone.app",
    "Game Center~iphone.app",
    "Maps~iphone.app",
    "MobileCal.app",
    "MobileMail.app",
    "MobileNotes.app",
    "MobilePhone.app",
    "MobileSafari.app",
    "MobileSlideShow.app",
    "MobileSMS.app",
    "MobileStore.app",
    "Music~iphone.app",
    "Nike.app",
    "Preferences.app",
    "Reminders.app",
    "Setup.app",
    "Stocks.app",
    "Videos.app",
    "Weather.app",
    "YouTube.app"
)
Program ended with exit code: 0

We also see that there is one entry that we want to ignore later on, the “.DS_Store”. So let’s filter out everything that does not end in .app as I showed in a previous post.

Now we need to iterate through the array of app paths and get the contents of these, filtering for lproj. Since the contents are without the full path we need to concatenate the individual names with the path.

// filter
NSArray *filter = [NSArray arrayWithObject:@"app"];
 
// execute filter in place
contents = [contents pathsMatchingExtensions:filter];
 
// iterate through .app
for (NSString *oneApp in contents)
{
    // make full path
    NSString *appPath = [path stringByAppendingPathComponent:oneApp];
 
    // get contents, ignore error
    NSArray *appContents = [fileManager contentsOfDirectoryAtPath:appPath
                                                            error:NULL];
 
    NSLog(@"app: %@, %@", oneApp, appContents);
 
}

Again we have a bunch of files that we one and one that we don’t, in this case I have have a _CodeResources. Granted I could have removed these by hand but we’re writing this program here to be smart and save us manual labor. So we code around that just the same.

Rinse and repeat. We end up with three loops, one for the apps, one for the lprojs and one for the strings.

int main (int argc, const char * argv[])
{
    @autoreleasepool 
    {
        // one parameter is required
        if (argc!=2)
        {
            // output usage
            printf("Usage: stringsextract <dir>\n");
 
            // leave with code 1
            exit(1);
        }
 
        // convert path to NSString
        NSString *path = [NSString stringWithUTF8String:argv[1]];
 
        // get file manager
        NSFileManager *fileManager = [NSFileManager defaultManager];
 
        // get directory contents
        NSError *error;
        NSArray *contents = [fileManager contentsOfDirectoryAtPath:path
                                                             error:&error];
 
        // handle error
        if (!contents)
        {
            printf("%s\n",[[error localizedDescription] UTF8String]);
            exit(1);
        }
 
        // filter
        NSArray *filter = [NSArray arrayWithObject:@"app"];
 
        // execute filter in place
        contents = [contents pathsMatchingExtensions:filter];
 
        // iterate through .app
        for (NSString *oneApp in contents)
        {
            // make full path
            NSString *appPath = [path stringByAppendingPathComponent:oneApp];
 
            // get contents, ignore error
            NSArray *appContents = [fileManager contentsOfDirectoryAtPath:appPath
                                                                    error:NULL];
 
            // new filter
            filter = [NSArray arrayWithObject:@"lproj"];
 
            // execute filter in place
            appContents = [appContents pathsMatchingExtensions:filter];
 
            // iterate through .lproj = one language
            for (NSString *oneLang in appContents)
            {
                // make full path
                NSString *langPath = [appPath stringByAppendingPathComponent:oneLang];
 
                // get contents, ignore error
                NSArray *langContents = [fileManager contentsOfDirectoryAtPath:langPath
                                                                         error:NULL];
 
                // new filter
                filter = [NSArray arrayWithObject:@"strings"];
 
                // execute filter in place
                langContents = [langContents pathsMatchingExtensions:filter];
 
 
                for (NSString *oneStrings in langContents)
                {
                    // make full path
                    NSString *stringsPath = [langPath stringByAppendingPathComponent:oneStrings];
 
                    // do work here
                    NSLog(@"%@", stringsPath);
                }
            }
        }
    }
    return 0;
}

If this code makes your stomach churn then this is because this more and more turns into spaghetti code. This can still be fine for a one-off tool, but should be avoided if you plan to hand out this tool for others to use.

For one thing I would add a category to NSFileManager that does the listing, filtering and full-path-assembling in one step. Or possibly even go one further to have the working code in a block and have this category iterate over the strings files.

This we will be exploring in the next installment of this two part tutorial.

A very interesting yet very undocumented functionality of Xcode is that you can have sub-projects in your project tree. You can add an xcodeproj to your project and link to this project’s output.

This is exceptionally useful if you are developing some functionality in a contained project and now want to access this polished functionality from another project. Like for example you want to add to your app the capability of accepting HTML code copied from Safari and use my DTWebArchive classes for that. You could either copy all classes to your project, build two libraries (one for Simulator and one for Device, or lipo these two together), or build a static universal framework.

Or there is an option number 4 which I want to tell you about in this post. This option does neither copy source code nor does it involve building something upfront.

Note that Xcode 4.2 might cause you a bit of trouble until this works right. Some people told me that they did what I am about to describe in Xcode 3 to get it working. One way or the other this here should work, and if it doesn’t work for you then “you’re holding it wrong”.

Sub-Projects, An Overlooked Option

This is the first case where I myself am using sub-projects. My DTRichTextEditor framework is making heavy use of my two open source projects. This is how the project tree looks on my machine:

The file system structure and the group structure are more or less identical. I have a folder Externals that contains clones of these two GitHub projects. When you add an xcodeproj to your project then it will show up as shown in the picture, allowing you to browse the source code contained in this sub-project as well. Better yet, when you’re debugging you get to step through the sub-project’s code as if it where local.

There are several caveats, besides sometimes having to restart Xcode several times for the project trees to show up. Xcode seems to take objection to you having the sub-project also opened at the same time. If you did that you will get an error message that the consistency of the sub-project cannot be verified. So make sure you don’t have the sub project open at the same time. You can edit the code from within the larger project anyway.

Preparing the Sub-Project

For being able to use an xcodeproj like shown above you need a couple of things, let me walk you through these.

You need at least one target that builds an iOS static library. Go to your project root and add such a target. Then pick all the headers and implementation files that you want to be part of this library.

I like to name the target different from the product. I want the target to be called “Static Library” and then I go into the build settings and call the product DTWebArchive, resulting in a product libDTWebArchive.a showing up under Products.

You do not need to modify the header use which can be project, private or public. Changing it from the default project just copies these headers into the product output path which is of little use. You might also want to adjust the name of the Scheme, or auto generate schemes for your targets. These schemes are just a convenience for you personally, they are not even saved in the xcodeproj file.

Make sure that your library builds without warnings or errors. Note that if you didn’t select any source files then the libtool will fail, too, without any useful error message. That happened to me on several occasions.

Adding the Sub-Project

Once you are satisfied with the static library close this project and open the other one that should contain it. Depending on how you work you could either keep all your projects in the same location next to each other or have the larger project actually also contain the smaller one. It’s up to you. I recommend the second option because your file system structure might be different from somebody else building this project. If you use absolute paths, or even relative ones pointing outside of the project root folder then you are asking for trouble.

When adding the sub-project you don’t need to select it to be a member of any targets just yet. Best if you just drag the xcodeproj from Finder into the project.

Next you need to add the path to look for the sub-project’s headers in to your build settings. You cannot add the .h files to your main project, but you can tell the compiler where it can find them. Specify the path relative to your project’s root folder. I chose recursive because I have multiple sub-projects and this way I only need to specify this setting once.

With this setup you can use the same import directives that you are used to.

You tell the linker to link against the static library target in the sub project. You set this up on the Build Phases tab, under “Link Binary With Libraries”. You probably want it to be “Required” for static libraries because we cannot build dynamic libraries (dylib) as Apple is using in their frameworks.

You also want to add the static library targets as dependencies for your app. This way the build system knows that it needs to rebuild the app object code if code in the dependency changes.

If you were only programming in c then that would be all, but since we are coding in Objective-C (and possible use ARC) there are “Other Linker Flags” we need to add:

• -ObjC because otherwise the linker does not properly load your classes
• -all_load if you have categories in the library, because otherwise those are not loaded
• -fobjc-arc if you are linking against a library built with ARC from an app that does not yet use ARC

Actually the first two are pretty standard, so if you have ever done anything with static libraries before you already know about these.

Conclusion

You see, there is not very much too it and you can stop copying your reused source code all over the place. Better you put it in your own foundation framework (as I have begun with DTFoundation), document it and add the appropriate unit tests. This way any new methods or polishing benefits all your apps instead of just the current one.

I would say I have the process of submitting iOS apps down, I could probably do it blindfolded. Actually I HAVE done it blindfolded on several occasions guiding people over the phone on their very first submission.

But today I am doing it for the first time on an app for the Mac App Store (aka MAS). Let’s see if I can see this through to a successful conclusion, or at least until the ball is out of my court and with the app review team.

A couple of months ago we made a deal with BytePoets, a small software company based in Graz, Austria. At that time I had an idea for a Mac tool that would greatly simplify how you localize apps. But at the same time I was afraid of starting to code on OSX. I did not have the time to devote to this project to get up to speed myself, so I let them do the development. Publication and marketing of the app now falls to me.

Certificates and Profiles

So as the first step – I figured – I probably need a distribution certificate for the app. So I logged into the Mac Developer Center and went looking for the Mac equivalent of the provisioning portal. I found the “Developer Certificate Utility”, which sounded like what I needed.

First thing to do there was to register an app ID. You choose an app bundle identifier similar as you would do on iOS, in reverse domain notation with the final part being the name of your app. Also you give the identifier a name or description so that you find it later.

The second step is to confirm your input as you can only delete such identifiers from your account, but you will never be able to edit them.

On the screen that follows you can configure iCloud and Apple Push Notification support for the app, but we didn’t need those at this time.

On the Certificates tab I was confused to see a Mac App and a Mac Installer certificate already there. I didn’t recall creating these. So I ignored them for the time being and clicked the big blue button to request a new certificate.

Then I felt at home again, you get a choice to create a developer or a distribution authority and just like on iOS you have to use the certificate assistant in your Keychain Access to get these. The assistant creates the private key in your keychain and you save the public key to a file. This you upload to Apple and they sign it. Then at the end of the process you download their result and it joins the private key in your keychain when you open it from your downloads folder. You have to create two distinct signing requests for a development and a distribution identity.

The product of this is that you end up with 5 certificates on your keychain, the “iPhone Developer/Distribution” you have seen before, as well as two new ones titled “3rd Party Mac Application” and “Mac Developer”. Looks ok, if all 5 certificates have the private key dangling under it.

On my first run through here I forgot about the second distribution profile for the MAS, you need to have both a Installer and Application certificate for that, where with iOS this is only one certificate. If you don’t get the Installer certificate then you will find that you cannot validate or submit apps, instead you get stuck with “Don’t sign package (No valid signing identities found)”.

The third tap is titled Systems and looks to be the same as where you enter your iOS device IDs. For practice I went ahead and registered my MacBook Air. I didn’t know that even Macs had hardware device identifiers! You can find the one for your machine on the “About this Mac” and then look for “Hardware UUID”.

Just like with iOS you are apparently limited to registering 100 such systems per year. Obviously this has to be only for testing purposes, like iCloud and Push Notifications, right?

The fourth and last step is to titled Provisioning Profiles, again a concept we met in the iOS world. Again a big blue button leads to a selection between either Development or Production, where I believe the latter to mean the same as Distribution.

For a Development profile you specify a name or description, choose the previously generated bundle identifier, select the systems that are allowed to run this and the developer certificate. For the Production profile I was presented with two Mac Application Distribution Signing Certificates, so I chose the one with a later expiration date. That must have been one of the two that I saw earlier, but couldn’t remember creating. Naturally production apps don’t have any restriction regarding systems, so this is grayed out.

Once the button on the following page turns from Processing to Download you get to see a new pref pane that you never noticed before. Just like on iOS there is also a profiles section in your system settings.

One thing that stumped me at first was that I got an error message when I tried the same exercise with the production provisioning profile which happens automatically after download. I suppose that you cannot install a production provisioning profile in your profiles pref pane because it does not have your device identifier. Also on iOS you wouldn’t ever install the app store distribution profile on a device.

Adding these new provisioning profiles to the Xcode organizer, under devices, worked fine. You can see my automatically managed team provisioning profile for iOS and the two new Mac provisioning profiles. And actually this confirms what I suspected for the production profile: “This profile cannot be installed on devices”. Aha. Thanks Apple for letting me think for a few minutes that I did something wrong …

That seems to be all in terms of general setup, next I did a fresh checkout of the app’s trunk from my Subversion repository … and made a fresh cup of tea to calm my nerves, I’ve started getting sweaty palms from all the excitement.

Creating a New App on iTunes Connect

Before getting into the intricacies of building the app I had to do the work required for apps on iTunes Connect to get a new app id number which is needed for the “Rate this app” button. This number is assigned by Apple when you create the app on the iTunes Connect portal.

So I logged into the well known http://itunesconnect.apple.com, clicked the “Add new app” and chose Mac OS X App.

You choose a name for the app, an SKU code (I usually take the app name in upper case characters) and choose the correct bundle identifier which we created earlier. You are again reminded that “the Bundle ID cannot be changed if the first version of your app has been approved or if you have enabled Game Center or the iAd Network.”

Next you select the availability date and a pricing tier as you are used to.

On the third screen you have the usual elements to fill out, a description, rating, URLs. Been there, done that.

The first problem I ran into was to provide an acceptable screen shot. Earlier I had simply opened the app and made a screen shot of the app window. Turns out it must be a JPEG, TIF or PNG file that is 1280×800 or 1440×900 pixels, at least 72 DPI and in the RGB color space. A bit of a problem for me on a MacBook Air, since the maximum resolution there is 1366×768.

The trick I used was to switch the display to 1152×720 (stretched), put the app in full screen mode and make a screenshot with Shift+Cmd+3 to not get a shadow. This was easily upscaled to 1280×800 by means of the Preview app. This PNG was accepted without problems.

After a nerve-wrecking long wait on hitting the Submit button, the creation of the app worked and I got my app identifier. I found it odd that there was no means to upload an app icon, but I figure that the icon contained in the app bundle is probably of sufficient resolution for Apple to take it from there.

Building the Code

So back to Xcode where I had the app already running in debug so that I could make the screenshot. So first check was for which mode the Archive scheme was set: release. Next I went into the target settings and adjusted code signing to match my production provisioning profile, the “3rd Party Mac Developer Application” option.

Building the code for the first time like this triggered lots of warnings that seem to come from using attributes in xib files that are not supported prior to 10.7. I also found a warning about receipt validation (by Nick Paulsen) not being enabled, I had heard about this mechanism that allows Mac apps to validate their honest origin as the MAS.

So I enabled the define after adjusting the bundle identifier (com.drobnik.Linguan) and bundle version (1.0), to see several linker errors. I had to add the IOKit, Security and libcrypto dynamic libraries to fix these.

After that the Archive build went right through and appeared in the Xcode organizer. I tried the Validate button there, but got another annoyingly unhelpful error message.

Well, the app version was still in status “Prepare to update” and I had to go in and push the “Ready to Upload Binary” button. Et voila! No error message any more on hitting Validate, but instead the selection for app and profile.

Next problem: No valid identity found, even though I was certain that I had set up code signing. I mentioned this earlier, I had forgotten to get the Installer certificate set up as well. This last step only works if you have TWO certificates set up for the app store process: Installer and App.

After getting the second certificate set up too, I still could not see it working, but after a restart Xcode picked up the installer certificate and let me upload the app. Hooray!

“… submitted to the App Store for further review” is again another part that we have met before, also called “the waiting game”.

Conclusion

Submitting apps to the Mac App Store works just like iOS apps with only very minimal differences.

• You have 2 certificates for app store distribution: Installer and App
• You can also have a certificate for development, though the real need for that escapes me
• You have to register your development devices and use a development provisioning profile, probably to test things like iCloud
• Setup for code signing is the same, though you use a different certificate to sign the app and to sign the installer package that the uploader generates for you.
• Screenshot format is different
• You don’t submit Icon PNGs, those get pulled by iTunes from the app itself
• The rest of the submission dialogs is identical

I did not go into Entitlements so far, as October 31st was said to be the last day before Apple said that they are going to be enforcing App Sandboxing. That in itself is a whole different world of pain, because of the way this app works with Xcode projects.

When starting to work on our iCatalog.framework I stumbled upon an annoying problem, the same that you will face if you ever need to work with large images. “Large” meaning of a resolution sufficient to cover the entire screen of an iPad or potentially double that (horizontally and vertically) when dealing with Retina Resolution on a future iPad.

Imagine you have a UIScrollView that displays UIImageViews for the individual pages of a catalog or magazine style app. As soon as even one pixel of the following page comes on screen you instantiate (or reuse) a UIImageView and pop it into the scroll view’s content area. That works quite well in Simulator, but when you test this on the device you find that every time you try to page to the next page, there is a noticeable delay. This delay results from the fact that images need to be decompressed from their file incarnation to be rendered on screen. Unfortunately UIImage does this decompression at the very latest possible moment, i.e. when it is to be displayed.

Since adding a new view to the view hierarchy has to occur on the main thread, so does the decompression and subsequent rendering of the image on screen. This is where this annoying stutter or pause is stemming from. You can see the same on app store apps where scrolling through something stutters whenever a new image appears on screen.

You have basically two main choices for an image format on disk, JPEG and PNG. Apple generally recommends that you use PNGs as graphics for your user interface because when building your app those get optimized by an open source tool named PNGCRUSH. This changes the PNGs such that they can be decompressed and rendered faster on iOS devices by making them easier to digest for the specific hardware. The first iPad magazines, like Wired, were using PNGs as the image format in which they transported the invidiual magazine pages causing one edition of Wired to take upwards of 500 MB.

Just because PNGs are automatically optimized for you, that does not mean they are the apex of wisdom for any kind of purpose. That’s all nice and dandy for images that you can bundle with your app, but what about images that have to be downloaded over the internet? There are distinct advantages and disadvantages to both formats. Let’s review…

PNGs can have an alpha channel, JPEGs cannot. PNGs have lossless compression, JPEGs allow you to choose a quality of anywhere between 0 and 100%. So if you need the alpha channel – how transparent each pixel is – you are stuck with PNG. But if you don’t need a pixel-perfect rendition of your image then you can go with the perceptually optimized JPEG which basically omits information you don’t see anyway. For most kinds of images you can go with around 60-70% of quality without any visual artifacts spoiling the visual fidelity. If you have “sharp pixels”, like for text you might want to go higher, for photos you can choose a lower setting.

Looking at memory used an image takes multiple chunks of it:

1. space on disk or being transferred over the internet
2. uncompressed space, typically width*height*4 Bytes (RGBA)
3. when displayed in a view the view itself also needs space for the layer backing store

There’s an optimization possible for 1) because instead of copying the compressed bits into memory they can also be mapped there. NSData has the ability of pretending that some space on disk is in memory. When it is being access then you actually access the bytes on disk and not in RAM. CGImage is rumored to know by itself which loading strategy is more efficient. UIImage is basically just a wrapper around CGImage.

Then there’s the question of “How fast can I get these pixels on screen?”. The answer to this is comprised of 3 main time intervals:

1. time to alloc/init the UIImage with data on disk
2. time to decompress the bits into an uncompressed format
3. time to transfer the uncompressed bits to a CGContext, potentially resizing, blending, anti-aliasing it

To be able to give a definitive answer to a scientific question, we need to take measurements. We need to benchmark.

Basic Assumptions and Ingredients

I made a benchmark app that I ran on several iOS device I had handy. Since I want to also compare crushed versus non-crushed PNGs I needed to start with a couple of source images that Xcode would crush for me. I would have been nice to also dynamically try out different size, but at present I lack a possibility of doing the pngcrush on the device. So I started with 5 resolutions of the same photo of a flower. Of course a true geek would have contrived of multiple different images representing both highly compressible, medium and non-compressable source material. But I could not be bothered. And please don’t get me started on sample size. TOO MUCH INFORMATION.

Please forgive odd outliers and the lack of resolution due to 1 ms being the minimum unit. We are not after a doctorate in image processing, but some practical information.

We are mostly interested in a general comparison of compression schemes and not specific niche cases. The benchmark includes 128*96, 256*192, 512*384, 1024*768 and 2048*1536 resolutions and we clocked Crushed versus Non-Crushed PNG, and JPEGS ranging from 10% to 100% quality in 10% increments. This benchmark was run on iPad 1+2, iPhone 3G, 3GS and 4. This should give us multiple measurements to evaluate performance between formats as well as between devices.

First, let’s look at raw device performance. Hardware descriptions from Wikipedia:

• iPhone 3G: “Most of the iPhone 3G’s internal hardware were based on the original iPhone. It still included a Samsung 32-bit RISC ARM11 620 MHz processor (underclocked to 412 MHz), a PowerVR MBX Lite 3D GPU, and 128 MB of eDRAM.”
• iPhone 3GS: “The iPhone 3GS is powered by the Samsung APL0298C05 chip, which was designed and manufactured by Samsung. This system-on-a-chip is composed of an ARM Cortex-A8 CPU core underclocked to 600 MHz (from 833 MHz), integrated with aPowerVR SGX 535 GPU. It also has 256 MB of eDRAM. The additional eDRAM supports increased performance and multi-tasking in iOS 4.”
• iPhone 4: “The iPhone 4 is powered by the Apple A4 chip, which was designed by Intrinsity and, like all previous iPhone models, manufactured by Samsung. This system-on-a-chip is composed of an ARM Cortex-A8 CPU integrated with a PowerVR SGX 535 GPU. The Apple A4 is also used in the iPad where it is clocked at its rated speed of 1 GHz. The clock speed in the iPhone 4 has not been disclosed. All previous models of the iPhone have underclocked the CPU, which typically extends battery life and lowers heat dissipation.”
• iPad: 1 GHz Apple A4 system-on-a-chip with 256 GB DDR RAM in chip package
• iPad 2: 1 GHz (dynamically clocked) dual-core Apple A5 system on a chip with 512 GB DDR2 RAM in chip package

Two of these devices have the same processor, the iPad 1 and iPhone 4. We see the major shift in performance at the same time when Apple moved to their own silicone. The (under clocked) A4 is still twice as fast as the ARM Cortext-A8 and PowerVR SGC 535 GPU.

Crushed by the Evidence

Total time for loading, decompressing and rendering  a 90% JPEG at 1024*768:

• iPhone 3G: 527 ms
• iPhone 3GS: 134 ms
• iPad: 79 ms
• iPhone 4: 70 ms
• iPad 2: 51 ms

Comparing Crushed versus Non-Crushed PNG, also 1024*768:

• iPhone 3G: 866 ms – 1032 ms = 16% faster
• iPhone 3GS: 249 ms – 458 ms = 46% faster
• iPad: 130 ms – 256 ms = 49% faster
• iPhone 4: 179 ms – 309 ms = 42% faster
• iPad 2: 105 ms – 208 ms = 49% faster

On iPhone 3GS and above it is a good rule of thumb to say that crushed PNGs are twice as fast as uncrushed ones.

Following are the charts for all 5 test resolutions. I have plotted alloc/init, decompress and drawing time needed separately. The bars are ordered front to back by speed, smaller bars mean faster. Charts courtesy of Christian Pfandler.

At first glance you can see that the relationship between compression schemes is more or less the same within one chart i.e. one resolution. But I still present all to you here for sake of completeness.

The first two charts represent resolutions that you would encounter when making images to supplement your user interface. Ignoring the fact that old hardware is slow you can draw a mental line at around 20 ms. This is the limit whereabouts the images are small enough to be drawn directly in a timely manner. Anything about this line means you either have a crappy old iPhone or you have to get the decompression off the main thread if you don’t want your app to be all stuttery.

The next three charts are of resolutions that – for the sake of analysis – we can deem as representative of phone full screen, pad full screen and page retina full screen.

Up to this point you might still survive with lots of UIImages and UIImageViews especially if you don’t care about fluidity on older devices. Though you see that you take a severe performance hit by using PNGs at this stage. At full screen phone resolution you definitely want to prefer JPEG.

At these resolutions we are way beyond the comfort zone. Even the fastest device can only decompress between 2 (iPad Retina) and 10 (iPad full screen) huge images per second.

Even if you get the decompressing off the main thread the drawing itself still takes some significant time. You want to have this done with CATiledLayer, split your images into tiles and cache the hell out of it.

Uniformly we can see that the red portion (decompress) is always the part that takes the most time. Drawing time is only dependent on the resolution, not the complexity of the compression scheme, which makes sense because at that point the game is won by points … pixels that is.

Generally a 100% JPEG is about equivalent to a crushed PNG. I can think of two reasons why you would choose that: a) you cannot dynamically create a crushed PNG on an iOS device, b) file size does not matter and you want to make sure you store a pixel-perfect replica.

Parallel to the file sizes (see below) you see how processing time increases linear from JPG 10% to JPG 90% and goes upwards steeply from there.

An interesting observation is buried in the last chart. In order for responsiveness to be on an iPad Retina display as we are used to with iPad 1+2 the image decoding would need to be 3-4 times faster than the iPad 2. From iPad 1 to 2 it only doubled. There you have the reason why iPad 2 cannot drive Retina just yet. It might not even be the next generation hardware, but 2 generations in the future that will be the first to manage that.

File Sizes

Let’s review the file sizes. Crushing PNGs makes them faster to render, but does it decrease file size?

Crushing PNGs reduces file size only for large images by a minimal amount. The maximum quality of JPEG somewhat compares to crushed PNGs, but setting it to 100% defeats the purpose of compression. You can see that choosing 90% instead of 100% more than halves the file size for any resolution. JPEG file size increases linear with quality but increases sharply over 90%.

Real Time?

When looking at the numbers for full screen images you see a problem that we’ve been having ever since Apple introduced the tablet form factor. A 70% JPEG at full screen resolution takes a whopping 75 ms to display on an iPad 1, 49 ms on an iPad 2 nowhere near real time. Meaning that this is way slower than the 60 fps you should be aiming for wherever possible. 13 fps or 20 fps are well below the human threshold of 30 frames per second that you need to consider a movement smooth. It turns my stomach seeing smooth scrolling and when a new image enters the POV the scroll motion gets stuck for a 20th of a second and then have it jump to again match your finger.

If we could take the decompression out of the equation then the numbers would look far more promising: 17 – 18 ms just for the drawing equate to about 55 fps, smooth as butter. Interestingly the iPads are not so much different in just blasting pixels onto layers, it’s the hardware accelerated decompression that makes the big difference.

One easy way around this was for me to simply draw the catalog pages in iCatalog on a CATiledLayer with fading disabled. This would cause the image display to occur on the background thread used for the tiling with no impact on the scrolling performance. The only thing you might notice there is that if you scroll quickly to the right pages might pop into sight after a brief delay. One disadvantage of this approach is that it is hard to get transitions between portrait and landscape orientations to look nice.

The other – albeit more advanced method – would be to force decompression on images just in time before they are needed.

Forceful Decompression

The first time you need an image’s pixels iOS is decompressing it. Usually this decompressed version sticks around – RAM permitting. So it makes very little sense to decompress an image by means of rendering it into a new one. This only has you end up with TWO uncompressed images, at least for a short time. It is sufficient to “just pretend”:

- (void)decompressImage:(UIImage *)image
{
	UIGraphicsBeginImageContext(CGSizeMake(1, 1));
	[image drawAtPoint:CGPointZero];
	UIGraphicsEndImageContext();
}

This causes the image to decompress, even though the image context is just 1 pixel large.

Strangely I could not consistently get the UIImage to keep the decompressed version around if it was just initWithContentsOfFile. Instead I had to use the ImageIO framework (available as of iOS 4) which provides an option to explicitly specify that you want to keep the decompressed version:

NSDictionary *dict = [NSDictionary dictionaryWithObject:[NSNumber numberWithBool:YES]
                      forKey:(id)kCGImageSourceShouldCache];
 
CGImageSourceRef source = CGImageSourceCreateWithURL((CFURLRef)url, NULL);
CGImageRef cgImage = CGImageSourceCreateImageAtIndex(source, 0, (CFDictionaryRef)dict);
 
UIImage *retImage = [UIImage imageWithCGImage:cgImage];
CGImageRelease(cgImage);
CFRelease(source);

Initializing the images like this I could see the times drop to indicate that decompression would only take place once. The first call to decompress took long time, the second no time at all. The magic code word is kCGImageSourceShouldCache which you can specify either for the CGImageSource or the CGImageSourceCreateImageAtIndex, according to the header this means:

Specifies whether the image should be cached in a decoded form. The value of this key must be a CFBooleanRef; the default value is kCFBooleanFalse.

If I set it to NO then I could see the drawing time also grow by the decoding time. If I set it to YES the decoding would only be happening once.

Conclusion

If you absolutely need an alpha channel or have to go with PNGs then it is advisable to install the pngcrush tool on your web server and have it process all your PNGs. In almost all other cases high quality JPEGs combine smaller file sizes (i.e. faster transmission) with faster compression and rendering.

It turns out that PNGs are great for small images that you would use for UI elements, but they are not reasonable to use for any full screen applications like catalogues or magazines. There you would want to choose a compression quality between 60 and 80% depending on your source material.

In terms of getting it all to display you will want to hang onto UIImage instances from which you have drawn once because those have a cached uncompressed version of the file in them. And where you don’t the visual pause for a large image to appear on screen you will have to force decompression for a couple of images in advance. But bear in mind that these will take large amounts of RAM and if you are overdoing it that might cause your app to be terminated. NSCache is a great place to place frequently used images because this automatically takes care of evicting the images when RAM becomes scarce.

It is unfortunate that we don’t have any way to know whether or not an image still needs decompressing or not. Also an image might have evicted the uncompressed version without informing us as to this effect. That might be a good Radar to raise at Apple’s bug reporting site. But fortunately accessing the image as shown above takes no time if the image is already decompressed. So you could just do that not only “just in time” but also “just in case”.

For the NSAttributedString+HTML Open Source project I chose to implement parsing of HTML with a set of NSScanner category methods. The resulting code is relatively easy to understand but has a couple of annoying drawbacks. You have to duplicate the NSData and convert it into an NSString effectively doubling the amount of memory needed. Then while parsing I am building an adhoc tree of DTHTMLElement instances adding yet another copy of the document in RAM.

When parsing HTML – and by extension XML – you have two kinds of operating mode available: you can have the Sequential Access Method (SAX) where walking through the document triggers events on the individual pieces of it. The second method is to build a tree of nodes, a Document Object Model (DOM). NSScanner lends itself to SAX, but in this case it is less than ideal because for CSS inheritance some sort of hierarchy is necessary to walk up on.

In this post we will begin to explore the industry-standard libxml library and see how we can thinly wrap it in Objective-C that it plays nicely with our code.

Getting libxml into your Xcode project is straightforward. Fortunately for us libxml is so old and established that you can find it already installed on Unix, Mac and iOS platforms. There are two kinds of libraries in C: static and dynamic. libxml is the latter which you can recognize by the .dylib extension.

Adding the Library

First we need to add the library providing all the XML and HTML structures and functions. We are actually using version 2.2 of libxml, the file libxml2.dylib is a symbolic link to libxml2.2.dylib.

Next – because libxml is not a framework that would package the necessary headers with it – we also need to tell Xcode where the headers can be found. Since libxml also comes with OSX, its headers – just like all other OSX system libraries can be found in /usr/include. Add /usr/include/libxml2  to the Header Search Paths and we’re set.

Now all we need to do to access libxml’s parsing methods and data structures is to add the appropriate import. Most of the internal structures are shared between the XML and HTML parsers and so we just need the HTMLparser header.

#import <libxml/HTMLparser.h>

Document Structure

Before we get into parsing let me show you how libxml represents HTML documents. Everything in libxml is a node. Because C does not have a concept of objects the classical method of representing a tree is by having C structs that have member variables pointing to other structs. A child is just a pointer to the child struct/node. If there can be more than one item, i.e. a list, this is represented by a linked list where the first node points to the next and so on until the very last node has a NULL pointer.

The smallest unit in libxml is xmlNode structure which is defined as such:

/**
 * xmlNode:
 *
 * A node in an XML tree.
 */
typedef struct _xmlNode xmlNode;
typedef xmlNode *xmlNodePtr;
struct _xmlNode {
    void           *_private;	/* application data */
    xmlElementType   type;	/* type number, must be second ! */
    const xmlChar   *name;      /* the name of the node, or the entity */
    struct _xmlNode *children;	/* parent->childs link */
    struct _xmlNode *last;	/* last child link */
    struct _xmlNode *parent;	/* child->parent link */
    struct _xmlNode *next;	/* next sibling link  */
    struct _xmlNode *prev;	/* previous sibling link  */
    struct _xmlDoc  *doc;	/* the containing document */
 
    /* End of common part */
    xmlNs           *ns;        /* pointer to the associated namespace */
    xmlChar         *content;   /* the content */
    struct _xmlAttr *properties;/* properties list */
    xmlNs           *nsDef;     /* namespace definitions on this node */
    void            *psvi;	/* for type/PSVI informations */
    unsigned short   line;	/* line number */
    unsigned short   extra;	/* extra data for XPath/XSLT */
};

The useful links depicted in the above chart as children, last, parent, next, prev and doc. The type value is the kind of role this node plays. If it is a tag then it is an XML_ELEMENT_NODE. The contents of a tag is represented by an XML_TEXT_NODE. Attributes are XML_ATTRIBUTE_NODE. Note that even if the original HTML does not contain a DTD, html or body tag these will be implied by the parser.

Let’s Parse Already

I sense that you grow impatient with me. Ok ok, we’re getting right to it now that you understand how libxml represents DOMs. Assume we have some HTML data downloaded from the web, the NSURL of it is in _baseURL.

// NSData data contains the document data
// encoding is the NSStringEncoding of the data
// baseURL the documents base URL, i.e. location 
 
CFStringEncoding cfenc = CFStringConvertNSStringEncodingToEncoding(encoding);
CFStringRef cfencstr = CFStringConvertEncodingToIANACharSetName(cfenc);
const char *enc = CFStringGetCStringPtr(cfencstr, 0);
 
htmlDocPtr _htmlDocument = htmlReadDoc([data bytes],
      [[baseURL absoluteString] UTF8String],
      enc,
      XML_PARSE_NOERROR | XML_PARSE_NOWARNING);

Since we don’t need any warnings or errors we can just ignore them by passing some options. The baseURL might be necessary to decode relative URLs contained in the document. And most importantly we cannot assume that UTF8 is used for encoding the bytes so we get the appropriate character set to pass to the parser.

Remember, this is pure C, so once we don’t need this DOM any more we need to trigger a routine that walks through this linked structures and frees up the reserved memory.

if (_htmlDocument)
{
   xmlFreeDoc(_htmlDocument);
}

If _htmlDocument is not NULL then we have successfully parsed the document. There are multiple methods how we could now use this, but for the final example in this post let me show you a function that just dumps the individual elements to to the log. This demonstrates how to follow the links and also how to access the contents of text elements.

xmlNodePtr currentNode = (xmlNodePtr)_htmlDocument;
 
BOOL beginOfNode = YES;
 
while (currentNode) 
{
    // output node if it is an element
    if (beginOfNode)
    {
        if (currentNode->type == XML_ELEMENT_NODE)
        {
            NSMutableArray *attrArray = [NSMutableArray array];
 
            for (xmlAttrPtr attrNode = currentNode->properties; 
                 attrNode; attrNode = attrNode->next)
            {
                xmlNodePtr contents = attrNode->children;
 
                [attrArray addObject:[NSString stringWithFormat:@"%s='%s'", 
                                      attrNode->name, contents->content]];
            }
 
            NSString *attrString = [attrArray componentsJoinedByString:@" "]; 
 
            if ([attrString length])
            {
                attrString = [@" " stringByAppendingString:attrString];
            }
 
            NSLog(@"<%s%@>", currentNode->name, attrString);
        }
        else if (currentNode->type == XML_TEXT_NODE)
        {
            NSLog(@"%s", currentNode->content);
        }
        else if (currentNode->type == XML_COMMENT_NODE)
        {
            NSLog(@"/* %s */", currentNode->name);
        }
    }
 
    if (beginOfNode && currentNode->children)
    {
        currentNode = currentNode->children;
        beginOfNode = YES;
    }
    else if (beginOfNode && currentNode->next)
    {
        currentNode = currentNode->next;
        beginOfNode = YES;
    }
    else
    {
        currentNode = currentNode->parent;
        beginOfNode = NO; // avoid going to siblings or children
 
        // close node
        if (currentNode && currentNode->type == XML_ELEMENT_NODE)
        {
            NSLog(@"</%s>", currentNode->name);
        }
    }
}

Note how I use %s so that I can use the zero-terminated C strings without having to convert them to NSStrings.

Obviously there are other ways to iterate through the document, for example by means of recursion. But this is meant to show how you can walk through nodes and their children and how you can also get the attributes.

Next time we will have a look how we can somehow wrap this pure C-code that we can more easily find and access parts of it. We cannot simply wrap xmlNode into an Objective-C class because then we might end up freeing the structure while an node instance is still present, thus creating a whole lot of junk pointers and introducing crash potential.

This is the case with the Objective-C HTML Parser project on GitHub by Ben Reeves. But even though I don’t share Ben’s philosophy, his prior work served as the starting point for this article.

If you show something that contains scrollable content, i.e. UITableView, UIScrollView etc. then you want to make an adjustment when the keyboard shows so that the user can still scroll to the entire content. He wouldn’t be able to do so if you didn’t do anything.

I’ve seen several approaches to this so far, but they often hard code a certain position of the view or sizes. Like assuming that the covered view always reaches towards the bottom of the screen or always has a certain amount of space taken away from it by the status bar, navigation bar and possibly toolbar.

The whole thing gets even more complicated by the fact the the coordinate system of the app’s window is always in portrait even though your app rotates. So is the frame of the keyboard which you can get from an info dictionary in several notifications. I’ll show you the most universally working method I was able to come up with.

The first part is obvious, you want to subscribe to two notifications, one for when the keyboard has fully animated in and one for when the keyboard will go away. I’ve seen people mess with the frame of the covered view, but I generally found this to be disturbing the contents, in the least causing an unnecessary redraw because of the bounds change.

Instead I prefer to set the content inset of the scroll view. This does not affect the contents, but permits for additional scroll distance so that the content at the bottom edge can be scrolled into the visible area above the keyboard. The inset for the scroll indicator is separate but needs the same treatment.

Note that all this is best done in your UIScrollView subclass.

NSNotificationCenter *center = [NSNotificationCenter defaultCenter];
[center addObserver:self selector:@selector(keyboardDidShow:) 
       name:UIKeyboardDidShowNotification object:nil];
[center addObserver:self selector:@selector(keyboardWillHide:) 
       name:UIKeyboardWillHideNotification object:nil];

And of course we remove the observer in the dealloc.

- (void)dealloc
{
	[[NSNotificationCenter defaultCenter] removeObserver:self];
	[super dealloc];
}

When the keyboard is hidden we don’t want any insets, so we reset them to zero.

- (void)keyboardWillHide:(NSNotification *)notification
{
	self.contentInset = UIEdgeInsetsMake(0, 0, 0, 0);
	self.scrollIndicatorInsets = self.contentInset;
}

And now the good part, calculating the correct covered area of the view, regardless of where it is positioned on screen. Yet another complication derives from the fact that if we use the scrollview’s coordinate system then we get different results depending on the content size.

So instead we need to use the superview’s coordinate system for transforming to and from the window coordinate system. This is necessary because the keyboard is mounted on the window and thus also has window coordinates.

- (void)keyboardDidShow:(NSNotification *)notification
{
	// keyboard frame is in window coordinates
	NSDictionary *userInfo = [notification userInfo];
	CGRect keyboardFrame = [[userInfo objectForKey:UIKeyboardFrameEndUserInfoKey] CGRectValue];
 
	// convert own frame to window coordinates, frame is in superview's coordinates
	CGRect ownFrame = [self.window convertRect:self.frame fromView:self.superview];
 
	// calculate the area of own frame that is covered by keyboard
	CGRect coveredFrame = CGRectIntersection(ownFrame, keyboardFrame);
 
	// now this might be rotated, so convert it back
	coveredFrame = [self.window convertRect:coveredFrame toView:self.superview];
 
	// set inset to make up for covered array at bottom
	self.contentInset = UIEdgeInsetsMake(0, 0, coveredFrame.size.height, 0);
	self.scrollIndicatorInsets = self.contentInset;
}

We use the handy CGRectIntersection function to get a rectangle that is common to both the keyboard and view frame. Since there might have been a rotation transform somewhere in between, we also need to convert back to our original superview’s coordinate system. And that’s it, now coveredFrame gives us exactly the area of our view that is covered and thus the hight is also the bottom inset we need to apply.

If you are making your application real-life-proof you will also have to deal with diminished or dropped connectivity. I already discussed how to detect the kind of connectivity your app is having at present.

But another real-life restriction stems from slower bandwidths available over cellular connections, especially if you have no 3G reception. Even Long of Scribd showed me this trick I am about to explain here. This enables you to artificially create a bottleneck on your connection so that you can test how the app behaves when only cellular bandwidth is available.

This helps you to see if your custom progress bar is showing nicely or possibly if your progressive image is indeed progressive. Also this might reveal synchronous URL loading operations that you should never ever do in production apps.

I would expect for Apple to add this artificial bandwidth limiting to iPhone Simulator, but until they do you heres a method to achieve this yourself. This is my regular network connection, please don’t laugh if you are used to higher speeds.

Now our trick creates a new IP firewall rule restricting the bandwidth to 112 kBits. You have to do this modification as root, hence the sudo and the first sudo will ask for your root password.

sudo ipfw add 500 pipe 1 ip from any to any
sudo ipfw pipe 1 config bw 112kbit/s plr 0 delay 20ms

The same speed test shows the much decreased connectivity:

Now you can test your app in simulator and enjoy the much faster turnaround when debugging the handling of slow connections.

To remove the firewall rule you use the following command:

sudo ipfw delete 500

This is a very handy trick to have in your arsenal. As I said many of the usability problems of production apps result from synchronous network operations as well as not having though through how to deal with slow connections. But these are a fact of life, so you better make sure that your apps can deal with them gracefully.

Update, a few minutes after the first post …

Now having so many smart readers of this here blog also means that often they know of smarter ways to achieve the same things. If you have Charles Debugging Proxy you already have a speed limiter built into that.

Another alternative that four people recommended independently is the speed limit preference pane by Mike Schrag which has the added advantage of allowing easy configuring for specific domains.

Both are comfortable alternate solutions, but I can never hurt knowing how to achieve the same thing in termal.

Still, thanks for all the suggestions!

You will often find yourself working with NSRange parameters and variables, especially when dealing with strings. I stumbled into a problem that I think is an SDK bug, that prompted me to look at the header and find out what kind of functions are provided to us for comfortably dealing with NSRange.

Don’t be fooled by the NS to think that this is an object. Similar to NSInteger this is just a scalar value, basically shorthand for the compiler to know that there are two components – location and length – with 4 bytes each. So if you access range.length you are interested in the second 4 bytes, for range.location you access the first 4.

This also means that you don’t have any instance methods available, all manipulation of NSRange has to be via C-style functions. So let’s have a look at the NSRange.h header contained in the Foundation.framework to see what interesting functions await for us to be unearthed there.

In fact, let me print the entire header and let’s read it together.

typedef struct _NSRange {
    NSUInteger location;
    NSUInteger length;
} NSRange;

This is how the NSRange is defined as a type. Types are what you need to define variables. You can see that NSRange consists of two unsigned integers.

typedef NSRange *NSRangePointer;

In some instances you might not deal with an NSRange directly, but with a pointer to 8 bytes of memory containing an NSRange. Instead of writing NSRange *every time, Apple defines an NSRangePointer for us to use. Since these are a couple more characters to type then the only reason I can think of is safety. Using NSRangePointer avoids the risk of forgetting the asterisk.

NS_INLINE NSRange NSMakeRange(NSUInteger loc, NSUInteger len) {
    NSRange r;
    r.location = loc;
    r.length = len;
    return r;
}

This is the most convenient way to fill an NSRange with values. It is defined as inline which means that this actual code is inserted into yours everywhere you use it as opposed to being execute as a true function call, which would involve copying stuff onto the stack, jumping to code, copying the result back and jumping again. Inlining happens transparently to you, but gives you the better performance of avoiding the function call. This usually makes sense for very small functions like this one.

NS_INLINE NSUInteger NSMaxRange(NSRange range) {
    return (range.location + range.length);
}

This is neat, using this inline function we don’t have to do the addition ourselves to find the maximum index contained in the range.

NS_INLINE BOOL NSLocationInRange(NSUInteger loc, NSRange range) {
    return (loc - range.location < range.length);
}

Another neat inline function lets us check if an index is contained in the range.

NS_INLINE BOOL NSEqualRanges(NSRange range1, NSRange range2) {
    return (range1.location == range2.location && range1.length == range2.length);
}

This saves us even more work if we wanted to see if two ranges are identical.

FOUNDATION_EXPORT NSRange NSUnionRange(NSRange range1, NSRange range2);

This creates a union of two ranges. The code for this is actually contained in one binary of the Foundation framework. The resulting union will go from the smallest location to include the maximum index. So if you create a union of two non-intersecting ranges the result also covers the space between the ranges.

FOUNDATION_EXPORT NSRange NSIntersectionRange(NSRange range1, NSRange range2);

This makes an intersection. This works only if the ranges are overlapping. If they don’t then {0,0} is returned. Knowing this we can easily check if one range is contained within another by creating an intersection and then checking if length is non-zero.

FOUNDATION_EXPORT NSString *NSStringFromRange(NSRange range);

If we needed a string representation of a range, then this function would quickly provide it for you. Use in NSLog for example.

FOUNDATION_EXPORT NSRange NSRangeFromString(NSString *aString);

In some even rarer cases you might want to do the reverse, move from NSString to NSRange. This is the function to achieve that.

@interface NSValue (NSValueRangeExtensions)
 
+ (NSValue *)valueWithRange:(NSRange)range;
- (NSRange)rangeValue;
 
@end

And the final block defines a category for NSValue to package NSRanges into objects for storing for whenever you need to have an Objective-C object, like if you want to store multiple ranges in an array.

Now, about the bug that I suspect to have found. NSAttributedString has a method to enumerate over all the attribute dictionaries and you get an NSRange for where this attributes are valid. enumerateAttributesInRange:options:usingBlock:

This usually works without problems unless you run into a low-memory condition. Apparently there is a situation when this method returns a range outside of the string. So this is how I worked around it to avoid problems.

NSRange validRange = NSMakeRange(0, [attributedString length]);
 
[attributedString enumerateAttributesInRange:range options:0 usingBlock:
         ^(NSDictionary *attrs, NSRange range, BOOL *stop)
{
         if (NSIntersectionRange(range, validRange).length)
         {
                  // work with attributes
         }
         else
         {
             NSLog(@"Invalid Range returned by attribute enumeration: %@", NSStringFromRange(range));
         }
}
];

So if the value returned from the enumeration is outside of the valid range then I ignore it.

Knowing of these functions allows you to more efficiently deal with NSRanges. So you might want to commit these to memory or make a mental or actual bookmark of this article.

In my previous post I demonstrated how to quickly whip up some lazy loading for a UIImageView. Today I revamped it to use NSURLConnection instead, because this would allow for cancelling the request. It also gives us the option of specifying if we want to make use of the cache and also how long we’re actually willing to wait for an answer.

That sounds more straightforward than it actually is. If you simply use an NSURLConnection with its delegate methods then you might not see anything wrong, unless you are using this lazy image view as subview of a UIScrollView. The problem there is that the scroll view blocks the run loop and so your connection events will not get delivered until you lift the finger.

But I found a solution for that. Let me know in the comments if you think something could be done in a more elegant way. The code for this is available in the NSAttributedString+HTML project.

DTLazyImageView.h

@interface DTLazyImageView : UIImageView 
{
	NSURL *_url;
 
	NSURLConnection *_connection;
	NSMutableData *_receivedData;
}
 
@property (nonatomic, retain) NSURL *url;
 
- (void)cancelLoading;
 
@end

DTLazyImageView.m

#import "DTLazyImageView.h"
 
 
@implementation DTLazyImageView
 
- (void)dealloc
{
	self.image = nil;
	[_url release];
 
	[_receivedData release];
	[_connection cancel];
	[_connection release];
 
	[super dealloc];
}
 
- (void)loadImageAtURL:(NSURL *)url
{
	NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
 
	NSURLRequest *request = [[NSURLRequest alloc] initWithURL:url 
				cachePolicy:NSURLRequestReturnCacheDataElseLoad timeoutInterval:10.0];
	_connection = [[NSURLConnection alloc] initWithRequest:request delegate:self
				 startImmediately:YES];
	[request release];
 
	// so that our connection events get processed even if a scrollview blocks the main run loop
	CFRunLoopRun(); 
	[pool release];
}
 
- (void)didMoveToSuperview
{
	if (!self.image && _url && !_connection)
	{
		//[self loadImageAtURL:_url];
		[self performSelectorInBackground:@selector(loadImageAtURL:) withObject:_url];
	}	
}
 
- (void)cancelLoading
{
	[_connection cancel];
	[_connection release], _connection = nil;
 
	[_receivedData release], _receivedData = nil;
 
	CFRunLoopStop(CFRunLoopGetCurrent());
}
 
#pragma mark NSURL Loading
 
- (void)connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)response
{
	// every time we get an response it might be a forward, so we discard what data we have
	[_receivedData release], _receivedData = nil;
 
	// does not fire for local file URLs
	if ([response isKindOfClass:[NSHTTPURLResponse class]])
	{
		NSHTTPURLResponse *httpResponse = (id)response;
 
		if (![[httpResponse MIMEType] hasPrefix:@"image"])
		{
			[self cancelLoading];
		}
	}
 
	_receivedData = [[NSMutableData alloc] init];
}
 
- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data
{
	[_receivedData appendData:data];
}
 
- (void)connectionDidFinishLoading:(NSURLConnection *)connection
{
	if (_receivedData)
	{
		UIImage *image = [[UIImage alloc] initWithData:_receivedData];
 
		//self.image = image;
		[self performSelectorOnMainThread:@selector(setImage:) withObject:image 
				waitUntilDone:YES];
 
		[image release];
 
		[_receivedData release], _receivedData = nil;
	}
 
	[_connection release], _connection = nil;
 
	CFRunLoopStop(CFRunLoopGetCurrent());
}
 
- (void)connection:(NSURLConnection *)connection didFailWithError:(NSError *)error
{
	NSLog(@"Failed to load image at %@, %@", _url, [error localizedDescription]);
 
	[_connection release], _connection = nil;
	[_receivedData release], _receivedData = nil;
 
	CFRunLoopStop(CFRunLoopGetCurrent());
}
 
 
#pragma mark Properties
 
@synthesize url = _url;
 
@end

The trick is to use CFRunLoopRun to have a new run loop to take care of delivering the events for this NSURLConnection. If you use that you also have to stop it if it’s no longer needed. There are now 3 places where this is the case: successful loading, error and canceling.

This also shows how you can check the MIME type of the received data, because you might want to cancel the operation if it is not an image you are getting back.

I found that I needed to combine this approach with running the NSURLConnection on a background thread because otherwise there would be unwanted side effects impacting the tracking in my UIScrollView. As I mentioned before, I’m on shaky territory as I have never messed with run loops before. So please tell me if there’s some problem with this approach.

… and so Nyxouf did, his suggestion in the comments allowed me to remove quite a few lines of this nasty code after I found this to be working just as well. The trick seems to be to not have the connection start right away, but first schedule it on the current run loop for the common modes and then start it.

DTLazyImageView.m

#import "DTLazyImageView.h"
 
 
@implementation DTLazyImageView
 
- (void)dealloc
{
	self.image = nil;
	[_url release];
 
	[_receivedData release];
	[_connection cancel];
	[_connection release];
 
	[super dealloc];
}
 
- (void)loadImageAtURL:(NSURL *)url
{
	NSURLRequest *request = [[NSURLRequest alloc] initWithURL:url 
			cachePolicy:NSURLRequestReturnCacheDataElseLoad timeoutInterval:10.0];
 
	_connection = [[NSURLConnection alloc] initWithRequest:request delegate:self 
			startImmediately:NO];
	[_connection scheduleInRunLoop:[NSRunLoop currentRunLoop] 
			forMode:NSRunLoopCommonModes];
	[_connection start];
 
	[request release];
}
 
- (void)didMoveToSuperview
{
	if (!self.image && _url && !_connection)
	{
		[self loadImageAtURL:_url];
	}	
}
 
- (void)cancelLoading
{
	[_connection cancel];
	[_connection release], _connection = nil;
 
	[_receivedData release], _receivedData = nil;
}
 
#pragma mark NSURL Loading
 
- (void)connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)response
{
	// every time we get an response it might be a forward, so we discard what data we have
	[_receivedData release], _receivedData = nil;
 
	// does not fire for local file URLs
	if ([response isKindOfClass:[NSHTTPURLResponse class]])
	{
		NSHTTPURLResponse *httpResponse = (id)response;
 
		if (![[httpResponse MIMEType] hasPrefix:@"image"])
		{
			[self cancelLoading];
		}
	}
 
	_receivedData = [[NSMutableData alloc] init];
}
 
- (void)connection:(NSURLConnection *)connection didReceiveData:(NSData *)data
{
	[_receivedData appendData:data];
}
 
- (void)connectionDidFinishLoading:(NSURLConnection *)connection
{
	if (_receivedData)
	{
		UIImage *image = [[UIImage alloc] initWithData:_receivedData];
 
		self.image = image;
 
		[image release];
 
		[_receivedData release], _receivedData = nil;
	}
 
	[_connection release], _connection = nil;
}
 
- (void)connection:(NSURLConnection *)connection didFailWithError:(NSError *)error
{
	NSLog(@"Failed to load image at %@, %@", _url, [error localizedDescription]);
 
	[_connection release], _connection = nil;
	[_receivedData release], _receivedData = nil;
}
 
 
#pragma mark Properties
 
@synthesize url = _url;
 
@end

In case you ever need to enhance UIImageView to load it’s image lazily… like if it’s coming from a web URL.

I used this on my NSAttributedStrings+HTML open source project to improve performance and also be able to have images that are not available locally. But you don’t want to do a synchronous call over the web, that would block your UI.

I admit, it’s a quickie, but hey, sometimes those are also fun!

DTLazyImageView.h

@interface DTLazyImageView : UIImageView 
{
	NSURL *_url;
 
	BOOL _loading;
}
 
@property (nonatomic, retain) NSURL *url;
 
@end

DTLazyImageView.m

#import "DTLazyImageView.h"
 
@implementation DTLazyImageView
 
- (void)dealloc
{
	self.image = nil;
	[_url release];
 
	[super dealloc];
}
 
- (void)loadImageAtURL:(NSURL *)url
{
	NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
 
	NSData *data = [[NSData alloc] initWithContentsOfURL:url];
 
	if (data)
	{
		UIImage *image = [[UIImage alloc] initWithData:data];
 
		[self performSelectorOnMainThread:@selector(setImage:) withObject:image 
				waitUntilDone:YES];
 
		[image release];
	}
 
	[data release];
 
	_loading = NO;
 
	[pool release];
}
 
- (void)didMoveToSuperview
{
	if (!self.image && _url && !_loading)
	{
		_loading = YES;
		[self performSelectorInBackground:@selector(loadImageAtURL:) withObject:_url];
	}	
}
 
#pragma mark Properties
 
@synthesize url = _url;
 
@end

You can see, it’s very simple. When the view is added to a super view by means of addSubview then it starts loading the image on a background thread. Setting the image has to occur on the main thread because UIKit is not threadsafe, or at least has not been until recently.

This topic is not for the faint of heart because it requires that you permit an additional level of abstraction in your brain to be able to grasp it. Until know all my ivars and properties where either scalar values (like an NSInteger) or pointers to Objective-C instances (like NSString *). But there are cases when you actually want to be able to store more complex functionality yet not have the overhead of object creation and messaging.

From the C days we have a mechanism called “function pointers” and today I’ll show you how you can pass a function itself to an Obj-C class and store it in an instance variable. There are a couple of SDK functions that make use of that.

The we’ll explore the modern-day equivalent of providing this sort of “plug in” dynamic functionality: doing the same thing with blocks. If you’re lucky to have iOS 4.x as minimum requirement for your project then those blocks might be nicer to work with than function pointers.

We assume that we have a class MyClass where another developer should be able to provide a custom function to calculate or perform something on data that you send via parameters.

Old: Function Pointers

Let’s say we want to pass an integer and the custom C-function would return double of the input.

float timesTwo(int x)
{
	return x * 2.0;
}

To make it easier on us, let’s define a type for a pointer to such a function. You can also do without that, but then you’ll have to write many more round brackets and asterisks. A simple typedef saves us lots of work and possible errors.

typedef float(*customFunc)(int);

I know this looks really weird and wrong, but trust me, that’s how you define a type customFunc that points to a function taking an int and returning a float. Note that this is a C-function, not an Objective-C instance method.

Next let me show you a custom class that has a property of this type, one method to make use of that and another where you directly pass in your function.

// .h
typedef float(*customFunc)(int);
 
@interface MyClass : NSObject 
{
	customFunc func;  // ivar for function pointer
}
 
@property (nonatomic, assign) customFunc func;   
 
- (void)useFunctionFrom:(NSInteger)from until:(NSInteger)until;
- (void)doFunction:(customFunc)function;
 
@end
 
// .m
 
@implementation MyClass
 
- (void)useFunctionFrom:(NSInteger)from until:(NSInteger)until
{
	// if func is NULL then calling it crashes
	NSAssert(func, @"Function needs to be set!");
 
	for (NSInteger i=from; i<=until; i++)
	{
		float result = self.func(i);
		NSLog(@"f(%d) = %.2f", i, result);
	}
}
 
- (void)doFunction:(customFunc)function
{
	for (NSInteger i=1; i<=5; i++)
	{
		float result = function(i);
		NSLog(@"f(%d) = %.2f", i, result);
	}
}
 
@synthesize func;
 
@end

You use the customFunc type just like you would use any other variable type. But you need to make sure that it is not NULL because calling a null function pointer raises an exception. That’s what the NSAssert is for.

In the first method we get the function from the synthesized getter (could also use the ivar directly) and call it, passing in i and logging the result. In the second method we call the passed function directly.

It’s more clear if you consider this example:

MyClass *m = [[MyClass alloc] init];
 
m.func = timesTwo;
[m useFunctionFrom:1 until:4];
 
[m doFunction:timesTwo];
 
[m release];

In the first example we pass timesTwo which is literally a pointer to the function in memory that we created above. In the second example we do the same, but directly pass the function pointer instead of parking it in an instance variable.

Now, this might sound very esoteric to you, but as I said there are certain use cases where it made sense to have your code call a custom function as opposed to just setting a multitude of parameters. For example NSArray has a method sortedArrayUsingFunction:context:

- (NSArray *)sortedArrayUsingFunction:(NSInteger (*)(id, id, void *))comparator 
       context:(void *)context

This weird stuff around the (*) we had typedef’ed away, I wonder why Apple didn’t… one function that you might use for this is shown in the documentation:

NSInteger intSort(id num1, id num2, void *context)
{
    int v1 = [num1 intValue];
    int v2 = [num2 intValue];
    if (v1 < v2)        
     return NSOrderedAscending;
        else if (v1 > v2)
    return NSOrderedDescending;
        else
    return NSOrderedSame;
}

You see: two id pointers and a generic context and the return value provides the sort order of the two objects. Without the typedef the function parameter would have looked similar: – (void)doFunction:(float (*)(int))function

New: Blocks

We don’t want to stick forever with the old ways if something better is available and broadly supported. Blocks are such a new technology, available as of iOS 4.0 and most of the user devices are now running this version or higher. So for new projects we can look at the new and shiny stuff.

Blocks are similar in function to function points, but you can put all your code inline and together with the code also all local variables are preserved. At least, that’s how I understand it, honestly I’m just beginning with blocks myself. The Pragmatic Studio did this great introduction to blocks.

We find that when changing MyClass from using a function pointer to using block variables the only relevant change we need to make is replace an * for a ^.

// .h
 
typedef float(^customBlock)(int);
 
@interface MyClass : NSObject 
{
	customBlock blk; // ivar for block
}
 
@property (nonatomic, copy) customBlock blk;  
 
- (void)useBlockFrom:(NSInteger)from until:(NSInteger)until;
- (void)doBlock:(customBlock)block;
 
@end
 
// .m
 
@implementation MyClass
 
- (void)dealloc
{
	[blk release];
	[super dealloc];
}
 
- (void)useBlockFrom:(NSInteger)from until:(NSInteger)until
{
	// if func is NULL then calling it crashes
	NSAssert(blk, @"Block needs to be set!");
 
	for (NSInteger i=from; i<=until; i++)
	{
		float result = self.blk(i);
		NSLog(@"f(%d) = %.2f", i, result);
	}
}
 
- (void)doBlock:(customBlock)block
{
	for (NSInteger i=1; i<=5; i++)
	{
		float result = block(i);
		NSLog(@"f(%d) = %.2f", i, result);
	}
}
 
@synthesize blk;
 
@end

It’s on the calling end that most of the changes are.

EDIT: Turns out that for a block property you need to use copy instead of assign, this will copy the block to the heap where it will continue to live. Blocks are treated like object and so you also need to release copy properties.

MyClass *m = [[MyClass alloc] init];
 
// inline
m.blk = ^(int x) { return (float)(x * 2.0); };
[m useBlockFrom:1 until:4];
 
// block variable
float (^blockTimesTwo)(int) = ^(int x) {
	return (float)(x * 2.0);
};
 
[m doBlock:blockTimesTwo];
 
[m release];

One thing that stumped me at first was that you have to make sure that the value you return from inside the block has to have matching datatypes with the block prototype. Without the specific typecast (float)(…) the result is a double and causes the compiler to complain about that.

The first use has the code to be put into the block property inline. The second use creates a block variable first and then passes this into the second method. This is just to show how it looks without using our special type. It works just the same using that and might be way easier to read:

// block variable
customBlock blockTimesTwo = ^(int x) {
	return (float)(x * 2.0);
};

So far we have exactly the same functionality as with function pointers, but now for the grande finale we’ll find why pros all around prefer blocks nowadays. Can you spot it?

MyClass *m = [[MyClass alloc] init];
 
float mul = 3.0;
 
// block variable
customBlock blockTimesAnything = ^(int x) {
	return (float)(x * mul);
};
 
mul = 10.0;
 
[m doBlock:blockTimesAnything];
 
[m release];

You see that within the {} of the block we still have access to the local variable mul. Furthermore the block sort of takes a snapshot of the variable at the time when it is defined. Setting mul to 10 afterwards has no effect, it’s still using 3. So besides of having the new variables available that you define in the as block parameters, you also get snapshots of local variables.

The typedef trick I have shown in this article greatly simplifies the code. One of the reasons why you generally don’t see this in the wild is that Apple seems to prefer to have the entire type definition in the block parameters so that you don’t have to look up what parameters the block has. It is really up to you whether or not you want this simplification at the cost of a bit of in-transparency. An onlooker would have to go into your header to find what the block parameters and return type should be.

Conclusion

The ability to access snapshots of local variables is what makes blocks orders of magnitude more useful if you want to enable other developers to provide “plug-in” custom functionality to your classes. The other guys can chose whatever ingredients they like at the time of setting the block and are no longer limited to using only the passed parameters or – in case of delegate methods – some ivars.

So if you have a choice then you might want to convert your function pointer properties into block properties, doing so might only require that you change a * to a ^. For this little tweak you open up a new world of customization possibilities for the happy developers who get to use your code.

It’s exactly these benefits why Apple has begun switching to blocks wherever they can, in most cases new methods where added (with blocks as parameters) that can achieve things where previously you had to painstakingly message back and forth.

We say: good idea!

Apple consciously separates mutable and immutable variants of classes in Objective-C. If you have an NSString you cannot modify it, only by creating a new one with additions to an old one. If you want to mutilate mutate the string itself you have to use it’s subclass NSMutableString. Internally those are the same CoreFoundation type, yet the design choice was to have an immutable class and add mutability methods in a subclass.

There are two items that are not immediately obvious if you start out learning to program Objective-C, that I’d like to chronicle. One of these stumped me just a few days ago.

Take for example the two methods to manipulate a string.

NSString *text = @"Constant String";
text = [text stringByAppendingString:@", really constant?"];
NSLog(@"%@", text);

You see that you can reuse the text variable and don’t have to create a new NSString pointer. This works because the first string is statically allocated by the compiler as part of the compiled code. The @”" is the shorthand for that. We have to assign the return value of stringByAppendingString to the pointer variable because this is a new NSString instance, marked as autoreleased.

How long do these strings live? The constant one stays around until the app gets unloaded from memory, i.e. when it is terminated. Though if you don’t hold onto the pointer to the static allocation then this amounts to a leak, the memory stays lost but you have no way to access it any more. But there’s nothing you can do about that. The autoreleased new string object will live until the next run loop, which is probably at the end of the method inside which you have place the code.

The mutable alternative:

NSMutableString *text = [NSMutableString stringWithString:@"Constant String"];
[text appendString:@", really constant?"];
NSLog(@"%@", text);

You see that appendString does not return a value because we don’t create a new object. Instead text itself is mutated. You still see the @”" because there is no shorthand to create an NSMutableString. So we still end up with “Constant String” statically allocated in memory and a new NSMutableString instance. But in this case this new instance is already created in the first line through the stringWithString. The following appendString does not create a new instance but mutates text.

What would the difference be if you had also declared text as NSString * in the second example? Not much, you would get a compiler warning that NSString might not have a appendString method because the compiler goes by the declaration. At runtime it’s still a pointer to an NSMutableString that ends up written into the variable, so messaging that with appendString works without problem. Also, since NSMutableString is a subclass of NSString you can use it anywhere you would need an NSString parameter, the additional mutability methods are irrelevant and will be ignored.

So much for a quick reminder about mutability basics. Besides of using these methods that create autoreleased new instances you can also copy objects. This is the first thing that somebody needs to tell you because it is counterintuitive. What does the following code produce?

NSMutableString *text = [NSMutableString stringWithString:@"Constant Text"];
id newText = [text copy];

You might assume that newText is now a mutable copy of text, but that would be wrong. copy always makes immutable copies, even if the original was mutable. To make a mutable copy use the mutableCopy method.

[NSString copy] = NSString
[NSMutableString copy] = NSString
[NSString mutableCopy] = NSMutableString
[NSMutableString mutableCopy] = NSMutableString

Anything with init, new or copy in the method name will not be autoreleased. So you have to make sure that you release it before you leave the current scope or else this will produce a leak, this time a preventable one.

When I started out to program Objective-C I made all synthesized properties on classes retain. But then you find – when looking at the documentation or Apple’s headers – that often properties are not retain, but copy instead. The reason being that a retained property is just a reference to an instance created earlier. If it’s an NSMutableString you pass into a method expecting an NSString that works nicely, but you can still mutate the string outside of the class. Which might lead to unexpected results.

Because of this any property there you want the “value” at this present time to be passed, you should make a copy instead. This copy is a snapshot at the time when the property was used to set the instance variable and no unexpected effects happen if you mutate the parameter afterwards. If you want a reference to the original and know for certain that you also will want to have access to subsequent updates of the passed instance, then retain will be your choice.

@interface MyClass : NSObject
{
	NSString *_name;
	NSDateFormatter *_birthdayDateFormatter;
}
 
@property (nonatomic, copy) NSString *name;
@property (nonatomic, retain) NSDateFormatter *birthdayDateFormatter;
 
@end

Both would have a release in the dealloc method. Usually you would simply @synthesize the methods for the properties defined above. But let’s look at how those would be written manually. Since I don’t like to rename the parameter of an overwritten setter, I usually prefix my instance variables with _. So you can imagine a @synthesize name = _name, property name equals instance variable name.

In this example we want to have the value of the name passed and preserved in our MyClass. But the dateFormatter might get a different format or locale during the runtime of our app, so we only retain it and don’t make a copy. Bear in mind, that to be able to copy any object the NSCopying protocol needs to be implemented. This means you have to implement the copyWithZone: for your own classes to support copy method. For mutableCopy the method to implement is in the NSMutableCopying protocol, i.e. mutableCopyWithZone.

The case where I run into an unexpected error was when I had an NSMutableArray instance variable that I wanted to also copy to copies of this class. Those lost mutability and when I tried to add items later I got the usual unrecognizedSelector exception.

#import "MyClass.h"
 
@implementation MyClass
 
- (void)dealloc
{
	[_name release];
	[_birthdayDateFormatter release];
	[super dealloc];
}
 
#pragma mark Properties
 
- (void)setName:(NSString *)name
{
	if (name != _name)
	{
		[_name release];
		_name = [name copy];
	}
}
 
- (void)setBirthdayDateFormatter:(NSDateFormatter *)birthdayDateFormatter
{
	if (birthdayDateFormatter != _birthdayDateFormatter)
	{
		[_birthdayDateFormatter release];
		_birthdayDateFormatter = [birthdayDateFormatter retain];
	}
}
 
@synthesize name = _name;
@synthesize birthdayDateFormatter = _birthdayDateFormatter;
 
@end

This is about the code that the @synthesize would create for you. The if serves to not release the ivar if it is the same because this might free the object. I’ve also seen some people ditch the if and instead use an autorelease, but that just feels wrong to me. Writing less code is not always a good thing.

You can see that the only difference here is that one uses a retain, the other a copy to preserve the object while it’s being used by MyClass. We still have the synthesizes because those create the getters for us, but these are the same for retain, copy and assign.

Now you know why mutability gets lost when you assign a mutable object to a copy property. Which was exactly what I did on the copyWithZone: implementation of DTCoreTextParagraphStyle.

#pragma mark Copying
 
- (id)copyWithZone:(NSZone *)zone
{
    DTCoreTextParagraphStyle *newObject = [[DTCoreTextParagraphStyle allocWithZone:zone] init];
 
    newObject.firstLineIndent = self.firstLineIndent;
    newObject.defaultTabInterval = self.defaultTabInterval;
    newObject.tabStops = self.tabStops; // copy
 
    return newObject;
}

So I overwrite the setter to substitute mutableCopy for the copy. Granted this is a very rare case to do that, but there is no other way to transfer a copy to a new object and still preserve mutability. Theoretically I could have also made the instance mutable lazily just before adding something new to it, but the code to do that is way longer than this overwritten setter and harder to understand.

Other smarter people have also stumbled over this. May this article prevent you from ever losing mutability when you depend on it.

In this article I want to summarize a couple of “barefoot techniques” for tuning your views. Sometimes you are painstakingly putting together a UI with multiple views and you cannot tell any more where one ends and another begins.

The debugger is not really working well on this because most of the interesting information about views is hidden behind properties and you cannot usefully inspect their current contents because properties are really methods the value of which is only set when you actually call them.

But I want to share some easy techniques that I started using so that I can get this visual puzzle untangled.

There are two kinds of approaches to debugging the visuals of a UIView. You can either look at some textual description of it and see if the numbers match up with what you’re expecting. Or you can mark and highlight a view such that you can visually judge whether it is in the right place.

Getting View Info

As any NSObject subclass UIView also has a descriptive description. It will tell you the view’s frame, class, autoresizing mask so you don’t have to fuss to extract the view’s frame with NSStringFromCGRect but you can simply NSLog the view directly.

NSLog(@"%@", view);

Output from this contains the current frame, autoresizing mask and info about the backing layer. Note: on iOS all views are layer backed and you can choose your own layer class to be used for any view. But if you don’t then CALayer is used.

<UIView: 0x4b15960; frame = (0 20; 320 460); autoresize = W+H; layer = <CALayer: 0x4b15a40>>

This simple technique can also reveal if you might inadvertently positioned the origin of the view on a non-integer value. This typically causes text to look blurry due to the antialiasing that get’s applied there. Here’s an example from one of my recent projects, you’ll have to open up the image in full resulution to see the difference.

Something like this might happen of you don’t set a view’s frame, but instead set it’s size via the bounds or center property. If the width or height are not dividable by 2 then you’ll get this affect, but you’ll only notice it when some perfectionist asks you why some label does not look as crisp as all the others.

NSLogging a simple view goes a long way, but it cannot reveal the hierarchy of views. One might be tempted to write some recursive category extension to also show a view’s subviews. But actually this work has already been done for us, by Apple!

UIView has a hidden (i.e. private) method named recursiveDescription which works just like description but recursively walks through a view’s children and their children and shows it in sort of a textual tree. You cannot have this in production code, but just for debugging all you need is to add a few lines to tell the compiler that this is a valid method.

@interface UIView (debug)
 
- (NSString *)recursiveDescription;
 
@end

Alternatively you can simply ignore the warning that the method might not exist. Output from this might look like:

<UIWindow: 0x4e31810; frame = (0 0; 320 480); opaque = NO; autoresize = RM+BM; layer = <CALayer: 0x4e319a0>>
| <UIView: 0x6015f60; frame = (0 20; 320 460); autoresize = W+H; layer = <CALayer: 0×6016040>>

You see a pipe symbol that shows which superview a view belongs to. You also see that the individual views are just shown with their regular description.  See a view’s structure of children might tell you some interesting things about stock views that come with the SDK. For example you would see that UIScrollView has some small UIImageViews as children: those are the scroll indicators!

Don’t forget to remove the NSLog statements you added just for debugging. You don’t have to remove all log statements for production code, but you can leave in log statements that give some interesting information. But if you leave in logs of view descriptions you deserve to be flogged.

Seeing the Dimensions of the View

If you have a view visible but are uncertain if all of it is showing then you can give it’s layer a distinct border color to visualize it. If you see the border all around and nothing is cut off then all is well.

By linking in the QuartzCore.framework and adding the <QuartzCore/QuartzCore.h> at the top of your file you gain access the properties of the UIView’s backing layer. Without that you can only get as far as the view’s layer, but all the properties of the CALayer class are defined in this header.

This allows you to set a border so you can see if your view is fully visible. This example sets it to red (note the use of CGColor instead of UIColor) and the border width to 1 so that it’s visible. I recently used this to see if a video preview layer is indeed showing fully.

view.layer.borderColor = [UIColor redColor].CGColor;
view.layer.borderWidth = 1.0;

Another technique that I love to use, especially when working with multi-page scrollviews is to have the child views and the parent views all with different background colors. Now you could just set the backgroundColor for all but I found that it gets boring after a while because you always think of the same primary colors. Red. Green. Blue. Yellow. Purple if you feel fancy. Wouldn’t it be much nicer if you saw a fresh random color every time you start the app again to see if the views line up?

A simple category on UIColor gives you a randomColor method so that you never run out of ideas for temporary background colors.

// this in the header
@interface UIColor (debug)
+ (UIColor *)randomColor;
@end
 
// this in the implementation
@implementation UIColor (debug)
+ (UIColor *)randomColor
{
    CGFloat red = (arc4random()%256)/256.0;
    CGFloat green = (arc4random()%256)/256.0;
    CGFloat blue = (arc4random()%256)/256.0;
 
    return [UIColor colorWithRed:red green:green blue:blue alpha:1.0];
}
@end

Never again those boring primary colors!

Conclusion

These are the two methods of how to get useful descriptions of views as well as how to visually mark views so that you can be certain that they are in the right spot. If you know any others that are also fun, then please comment.

I found myself in need of splitting an NSString into paragraphs. Or more precisely to analyze a string and find the NSRange for each such paragraph. At first I wrote a C-style function that looked for the ‘\n’ in the NSString’s utf8String, but it turns out that this approach has problems with multi-character UTF8 sequences.

For the longest time I shirked from using Blocks, which became part of iOS with iteration 4.0. But since this project will have a minimum deployment target of higher than this, I gained the ability to use a block-based enumeration function to achieve my goal with a record minimum of code.

Serendipity helped me find that with this new substring enumeration function you can enumerate over a string’s substrings by words, lines, paragraphs or even sentences. And if you feel so inclined even reverse.

Here’s my  code:

- (void)buildParagraphRanges
{
    // get naked NSString
    NSString *string = [self.attributedString string];
 
    // entire string
    NSRange range = NSMakeRange(0, [string length]);
 
    NSMutableArray *tmpArray = [NSMutableArray array];
 
    [string enumerateSubstringsInRange:range options:NSStringEnumerationByParagraphs usingBlock:^(NSString *substring, NSRange substringRange, NSRange enclosingRange, BOOL *stop)
     {
         NSValue *value = [NSValue valueWithRange:substringRange];
         [tmpArray addObject:value];
     }
     ];
 
    self.paragraphRanges = tmpArray;
}

As you can see there is an attributed string involved so at the very top I’ll have to get the plain NSString from this. In the middle I’ll do the enumeration and by converting the substringRange into an NSValue I can add it to an NSArray. This way I have the index location and length of each paragraph handy when I need it.

We don’t know what goes on under the hood, but not having to do any kind of calculating to get the ranges can only be a good thing.

Blocks work such that all local variables currently in scope when the method is called are still available inside the block. Additionally the parameters inside the ^() are also filled in for you to do with as you please. The block is sort of a function pointer on steroids. Since it is a parameter in this method you could even set up the block elsewhere and then just pass the block variable to it.

But let’s not go overboard here. This is only the second time I get to use a method with a block and I’m lovin’ it.

I was setting up a project with one submodule on my working iMac and was wondering how to do this most quickly. After tweeting the approach I had found, there where quickly some very smart people responding on how to do that better. I found this kind of crowd-sourced incremental improvement exhilarating, so I’m sharing it with you.

Being a git beginner I started out with the slowest approach, one step after each other.

git clone URL
cd FOLDER
git submodule init
git submodule update

This first clones the main project, then setup the submodule and then we have an update pull the submodule code.

Simon Ljungberg was the first to point out that I could save one step by contracting both submodule commands like this.

git clone URL
cd FOLDER
git submodule update --init

The question did arise why the init would be necessary, but I found that if you don’t init the submodule then the update does not do anything and the submodule folder stays empty.

We already thought to have found the shortest possible method, comes along Alex Blewitt and informs us that – provided you use git 1.6.5 or higher – the whole affair can also be shortend in a single line:

git clone --recursive URL

Of course we all use a git that is way newer than this minimum requirement, mine is 1.7.3.4 as of this writing. So let’s take notice of the elegance of the recursive option and use this from now on.