RegexKit Change Log


This document contains a list of the changes for each version of RegexKit.

This is not a comprehensive manifest of changes between releases. For an exact list of changes between releases, the SVN repository should be used.


Change Log for 0.6.0 Beta

DTrace and

NSObject Class Category Additions

Added the following category method additions to the NSObject class for concurrent, multithreaded evaluation of regular expressions in collections:


Using the standard Cocoa localization techniques, two files were created that contain the strings to be localized:

To assist translators, the comments in the files contain the relative importance for localization of the various strings. For example, strings that are likely to be displayed directly to end users are considered the highest priority, strings that are useful and frequently used by programmers, such as an objects description, are considered medium priority, and strings used for exception reasons are the lowest priority.

The localization of RegexKit will depend on user contributions.
Users who wish to contribute a localization must read and agree to the terms in User Contributions.

Stack Protection

As I began the final checks of the 0.6 distribution, I ran in to a fairly serious problem when I began executing the unit tests on 10.4 with the framework binary that was to be distributed. With 0.6, the framework is built with the Xcode 3.0 GCC flags -fstack-protector-all -mmacosx-version-min=10.4. However, as with the DTrace issue under 10.4, when you specify the minimum OS version required, it doesn't actually honor the request in any way. The stack protection feature works by having the compiler perform a bit of magic. Part of that magic is to copy some bytes on to the stack and then check that those bytes are unaltered when the function returns. The symbol __stack_chk_guard contains the bytes copied and checked. These bytes are initialized with bytes from /dev/urandom during the C runtime initialization that happens before main() is called, and all of this lives in libSystem.dylib. Of course, none of this exists in 10.4's libSystem.dylib and loading the RegexKit.framework at run time promptly aborts executing due to unresolved symbols.

Lovely. Apple Bug ID# 5712021.

The solution is to borrow the stack protection initialization code from Libc-498/sys/stack_protector-obsd.c. The initialization code is modified to check and see if the __stack_chk_guard symbol is defined anywhere else. If it is, it's obviously running on a system that includes all the proper support so it copies the systems version of random bytes at __stack_chk_guard in to its local copy, which is used by RegexKit. On 10.4, __stack_chk_guard is not defined, so it falls through and initializes with random data from /dev/urandom, just as 10.5 would have done.

This code lives in the file RK_stack_chk.c and has a special target, Stack Protector, because it needs to be compiled without the -fstack-protector-all flag. The initialization function has the attribute of __attribute__((constructor)) so that it is executed immediately at load time.

So, I managed to save full stack protection for both 10.4 and 10.5.

Other changes

Bug fixes

Change Log for 0.5.0 Beta

DTrace Instrumentation

Created several macro wrappers for DTrace probe insertion in RKRegexPrivate.h. This allows a simple macro, RK_PROBE(), to be placed in the code where a probe point is desired. The macro will rewrite to nothing if ENABLE_DTRACE_INSTRUMENTATION is not defined. If it is defined, then it re-writes to an if() {} statement that first calls the 'Is Enabled' probe function, and only if the probe is enabled does it go through the work of assembling the arguments and calling the actual probe firing function.

A few modifications here and there to assist in passing information via the probe efficiently. For example, RKRegex has a new private function, regexUTF8String(), which will create a copy of the regex string in UTF8 form and stores the pointer in an ivar. This allows for lazy evaluation, it's only created if the function is called, and right now the function is only called when a probe requires it. Once created, the function obviously returns the pointer to the already created UTF8 string.

The majority of probes inserted at this time have to do with extracting timing information. RKRegex had the PCRE compile and match functions instrumented, and RKCache had the add, insert, lookup, and clear methods instrumented. A probe named 'PerformanceNote' was created and sprinkled in a few places. This is for passing text strings to a dtrace listener at certain spots in the code which have a negative performance impact. For example, if we're unable to get direct access to the backing store of a string, and are required to call [string UTF8String] as a result, a PerformanceNote is generated. Instruments

To make use of the new dtrace probe points, several .instruments were created for This tool shows a hell of a lot of potential. It's got some nasty bugs when creating your own .instruments, but it's a 1.0 product, on top of a whole new kernel facility (dtrace), so it's a bit understandable. While there's documentation for using the application proper, there's just a token mention of creating your own probes. From digging through the guts to create the RegexKit functionality, I'd say that a third-party plugin architecture is on the way. Not just beatified dtrace .d scripts, but the ability to write objective-c plugins with custom interfaces. The ObjectAlloc tool is a good example of this.

I hope Apple hurries up with the third party plug in support. While you can get a hell of a lot done with the few probes and .instrument scripts I put together, I can see creating some incredibly powerful performance tools with a custom interface. It would almost be like Shark, but for regular expressions. There's only so much you can do with dtrace scripts, but you can do so much more if you can post process the raw data and make changes on the fly. Say you're having performance problems with your regex, and you want to know why. And you have a lot of regex, and you're note sure which one is causing the problem. Enabling the matching probes would give you a stream of raw data, and you could post process it to aggregate timing information on a per regex basis. Regexes that take a long time to perform a match would bubble right to the top, and the great thing is you can even get snapshots of the buffers involved. And once you start to narrow things down, and you're only interested in a single regex out of all of them, because each regex has a unique hash, this can be easily placed in the dtrace predicate which means you're only collecting data on the ones you're interested in, the others will have just a minor performance impact since the dtrace kernel module has a constant, cacheable predicate that comes down to a simple integer compare.

These instruments, and the probe points, should be considered expiremental. The specifics are very likely to change as I gain more experience with both DTrace and I've marked the DTrace probe points 'Unstable', which is an advisory to users of the probe points that things are likely to change from release to release. DTrace has a versioning scheme for the various parts of the DTrace interfaces to help DTrace users gauge how likely it is their scripts and what not are to break between patches, releases, etc.

NSArray Additions

Added indexSetOfObjectsMatchingRegex: and indexSetOfObjectsMatchingRegex:inRange: to the NSArray category additions.

NSData Class Category Additions

Added the following category method additions to the NSData class:

See NSData RegexKit Additions Reference for additional information.

Other changes

Bug fixes

Change Log for 0.4.0 Beta

Unicode changes

Previously, most convenience methods would use RKCompileDupNames as the compile time options. This was removed (you can now specify this via the regex string with (?J)), and replaced with (RKCompileUTF8 | RKCompileNoUTF8Check). This activates various Unicode behaviors that I expect most people would assume to be "on" automatically. It also changes some matching behavior, but both of these are probably only important if you're making use of Unicode functionality.

NSRange results changes

Previously, RegexKit always returned the ranges as reported by the pcre engine. This could cause problems for Unicode strings because PCRE uses byte oriented UTF8, where as Foundation defines a character to be the characters UTF16 representation. The majority of common characters map to a single UTF16 character, but anything beyond ASCII (<128) takes multiple bytes in UTF8.

The Foundation additions (NSString, etc) now accept and return results in UTF16 'coordinate space'. The RKRegex class continues to work in the native UTF8 'coordinate space'. Mapping between the two can be costly, and at this time no particular optimizations are made to improve performance.

The following two functions map ranges for a string from one coordinate space to the other:

RKConvertUTF8ToUTF16RangeForString(string, range)
RKConvertUTF16ToUTF8RangeForString(string, range)

The buffers internally are always in UTF8 format, so to map between the two these routines start at offset 0 and advance both a UTF8 offset (which is always the current byte offset), and the UTF16 offset (which is less than or equal to the UTF8 offset, depending on how many multi-byte UTF8 characters have been processed).

The Mac Roman and ASCII string encodings always have a 1:1 mapping of UTF8 offsets and UTF16 offsets, since there can be no multi-byte long characters in these encodings, so mapping in this case is very fast (returns the requested range unaltered).

Ranges as returned by the various Foundation extensions can now be used interchangeably with other Foundation methods that accept a NSRange as an argument.

Other changes

Bug fixes

Change Log for 0.3.1 Beta

Xcode 3.0 DocSet Subscriptions

Added DOCUMENTATION_DOCSET_FEED_SCHEME and DOCUMENTATION_DOCSET_FEED_URL to the build settings which control which URL is placed in the assembled DocSet for subscriptions and auto-updates. Realized that it'd probably be helpful to have the versions in question be displayed, so updated a number of templates to place the PCRE and RegexKit versions in the zebra banners. SCHEME and URL are separate because http:// is interpreted as a comment start in .xcconfig files. :(

Change Log for 0.3.0 Beta

Xcode 3.0 DocSet Documentation

The DocSet documentation is essentially the HTML documentation with a few minor tweaks. It is created by the scripts:

Documentation/Scripts/ is pretty horrific, and was kludged together very quickly. It creates the required Nodes.xml and Tokens.xml by extracting the information from the documentation database and rewrites the anchors in the .html files to use the //apple_ref/ style format. This is not used in the normal HTML documentation because it isn't valid HTML (tidy throws a fit and there's hundreds of warnings generated.) However, the DocSet tool chain appears to be much happier with //apple_ref/ style anchors, so the legal html name anchors are rewritten.

Once all these pieces are in placed, and a Info.plist is created, the docsetutil command is run to build the DocSet index and that's pretty much it.

Garbage Collection

Mac OS X 10.5 Garbage Collection is controlled by the C Preprocessor flag ENABLE_MACOSX_GARBAGE_COLLECTION. A number of macros were created to ease various portability issues. Of special note is the framework global variable RKRegexGarbageCollect. If ENABLE_MACOSX_GARBAGE_COLLECTION is defined, then RKRegexGarbageCollect is a int32_t defined in RKRegex.m. If ENABLE_MACOSX_GARBAGE_COLLECTION is not defined, then RKRegexGarbageCollect is #defined in RegexKitPrivate.h as 0. Also in RegexKitPrivate.h are a number of macros for managing various retain / release / autorelease.

If ENABLE_MACOSX_GARBAGE_COLLECTION is defined, then RKRegex.m, RKCache.m, and NSString.m will all check and see if the class NSGarbageCollector exists, and if so, check to see if garbage collection is enabled in their +load methods.

If ENABLE_MACOSX_GARBAGE_COLLECTION is not defined, then most of the macros become a compile time static conditional which the compiler will optimize away, so compiling on pre-10.5 or GNUstep is branchless. It was the best I could come up with for maximum portability while still maintaining readability.

In RKCache.m, if ENABLE_MACOSX_GARBAGE_COLLECTION is defined, AND if garbage collection is enabled at load time, then the class switches to using the new NSMapTable class for managing the cache. It places cached RKRegex objects in a NSMapTable with a weak reference to the object. This should allow for auto trimming of the cache to the working set. Not sure how it will work out in real life.

64 bit support

A set of macros was adopted to ease multi-platform 64 bit issues. They are essentially parallel to the Leopard NSInteger / NSUInteger types, except that they are cpp macros. They have the names RKInteger and RKUInteger / etc. They are re-written to either NSInteger or int depending on the detected run time environment. This allows for type checking / compatibility to be maintained for each environment. GNUstep still has things such as hash defined as unsigned int, and for some of our objects we return the pointer to the instantiated object as the hash (there can be only one) and this causes truncation on GNUstep, but it should be OK.


Cleaned up various GNUstep pieces here and there while putting the 64 bit stuff in. Should build much cleaner now, with only a handful of warnings when compiling on a 64 bit platform for various hash functions, but this shouldn't be a problem in the real world.

RegexKit.framework Info.plist version bug fix

The Info.plist in RegexKit 0.2.0 had a bug in the Info.plist file. The settings for the keys CFBundleShortVersionString and CFBundleVersion where set to the variable ${REGEXKIT_CURRENT_VERSION} when they should have been set to ${PROJECT_CURRENT_VERSION}. Since the variable ${REGEXKIT_CURRENT_VERSION} was undefined, this ended up expanding to nothing, so the correct version number is missing. This should be a fairly trivial bug in the grand scheme of things.

Mac OS X Installer .dmg distribution

RegexKit 0.3.0 marks the introduction of a Mac OS X Installer based distribution. It bundles together the Mac OS X Binary executable RegexKit.framework, the Documentation, and the source code in to one convenient package. Each is grouped in to their own package, and by default RegexKit.framework and Documentation are selected to install while the source code package is not. There's some javascript code to disable a package if the version that is already installed is newer than the version that is attempting to install. Also, there's some logic so that the source code package will automatically default to selected if it detects that a previous install of the source code has been performed, thus auto upgrading without having to think about it.

The Installer has a volume check script that requires 10.4 or later in order to install on a volume, and an installation check that checks to see if the file /Developer/Applications/ exists. If it does not, it presents a warning that the Xcode Development Tools should be installed, but does not stop the installation.

There's three main tuneables in the build configs:

Distribution Build Settings
Build SettingDefaultValid Values

Since the overall distribution package gets placed on to a .dmg disk image, which is then BZip2 compressed (-format UDBZ), compressing the packages with gzip actually causes the final .dmg size to grow by about ~60-70KB, or roughly 10% in size. So, by default, we leave the packages uncompressed and let the disk image BZip2 deal with all the compression.

The other option, DISTRIBUTION_INCLUDE_PCRE_PACKAGE, toggles whether or not the PCRE tarball used to build the framework is included in the Installer distribution. By default the choice is NO since this causes the final .dmg disk image to nearly double in size. There's some extra javascript logic so that the PCRE distribution is if and only if the source code package is selected to be installed. If the user selects the source code package, the PCRE package then auto selects itself to be installed as well. The PCRE package version is the PCRE Major.Minor, not the RegexKit version. Unlike the others it will allow an earlier version to be installed 'on top off' a newer version since this is largely harmless- Each PCRE tarball distribution is separate from the others.

The file that creates the top level distribution package distribution.dist file is in Source/Build/Scripts/ It handles the logic of selectively inserting the PCRE choice depending on whether or not DISTRIBUTION_INCLUDE_PCRE_PACKAGE is set to YES. It also gathers some necessary information, such as a packages installed size, by using perls Objective-C / Foundation bridge to read the individual packages Info.plist files and extracting the appropriate keys.

For each package (of which there are four, RegexKit.framework binary package, Documentation package, Source code package, and optionally the PCRE distribution package) there exists two .plist files, one info.plist and one desc.plist. The important one is the info.plist, and it contains the bundle identification and version information. The version information is replaced dynamically at build time by the Source/Build/Scripts/ script. That script makes a copy of the .plists to a private build staging area. Then, it calls the script Source/Build/Scripts/ which is a very simple perl script that again uses the Objective-C / Foundation bridge to take three arguments: the .plist file to operate on, the key to update, and the value to set the key to. uses this to update all the .plists to the current RegexKit version as required by the package .plist format. An additional .plist file, RegexKit_mpkg.plist, is largely for cosmetic purposes. It gets the version updated as well, but its sole purpose is to provide a version string to the finder when a Get Info is performed and that's it.

The text that's displayed during the installation process comes from four files, though only two are currently used. These are located in Source/Build/Packagemaker/Resources/..., which is a localized .lproj style directory for localization purposes. Currently only English.lproj is present. The files are currently simple .rtf files, but .rtfd and .html are supported as well. Any changes to the file types would have to be updated in the file (it should be obvious where). Only ReadMe.rtf and License.rtf are currently used, but both Welcome.rtf and Conclusion.rtf can be activated trivially by editing They get copied in to the distribution package right now, but the necessary lines in distribution.dist that is created by are missing.

The tuneable DISTRIBUTION_DEFAULT_INSTALL_DIR is self-explanatory.

Minor bug fixes

Build system tweaks

Replaced the debug: portion of various script status lines with note: instead. This is mostly cosmetic. In the Makefile.pcre, added the makefile file name and line number of the message so clicking on it will take you right to that line. Updated most status messages to always output the filename and line number with the status indicator.

The PCRE build target is largely driven by Makefiles. The scripts that invoke `make` were modified to set the -j argument, or the number of jobs to run in parallel, and the -l argument, or the load average where no new jobs can be started. The values for these parameters are derived from the sysctl variable hw.activecpu. The -j parameter is set to the number of active CPU's, and the max load is set to the number of active CPU's + 2. This should allow those of you with multiple CPU's to build the PCRE target faster. However, sometimes building in parallel causes certain make dependency issues to crop up that don't happen when running sequentially.

This can be disabled with the tuneable PCRE_PARALLEL_BUILD = NO.

The Distribution target was also partially parallel build enabled as well.

Change Log for 0.2.0 Beta

End User Framework Visible Changes

The nature of changes for this release are almost all documentation or build system related. Only a handful of changes resulted in any API level changes, the largest probably being the PCRE 7.3 to 7.4 change.

Only one change stands out that may impact existing users of the framework. The Alpha version did not contain an explicit version of the framework, and used the default of 1 for the version that was used in the Mach-O library version information.

This release begins the use of a framework versioning system. It is a standard "three point" version type system comprised of a Major, Minor, and Point version. These versions are reflected in the Mach-O library version.

Refer to the RegexKit Programming Guide section on Release Information or more information.

Distribution Bundles

The file bundles for a distribution release has changed. There is now a Mac OS X oriented binary + documentation release and a no binaries, source only + documentation bundle.

PCRE Version Changed

The PCRE library that the framework builds against was updated to 7.4.

const Use Expanded

Where appropriate, the const keyword was added. Most of this was marking both the pointer and data pointed by the pointer as const.

For example:

const NSRange *matchRanges

changed to

const NSRange * const matchRanges

Framework Initialization

Previously, initialization that was required at load time was performed with functions marked with the attribute constructor. This was replaced with the more portable, but functionally identical, Objective-C +load method.

C99 restrict Added

A new pre-processor macro, RK_C99(), was added that causes the macros argument to either be replaced as-is, or removed depending on whether or not __STDC_VERSION__ >= 199901L (ie, C99). This is used to selectively allow the C99 restrict qualifier through for various function / method prototypes. This allows projects who are currently not using C99 to transparently use the framework without warnings/etc.

Framework Version

The build setting ${PROJECT_CURRENT_VERSION} now contains the frameworks version. It is composed as the dot separated form of ${PROJECT_VERSION_(MAJOR|MINOR|POINT)}, ie 0.2.0.

The version of the framework is now incorporated in to the Mach-O shared library version information. The project build setting ${PROJECT_CURRENT_VERSION} sets the dynamic libraries version, and ${PROJECT_COMPATIBLE_VERSION} sets the compatibility version.

Project Build Settings

The various top-level project settings were moved out from the Xcode project.pbxproj file and in to a separate .xcconfig file. The Xcode project build settings were then set to be 'based on' this file. The file is at Source/Build/Xcode/RegexKit Build Settings.xcconfig. This makes it much easier to add and modify the build settings. It also allows comments to be placed in the file, documenting the settings at the same time.

These settings can then be easily over-ridden at either the project or per-target level within Xcode by selecting the appropriate Project > Edit .... menu item and then simply entering the desired over-ride.

A notable setting is GCC_OPTIMIZATION_LEVEL. Xcode normally only allows for 0|1|2|3|s (ie, -O1, etc). However, the Apple branch of GCC also supports a 'z' level optimization which is loosely defined as 'Like -Os (-O2, preferring smaller code size to larger/faster code), but aggressively preferring smaller code size to any speed advantage.'. There was approximately 24KB smaller executable size difference for each step in 2 to s to z settings and no timing difference reported by the timing unit test, so z was selected for a total executable size reduction of ~49KB.

Executable Size Reduction Table
OptimizationSizeDead Code StrippedDifferenceOptimization Difference
-O2495776 / 484KB494608 / 483KB1168 / 1KB
-Os475152 / 464KB469884 / 458KB5268 / 5KB24724 / 24KB
-Oz450624 / 440KB445356 / 434KB5288 / 5KB24528 / 24KB

Total Optimization plus Dead Code Stripping savings: 50420 / 49KB ([-O2] 495776 - [-Oz dcs] 445356)

PCRE Build Target

Substantial updates to the PCRE build target. There is very little user visible changes, however.

The PCRE target has a 'Run Script' build phase which executes the Source/Build/Scripts/ script. Since this is a target that is outside the control of Xcode, it must perform several dependency checks to ensure that what has been built is valid given the current build settings and target dependencies. This script performs those checks manually by comparing the output of `env`, Source/Build/Scripts/, and Source/Build/Makefiles/Makefile.pcre to a copy of those settings / files that were used when the target was built. Any change effectively causes the target to be rebuilt from scratch (ie, rm -rf / untar) to ensure any dependencies are accurately reflected in the built library.

Altered the build system to place all temporary files used during the build process, including the untaring of the distribution, in to ${PCRE_TEMP_ROOT}, which translates in to build/ CONFIGURATION being either Debug or Release.

Build System Changes

Numerous build system changes, too many to list and nearly all are not important to end users. Examples are the more robust PCRE build subsystem, clean up of the documentation build system, clean up of the aspell spell checking of documentation, clean up of tidy checking of the HTML output, etc.

SVN at

This release represents the first import to the SVN repository at

File Encodings

Updated/converted all appropriate files to Unicode UTF8 encoding and updated the Xcode reference to reflect the new encoding.

Change Log for Alpha


RegexKit project hosted by: