RegexKit Change Log

Change Log for 0.6.0 Beta

DTrace and Instruments.app

• Added the following probe points: BeginSortedRegexSort, EndSortedRegexSort, BeginSortedRegexMatch, EndSortedRegexMatch, SortedRegexCache, and SortedRegexCompare. These instrument the new, framework private, RKSortedRegexCollection class.
• Added two new Instruments.app instruments, RegexKit_sortedCollection_cache.instrument and RegexKit_sortedCollection_timing.instrument, to collect information on the behavior of RKSortedRegexCollection and help quantify any benefit from the sorted regex matching.

NSObject Class Category Additions

Added the following category method additions to the NSObject class for concurrent, multithreaded evaluation of regular expressions in collections:

• - anyMatchingRegexInArray:
• - anyMatchingRegexInArray:library:options:error:
• - anyMatchingRegexInSet:
• - anyMatchingRegexInSet:library:options:error:
• - firstMatchingRegexInArray:
• - firstMatchingRegexInArray:library:options:error:
• - isMatchedByAnyRegexInArray:
• - isMatchedByAnyRegexInArray:library:options:error:
• - isMatchedByAnyRegexInSet:
• - isMatchedByAnyRegexInSet:library:options:error:

Localization

Using the standard Cocoa localization techniques, two files were created that contain the strings to be localized:

• Localizable.strings
• pcre.strings

To assist translators, the comments in the files contain the relative importance for localization of the various strings. For example, strings that are likely to be displayed directly to end users are considered the highest priority, strings that are useful and frequently used by programmers, such as an objects description, are considered medium priority, and strings used for exception reasons are the lowest priority.

Stack Protection

As I began the final checks of the 0.6 distribution, I ran in to a fairly serious problem when I began executing the unit tests on 10.4 with the framework binary that was to be distributed. With 0.6, the framework is built with the Xcode 3.0 GCC flags -fstack-protector-all -mmacosx-version-min=10.4. However, as with the DTrace issue under 10.4, when you specify the minimum OS version required, it doesn't actually honor the request in any way. The stack protection feature works by having the compiler perform a bit of magic. Part of that magic is to copy some bytes on to the stack and then check that those bytes are unaltered when the function returns. The symbol __stack_chk_guard contains the bytes copied and checked. These bytes are initialized with bytes from /dev/urandom during the C runtime initialization that happens before main() is called, and all of this lives in libSystem.dylib. Of course, none of this exists in 10.4's libSystem.dylib and loading the RegexKit.framework at run time promptly aborts executing due to unresolved symbols.

Lovely. Apple Bug ID# 5712021.

The solution is to borrow the stack protection initialization code from Libc-498/sys/stack_protector-obsd.c. The initialization code is modified to check and see if the __stack_chk_guard symbol is defined anywhere else. If it is, it's obviously running on a system that includes all the proper support so it copies the systems version of random bytes at __stack_chk_guard in to its local copy, which is used by RegexKit. On 10.4, __stack_chk_guard is not defined, so it falls through and initializes with random data from /dev/urandom, just as 10.5 would have done.

This code lives in the file RK_stack_chk.c and has a special target, Stack Protector, because it needs to be compiled without the -fstack-protector-all flag. The initialization function has the attribute of __attribute__((constructor)) so that it is executed immediately at load time.

So, I managed to save full stack protection for both 10.4 and 10.5.

Other changes

• Modified the RKPrettyObject… macros to be a function instead of a preprocessor macro. This shaves ~30K off the executable, or 4-7K per architecture.
• Added a PerformanceNote if pcre_study was able to optimize the regular expression.
• Added XCODE_CFLAGS_* and PCRE_CFLAGS_* to RegexKit Build Settings.xcconfig to add OS specific CFLAGS to Xcode and PCRE built sources.
• Using the new CFLAGS added above, added -fstack-protector-all to the Mac OS X 10.5 targets. -fstack-protector-all is a GCC feature that Apple backported from later releases of GCC that can catch many stack smashes.
• Modified projectExportedSymbols to export the new RK…ErrorKey, RK…ErrorDomain, and RKRegexPCRELibrary NSString constants.
• Added RegexKit private extensions to NSException and NSError to simplify creating localized versions of each for simple instantiations.
• Added several RegexKit private pre-processor macros to simplify creating localized strings. These are similar in nature to Foundations NSLocalizedString family of macros.
• Converted RKConvertUTF8ToUTF16RangeForString and RKConvertUTF16ToUTF8RangeForString to StringBuffer equivalents so internal routines could call the more efficient StringBuffer functions. The plain string methods became wrappers around the StringBuffer functions.
• Created a RegexKit private function RKLocalizedStringForPCRECompileErrorCode that returns an error description string that is better suited to NSError descriptions that end users see. The localized strings ultimately come from the frameworks bundle resource file pcre.strings.
• Created a RegexKit private global variable, RKFrameworkBundle that is created by RKRegex at +load time that provides the means to access localized strings.
• Created a private function that creates a NSException from an NSError in the same way that initWithRegexString:options: method previously did since that method now creates and returns an NSError for most error conditions.
• Modified RKRegex isEqual: to directly access the comparison objects instance variables if it's a RKRegex class object for speed.
• Added a RegexKit private function to return the number of bytes for a UTF8 encoded character at a given pointer, and a function that returns the range for a UTF8 encoded character at a given offset of a pointer. If the offset points to the middle of a UTF8 encoded character, it will back up to the start of the UTF8 character that is at the given offset, then determine the number of bytes required to represent a single Unicode code point encoded in UTF8 format.

• Expanded the locking strategies that the framework private locking class provides. Previously the locking class only provided a blocking acquire strategy. Strategies now available include:

  • Try for reading.
  • Try for writing.
  • Try for writing, then try for reading.
  • Try for writing, then blocking acquire for reading.
  • Blocking acquire for reading.
  • Blocking acquire for writing.

  This change was made to permit certain caching functions to be non-blocking when a thread is unable to acquire the requested degree of mutual exclusion for a shared resource immediately.

• Added a new private locking class, RKConditionLock, that is similar to its Foundations counterpart. This was done to provide a faster function call interface and enhanced locking methodologies, along with NSTimeInterval based relative times instead of the much more expensive NSDate object based times. Using NSDate objects incurs a significant performance penalty because of the object creation and destruction overhead to ultimately convey a double value as an argument. By directly providing a relative time double passed parameter argument, that overhead is completely bypassed. The two functions RKFastConditionLock and RKFastConditionUnlock provide all the functionality, which can be called directly, and the object oriented interface methods are just stubs for these two functions. These functions and objects are not exported and are framework private.
• Consolidated a lot of common logic for the locks in to the two functions RKFastMutexLock and RKFastMutexUnlock. The RKLock class was moved to this common code base, but for the time being, RKReadWriteLock remains unaltered.
• The pthread mutexes created by RKLock and RKConditionLock are no created with the pthread mutex attribute PTHREAD_MUTEX_ERRORCHECK which causes extra sanity checks to be performed, such as the same thread locking a locked mutex, unlocking an unlocked mutex, or a thread attempting to unlock a mutex that was locked by a different thread.
• Updated the license displayed in the installer to include the PCRE license explicitly.
• Added the PCRE license to the project root directory LICENSE file.
• Added the LICENSE file to the RegexKit Framework targets Copy Bundle Resources build phase so the license is present in the frameworks Resources directory.
• Added RKAtomic…Barrier macros / functions that perform full memory barrier semantics for architectures where this makes a difference.
• Changed the BeginLock and EndLock dtrace probes second argument from int to NSInteger to match the information now provided by RKLock and RKReadWriteLock. The locking strategy requested and the final lock level acquired are now reported instead of a simple boolean read / write indication.
• Added BeginLock, EndLock, and Unlock to RegexKit.usdt.
• Split the header file RegexKitPrivate.h in to several files, RegexKitPrivateAtomic.h, RegexKitPrivateDTrace.h, RegexKitPrivateLocalization.h, RegexKitPrivateMemory.h, RegexKitPrivateThreads.h, and NSStringPrivate.h.
• Changed RKRegex so that all of the class initialization takes place in the +initialize method. Previously, some initialization took place in the +load method which meant it was executed even if the class was ultimately not used. Also added guard checks at some function entry points since calling a function would not trigger the +initialize behavior.
• Updated generateHTML.pl to properly iterate over the groups in the Constants Table of Contents entry. Previously, this was manually updated for each new group.
• Updated Copyright © for 2008.
• Added the ability to specify the availability (introduced in, deprecated in, removed in version, etc) to the documentation system with the file availability.sql. Updated the docset tools to use this information when creating the Tokens.xml file.
• Altered the structure of the various unit tests. There was a lot of code that, over time, had managed to get replicated in several files and then drift apart over time. This was consolidated in to RKTestCase.m, which creates a common base object that is a subclass of SenTestCase that the RegexKit unit test objects inherit from.
• Removed the Mac OS X malloc stats functionality from the unit tests NSDate object. Also removed the NSHighResTimeInterval type and replaced it with NSTimeInterval as both were of type double.
• Added sortedRegexCollection.m to hold tests related to the new multithreaded sorted regex collection functionality.

Bug fixes

• In RegexKit.usdt, the PerformanceNote probe arguments for Severity and generalStartEnd were swapped. This was corrected.
• Fixed a type-o in RegexKit_match_timing.instrument. Somehow, "%x" was changed to b which caused the instrument to not be legal to parse and thus not show up in Instruments.app.
• Fixed some errors in some HTML files and the print.css style sheet that would cause some titles to be negatively offset past the printable border.
• Fixed a Firefox display bug in common.css that caused sourcecode boxes to not be <pre> formatted.
• The RKReadWriteLock class would harmlessly display an incorrect count of the number of spurious errors attempts out of a maximum number of attempts.
• The RKReadWriteLock class would harmlessly increment an internal debugging counter twice if it was unable to acquire a write level lock on the first attempt.
• The RKReadWriteLock class would incorrectly update an internal ivar regarding the read or write condition of the lock regardless of whether or not a pthread error prevented the lock from being acquired.
• Changed the RKRegex retain and release methods to use the RKAtomic…Barrier routines to enforce a full memory barrier. This may have led to race conditions on architectures in which this makes a difference, such as the PowerPC architecture, and when multiple CPUs are attempting to update the same memory location simultaneously.
• Changed the framework internal RKRegexFromStringOrRegex… functions so that when an object is determined to be a member of the RKRegex class, but the options specified in the instantiated regex do not match the required options, the class of the instantiated regex is used to create a new RKRegex with the required options instead of using the base RKRegex class. This would make a difference only to a subclass of RKRegex that over-rode the object creation process.
• SourceForge bug 1850418 - 'Error linking under 10.4'. This issue is covered in Release Information - Release Notes for 0.6.0 Beta. Update: Resubmitted this bug to Apple as bug # 5708443. The original bug report was closed as Behaves correctly. The justification given is "It is the same for linking - the 10.4 based linker errors out when it sees things it does not understand in the 10.5 libSystem.dylb. (sic)" and "In regards to the second post, there is no bug in the 10.5 linker. It is fine to link against the 10.5 libSystem.dylib with -macosx_version_min=10.4. The dtrace section is part of the implementation of libSystem.dylib. It is not part of the interface to the dylib (but the 10.4 linker does not know that)." To be honest, I'm sort of at a loss as to how the engineer managed to leap from the bug to the justifications given for closing it out as Behaves correctly.
• SourceForge bug 1878659 - 'Does not build on 10.5 systems building a 10.4+ target'. Fixed the conditional of RK_REQUIRES_NIL_TERMINATION. The previous conditional incorrectly redefined it as NS_REQUIRES_NIL_TERMINATION when building on a 10.5 system, but strictly targeting 10.4. Added defined(NS_REQUIRES_NIL_TERMINATION) to further restrict the conditional. Also updated ENABLE_MACOSX_GARBAGE_COLLECTION to be further restricted with defined(__OBJC_GC__) and ENABLE_DTRACE_INSTRUMENTATION to be further restricted by defined(S_DTRACE_DOF), which is defined in mach-o/loader.h.
• Fixed a bug first reported by Doug Dickinson in the SourceForge RegexKit forum message 'match/replace with empty reference string?'. This turned out to be a bug in the NSString.m private function RKStringByMatchingAndExpanding. This function had an optimization where if no replacements took place, no changes were made to the original, and it could return the original string un-edited instead of creating a new one. Unfortunately the case where the regex matched the beginning of the string to be searched, but the replacement string was 'empty' (i.e., @""), this appeared as if no changes took place since no replacements were required. Fixed by also checking the final NSRange of the edited string against the original string to search. If a 'match at the start, but replace with nothing' happens now (i.e., as if the NSRange location had moved from 0 to a value > 0), these ranges will be different and will now correctly return a string with the start cut off. This may have also effected similar search and replaces that took place on the tail end of a string, but the fix will catch that condition as well.

Change Log for 0.5.0 Beta

DTrace Instrumentation

Created several macro wrappers for DTrace probe insertion in RKRegexPrivate.h. This allows a simple macro, RK_PROBE(), to be placed in the code where a probe point is desired. The macro will rewrite to nothing if ENABLE_DTRACE_INSTRUMENTATION is not defined. If it is defined, then it re-writes to an if() {} statement that first calls the 'Is Enabled' probe function, and only if the probe is enabled does it go through the work of assembling the arguments and calling the actual probe firing function.

A few modifications here and there to assist in passing information via the probe efficiently. For example, RKRegex has a new private function, regexUTF8String(), which will create a copy of the regex string in UTF8 form and stores the pointer in an ivar. This allows for lazy evaluation, it's only created if the function is called, and right now the function is only called when a probe requires it. Once created, the function obviously returns the pointer to the already created UTF8 string.

The majority of probes inserted at this time have to do with extracting timing information. RKRegex had the PCRE compile and match functions instrumented, and RKCache had the add, insert, lookup, and clear methods instrumented. A probe named 'PerformanceNote' was created and sprinkled in a few places. This is for passing text strings to a dtrace listener at certain spots in the code which have a negative performance impact. For example, if we're unable to get direct access to the backing store of a string, and are required to call [string UTF8String] as a result, a PerformanceNote is generated.

Instruments.app Instruments

To make use of the new dtrace probe points, several .instruments were created for Instruments.app. This tool shows a hell of a lot of potential. It's got some nasty bugs when creating your own .instruments, but it's a 1.0 product, on top of a whole new kernel facility (dtrace), so it's a bit understandable. While there's documentation for using the application proper, there's just a token mention of creating your own probes. From digging through the guts to create the RegexKit functionality, I'd say that a third-party plugin architecture is on the way. Not just beatified dtrace .d scripts, but the ability to write objective-c plugins with custom interfaces. The ObjectAlloc tool is a good example of this.

I hope Apple hurries up with the third party plug in support. While you can get a hell of a lot done with the few probes and .instrument scripts I put together, I can see creating some incredibly powerful performance tools with a custom interface. It would almost be like Shark, but for regular expressions. There's only so much you can do with dtrace scripts, but you can do so much more if you can post process the raw data and make changes on the fly. Say you're having performance problems with your regex, and you want to know why. And you have a lot of regex, and you're note sure which one is causing the problem. Enabling the matching probes would give you a stream of raw data, and you could post process it to aggregate timing information on a per regex basis. Regexes that take a long time to perform a match would bubble right to the top, and the great thing is you can even get snapshots of the buffers involved. And once you start to narrow things down, and you're only interested in a single regex out of all of them, because each regex has a unique hash, this can be easily placed in the dtrace predicate which means you're only collecting data on the ones you're interested in, the others will have just a minor performance impact since the dtrace kernel module has a constant, cacheable predicate that comes down to a simple integer compare.

These instruments, and the probe points, should be considered expiremental. The specifics are very likely to change as I gain more experience with both DTrace and Instruments.app. I've marked the DTrace probe points 'Unstable', which is an advisory to users of the probe points that things are likely to change from release to release. DTrace has a versioning scheme for the various parts of the DTrace interfaces to help DTrace users gauge how likely it is their scripts and what not are to break between patches, releases, etc.

NSArray Additions

Added indexSetOfObjectsMatchingRegex: and indexSetOfObjectsMatchingRegex:inRange: to the NSArray category additions.

NSData Class Category Additions

Added the following category method additions to the NSData class:

• - isMatchedByRegex:
• - isMatchedByRegex:inRange:
• - rangeOfRegex:
• - rangeOfRegex:inRange:capture:
• - rangesOfRegex:
• - rangesOfRegex:inRange:
• - subdataByMatching:
• - subdataByMatching:inRange:

See NSData RegexKit Additions Reference for additional information.

Other changes

• Altered RKAppendInstruction() and RKAppendCopyInstruction() to grow heap allocated instruction buffers by RK_DEFAULT_STACK_INSTRUCTIONS number of instructions instead of 16.
• Created some memory allocation macros to pick the appropriate allocation, free, and other potentially GC related / impacting functions dynamically. Keeps the code considerably cleaner.
• Using the above new allocation macros, moved several memory allocations over to the new macros. Examples include: RKEnumerator range buffers, NSString temporary string buffers, RKRegex and RKCache UTF8 strings of the objects description. Updated various dealloc / finalize methods to account for differences.
• For RKRegex and RKCache, created private functions to on demand create a long lasting UTF8 string of the objects description. This is demand created by various DTrace probes to provide access to a string that can be easily copied when required.
• Created various RKAtomic primitives of 'Integer' size. Previous atomic operations were only of 'int' size. This is in keeping with the NSInteger 32 / 64 bit dynamic sizing.
• Altered the order of some comparison operators in RKParseReference to be slightly more optimal. This can cause the conditional part of an if statement to be executed slightly faster.

Bug fixes

• Fixed a "search and replace" family of methods bug which would cause a crash if there were a lot (~1000) of replacements made on a string. This was the result of an incorrect sizeof() which resulted in copying too much data. The bug itself was harmless and didn't damage the stack, but if there wasn't enough 'buffer space' on the stack to absorb the over-run, then you'd walk past the start of the stack at 0xc0000000. This would be most likely to happen when a search and replace method was called from main().
• Switched to using NSINTEGER_DEFINED to determine if NSInteger and friends are typedef'd. Previously tried to infer whether or not the type existed from checking if MAC_OS_X_VERSION_10_5 existed.
• Fixed potential race condition in non-Garbage Collection mode where the RKRegex self maintained reference count was not atomically checked correctly to determine if the reference count became zero.
• Under garbage collection, addObjectToCache:withHash: would always return NO, even though the object was added to the cache.

Change Log for 0.4.0 Beta

Unicode changes

Previously, most convenience methods would use RKCompileDupNames as the compile time options. This was removed (you can now specify this via the regex string with (?J)), and replaced with (RKCompileUTF8 | RKCompileNoUTF8Check). This activates various Unicode behaviors that I expect most people would assume to be "on" automatically. It also changes some matching behavior, but both of these are probably only important if you're making use of Unicode functionality.

NSRange results changes

Previously, RegexKit always returned the ranges as reported by the pcre engine. This could cause problems for Unicode strings because PCRE uses byte oriented UTF8, where as Foundation defines a character to be the characters UTF16 representation. The majority of common characters map to a single UTF16 character, but anything beyond ASCII (<128) takes multiple bytes in UTF8.

The Foundation additions (NSString, etc) now accept and return results in UTF16 'coordinate space'. The RKRegex class continues to work in the native UTF8 'coordinate space'. Mapping between the two can be costly, and at this time no particular optimizations are made to improve performance.

The following two functions map ranges for a string from one coordinate space to the other:

RKConvertUTF8ToUTF16RangeForString(string, range)
RKConvertUTF16ToUTF8RangeForString(string, range)

The buffers internally are always in UTF8 format, so to map between the two these routines start at offset 0 and advance both a UTF8 offset (which is always the current byte offset), and the UTF16 offset (which is less than or equal to the UTF8 offset, depending on how many multi-byte UTF8 characters have been processed).

The Mac Roman and ASCII string encodings always have a 1:1 mapping of UTF8 offsets and UTF16 offsets, since there can be no multi-byte long characters in these encodings, so mapping in this case is very fast (returns the requested range unaltered).

Ranges as returned by the various Foundation extensions can now be used interchangeably with other Foundation methods that accept a NSRange as an argument.

Other changes

• Bumped the RegexKit version to 0.4.0 and updated the PROJECT_COMPATIBLE_VERSION to 0.4.0 as well due to the various NSRange API changes.
• Changed PCRE_URL_ROOT build setting to point to sourceforge.net and download via http://, which is probably more firewall traversing friendly than the older ftp:// from the primary site. The curl options -LR were added so that curl will follow the initial requests redirect to a mirror site and set the timestamp to the download servers timestamp of the file. Also split the PCRE_URL_ROOT into _SCHEME, _SITE, and _PATH components and added PCRE_FETCH_OPTIONS.
• Altered the way the RegexKit Framework and Unit Tests targets link to the pcre static library that we build. Previously it used the -L${PATH} -lpcre way of linking. This was changed to (verbatim) ${PCRE_TARGET_LIB_FILE}. No flags, just the full path to the .a static library archive. The reason for this is /usr/local/lib happens to be a directory that ld searches. If you have a version of PCRE installed in /usr/local/lib, the ld linker would prefer and link to the /usr/local/lib/libpcre.dylib and not the version that we built. The new method unambiguously removes this problem.
• Moved all information from the ReleaseNotes and ChangeLog plain text files in the project root directory to the ReleaseNotes.html and ChangeLog.html files, respectively, adding additional markup where required. The body of the plain text files now refers to their respective .html versions.
• Removed the Release Information from RegexKitProgrammingGuide.html and placed it in its own file, ReleaseInformation.html.
• Added Release Information to the table of contents, with subsections that point to ChangeLog.html, ReleaseNotes.html, and ReleaseInformation.html.
• Added unicode.m to Unit Tests to test that the various NSString additions are correctly matching strings with unicode characters in them, and that NSRange results are in UTF16 based character indexes.
• Added several stringByMatching:replace:withReferenceString: (and related methods) tests to collectionAdditions.m. Oddly, I forgot to convert these from some previous test code, an they were left in a comment.
• Added tests to exercise the new case conversion \u \l \U \L \E escape sequences and the \[0-9] capture reference syntax.
• Added a Release Notes link to ReleaseNotes.html in the zebra strip banners next to the RegexKit version information.
• Added RKConvertUTF8ToUTF16RangeForString and RKConvertUTF16ToUTF8RangeForString to projectExportedSymbols for external framework users.
• Added Unicode Character Index Conversions to the Functions table of content section for RKConvertUTF8ToUTF16RangeForString and RKConvertUTF16ToUTF8RangeForString and added HeaderDoc comments for both.
• Updated NSString.tmpl with documentation for the \u \l \U \L \E case conversion escape sequences, information on the NSRange updates, and \[0-9] capture subpattern reference syntax.
• Updated RKRegex.tmpl with information regarding the various Unicode and NSRange changes.
• Updated RKEnumerator.tmpl with information regarding the various Unicode and NSRange changes.
• Several unit tests required minor modifications due to the various Unicode and NSRange changes.
• Added some additional information to README.MacOSX about building RegexKit from the command line, such as targets available, and overriding options.

Bug fixes

• Added the missing target="doc" to generatePCREToC.pl for the individual sub-entries so they open in the main frame, not in the tocs frame.
• RKArrayFromMatchOption, RKArrayFromCompileOption, and RKArrayFromBuildConfig were incorrectly reporting newline bits as undecoded.
• Updated RKCaptureIndexForCaptureNameCharacters to check both the pcre_fullinfo PCRE_INFO_JCHANGED indicator AND RKCompileDupNames as opposed to the older check of just RKCompileDupNames. If (?J) was used in the regex and RKCompileDupNames was not set, then some name to index conversions would be faulty under the right circumstances.
• RKEnumerator and the NSString additions were incorrectly throwing a NSInvalidArgumentException for named captures that did not exist in a regular expression instead of the expected RKRegexCaptureReferenceException.
• With the Unicode improvements, there were a few places where various functions had a choice of encoding between two source strings. Occasionally, this choice would be the wrong one. In general, the conversion choice was hard set to UTF8 which should be correct for all source string encodings.
• Several build system issues dealing with Xcode 2.4.1 / Xcode 3.0 and Mac OS X 10.4 / 10.5, such as proper SDKROOT and ARCHS selection. Now detected and handled automatically.
• Changed the DocSet to display index.html, which contains the Table of Contents, instead of just content.html.

Change Log for 0.3.1 Beta

Xcode 3.0 DocSet Subscriptions

Added DOCUMENTATION_DOCSET_FEED_SCHEME and DOCUMENTATION_DOCSET_FEED_URL to the build settings which control which URL is placed in the assembled DocSet for subscriptions and auto-updates. Realized that it'd probably be helpful to have the versions in question be displayed, so updated a number of templates to place the PCRE and RegexKit versions in the zebra banners. SCHEME and URL are separate because http:// is interpreted as a comment start in .xcconfig files. :(

Change Log for 0.3.0 Beta

Xcode 3.0 DocSet Documentation

The DocSet documentation is essentially the HTML documentation with a few minor tweaks. It is created by the scripts:

Documentation/Scripts/buildDocSet.sh
Documentation/Scripts/createDocSet.pl

createDocSet.pl is pretty horrific, and was kludged together very quickly. It creates the required Nodes.xml and Tokens.xml by extracting the information from the documentation database and rewrites the anchors in the .html files to use the //apple_ref/ style format. This is not used in the normal HTML documentation because it isn't valid HTML (tidy throws a fit and there's hundreds of warnings generated.) However, the DocSet tool chain appears to be much happier with //apple_ref/ style anchors, so the legal html name anchors are rewritten.

Once all these pieces are in placed, and a Info.plist is created, the docsetutil command is run to build the DocSet index and that's pretty much it.

Garbage Collection

Mac OS X 10.5 Garbage Collection is controlled by the C Preprocessor flag ENABLE_MACOSX_GARBAGE_COLLECTION. A number of macros were created to ease various portability issues. Of special note is the framework global variable RKRegexGarbageCollect. If ENABLE_MACOSX_GARBAGE_COLLECTION is defined, then RKRegexGarbageCollect is a int32_t defined in RKRegex.m. If ENABLE_MACOSX_GARBAGE_COLLECTION is not defined, then RKRegexGarbageCollect is #defined in RegexKitPrivate.h as 0. Also in RegexKitPrivate.h are a number of macros for managing various retain / release / autorelease.

If ENABLE_MACOSX_GARBAGE_COLLECTION is defined, then RKRegex.m, RKCache.m, and NSString.m will all check and see if the class NSGarbageCollector exists, and if so, check to see if garbage collection is enabled in their +load methods.

If ENABLE_MACOSX_GARBAGE_COLLECTION is not defined, then most of the macros become a compile time static conditional which the compiler will optimize away, so compiling on pre-10.5 or GNUstep is branchless. It was the best I could come up with for maximum portability while still maintaining readability.

In RKCache.m, if ENABLE_MACOSX_GARBAGE_COLLECTION is defined, AND if garbage collection is enabled at load time, then the class switches to using the new NSMapTable class for managing the cache. It places cached RKRegex objects in a NSMapTable with a weak reference to the object. This should allow for auto trimming of the cache to the working set. Not sure how it will work out in real life.

64 bit support

A set of macros was adopted to ease multi-platform 64 bit issues. They are essentially parallel to the Leopard NSInteger / NSUInteger types, except that they are cpp macros. They have the names RKInteger and RKUInteger / etc. They are re-written to either NSInteger or int depending on the detected run time environment. This allows for type checking / compatibility to be maintained for each environment. GNUstep still has things such as hash defined as unsigned int, and for some of our objects we return the pointer to the instantiated object as the hash (there can be only one) and this causes truncation on GNUstep, but it should be OK.

GNUstep

Cleaned up various GNUstep pieces here and there while putting the 64 bit stuff in. Should build much cleaner now, with only a handful of warnings when compiling on a 64 bit platform for various hash functions, but this shouldn't be a problem in the real world.

RegexKit.framework Info.plist version bug fix

The Info.plist in RegexKit 0.2.0 had a bug in the Info.plist file. The settings for the keys CFBundleShortVersionString and CFBundleVersion where set to the variable ${REGEXKIT_CURRENT_VERSION} when they should have been set to ${PROJECT_CURRENT_VERSION}. Since the variable ${REGEXKIT_CURRENT_VERSION} was undefined, this ended up expanding to nothing, so the correct version number is missing. This should be a fairly trivial bug in the grand scheme of things.

Mac OS X Installer .dmg distribution

RegexKit 0.3.0 marks the introduction of a Mac OS X Installer based distribution. It bundles together the Mac OS X Binary executable RegexKit.framework, the Documentation, and the source code in to one convenient package. Each is grouped in to their own package, and by default RegexKit.framework and Documentation are selected to install while the source code package is not. There's some javascript code to disable a package if the version that is already installed is newer than the version that is attempting to install. Also, there's some logic so that the source code package will automatically default to selected if it detects that a previous install of the source code has been performed, thus auto upgrading without having to think about it.

The Installer has a volume check script that requires 10.4 or later in order to install on a volume, and an installation check that checks to see if the file /Developer/Applications/Xcode.app exists. If it does not, it presents a warning that the Xcode Development Tools should be installed, but does not stop the installation.

There's three main tuneables in the build configs:

Since the overall distribution package gets placed on to a .dmg disk image, which is then BZip2 compressed (-format UDBZ), compressing the packages with gzip actually causes the final .dmg size to grow by about ~60-70KB, or roughly 10% in size. So, by default, we leave the packages uncompressed and let the disk image BZip2 deal with all the compression.

The other option, DISTRIBUTION_INCLUDE_PCRE_PACKAGE, toggles whether or not the PCRE tarball used to build the framework is included in the Installer distribution. By default the choice is NO since this causes the final .dmg disk image to nearly double in size. There's some extra javascript logic so that the PCRE distribution is if and only if the source code package is selected to be installed. If the user selects the source code package, the PCRE package then auto selects itself to be installed as well. The PCRE package version is the PCRE Major.Minor, not the RegexKit version. Unlike the others it will allow an earlier version to be installed 'on top off' a newer version since this is largely harmless- Each PCRE tarball distribution is separate from the others.

The file that creates the top level distribution package distribution.dist file is in Source/Build/Scripts/packageDist.pl. It handles the logic of selectively inserting the PCRE choice depending on whether or not DISTRIBUTION_INCLUDE_PCRE_PACKAGE is set to YES. It also gathers some necessary information, such as a packages installed size, by using perls Objective-C / Foundation bridge to read the individual packages Info.plist files and extracting the appropriate keys.

For each package (of which there are four, RegexKit.framework binary package, Documentation package, Source code package, and optionally the PCRE distribution package) there exists two .plist files, one info.plist and one desc.plist. The important one is the info.plist, and it contains the bundle identification and version information. The version information is replaced dynamically at build time by the Source/Build/Scripts/buildDistribution.sh script. That script makes a copy of the .plists to a private build staging area. Then, it calls the script Source/Build/Scripts/plistUtil.pl which is a very simple perl script that again uses the Objective-C / Foundation bridge to take three arguments: the .plist file to operate on, the key to update, and the value to set the key to. buildDistribution.sh uses this to update all the .plists to the current RegexKit version as required by the package .plist format. An additional .plist file, RegexKit_mpkg.plist, is largely for cosmetic purposes. It gets the version updated as well, but its sole purpose is to provide a version string to the finder when a Get Info is performed and that's it.

The text that's displayed during the installation process comes from four files, though only two are currently used. These are located in Source/Build/Packagemaker/Resources/..., which is a localized .lproj style directory for localization purposes. Currently only English.lproj is present. The files are currently simple .rtf files, but .rtfd and .html are supported as well. Any changes to the file types would have to be updated in the packageDist.pl file (it should be obvious where). Only ReadMe.rtf and License.rtf are currently used, but both Welcome.rtf and Conclusion.rtf can be activated trivially by editing packageDist.pl. They get copied in to the distribution package right now, but the necessary lines in distribution.dist that is created by packageDist.pl are missing.

The tuneable DISTRIBUTION_DEFAULT_INSTALL_DIR is self-explanatory.

Minor bug fixes

• There were a handful of very minor, single line changes to fix some trivial bugs. These mostly had to do with inter-version differences between tools (ie, sqlite 3.1.3 -> later versions tweaks, minor accommodations to make building from the installer distribution run clean)

Build system tweaks

Replaced the debug: portion of various script status lines with note: instead. This is mostly cosmetic. In the Makefile.pcre, added the makefile file name and line number of the message so clicking on it will take you right to that line. Updated most status messages to always output the filename and line number with the status indicator.

The PCRE build target is largely driven by Makefiles. The scripts that invoke `make` were modified to set the -j argument, or the number of jobs to run in parallel, and the -l argument, or the load average where no new jobs can be started. The values for these parameters are derived from the sysctl variable hw.activecpu. The -j parameter is set to the number of active CPU's, and the max load is set to the number of active CPU's + 2. This should allow those of you with multiple CPU's to build the PCRE target faster. However, sometimes building in parallel causes certain make dependency issues to crop up that don't happen when running sequentially.

This can be disabled with the tuneable PCRE_PARALLEL_BUILD = NO.

The Distribution target was also partially parallel build enabled as well.

Change Log for 0.2.0 Beta

End User Framework Visible Changes

The nature of changes for this release are almost all documentation or build system related. Only a handful of changes resulted in any API level changes, the largest probably being the PCRE 7.3 to 7.4 change.

Only one change stands out that may impact existing users of the framework. The Alpha version did not contain an explicit version of the framework, and used the default of 1 for the version that was used in the Mach-O library version information.

This release begins the use of a framework versioning system. It is a standard "three point" version type system comprised of a Major, Minor, and Point version. These versions are reflected in the Mach-O library version.

Refer to the RegexKit Programming Guide section on Release Information or more information.

Distribution Bundles

The file bundles for a distribution release has changed. There is now a Mac OS X oriented binary + documentation release and a no binaries, source only + documentation bundle.

PCRE Version Changed

The PCRE library that the framework builds against was updated to 7.4.

const Use Expanded

Where appropriate, the const keyword was added. Most of this was marking both the pointer and data pointed by the pointer as const.

For example:

const NSRange *matchRanges

changed to

const NSRange * const matchRanges

Framework Initialization

Previously, initialization that was required at load time was performed with functions marked with the attribute constructor. This was replaced with the more portable, but functionally identical, Objective-C +load method.

C99 restrict Added

A new pre-processor macro, RK_C99(), was added that causes the macros argument to either be replaced as-is, or removed depending on whether or not __STDC_VERSION__ >= 199901L (ie, C99). This is used to selectively allow the C99 restrict qualifier through for various function / method prototypes. This allows projects who are currently not using C99 to transparently use the framework without warnings/etc.

Framework Version

The build setting ${PROJECT_CURRENT_VERSION} now contains the frameworks version. It is composed as the dot separated form of ${PROJECT_VERSION_(MAJOR|MINOR|POINT)}, ie 0.2.0.

The version of the framework is now incorporated in to the Mach-O shared library version information. The project build setting ${PROJECT_CURRENT_VERSION} sets the dynamic libraries version, and ${PROJECT_COMPATIBLE_VERSION} sets the compatibility version.

Project Build Settings

The various top-level project settings were moved out from the Xcode project.pbxproj file and in to a separate .xcconfig file. The Xcode project build settings were then set to be 'based on' this file. The file is at Source/Build/Xcode/RegexKit Build Settings.xcconfig. This makes it much easier to add and modify the build settings. It also allows comments to be placed in the file, documenting the settings at the same time.

These settings can then be easily over-ridden at either the project or per-target level within Xcode by selecting the appropriate Project > Edit .... menu item and then simply entering the desired over-ride.

A notable setting is GCC_OPTIMIZATION_LEVEL. Xcode normally only allows for 0|1|2|3|s (ie, -O1, etc). However, the Apple branch of GCC also supports a 'z' level optimization which is loosely defined as 'Like -Os (-O2, preferring smaller code size to larger/faster code), but aggressively preferring smaller code size to any speed advantage.'. There was approximately 24KB smaller executable size difference for each step in 2 to s to z settings and no timing difference reported by the timing unit test, so z was selected for a total executable size reduction of ~49KB.

Total Optimization plus Dead Code Stripping savings: 50420 / 49KB ([-O2] 495776 - [-Oz dcs] 445356)

PCRE Build Target

Substantial updates to the PCRE build target. There is very little user visible changes, however.

The PCRE target has a 'Run Script' build phase which executes the Source/Build/Scripts/buildPCRE.sh script. Since this is a target that is outside the control of Xcode, it must perform several dependency checks to ensure that what has been built is valid given the current build settings and target dependencies. This script performs those checks manually by comparing the output of `env`, Source/Build/Scripts/buildPCRE.sh, and Source/Build/Makefiles/Makefile.pcre to a copy of those settings / files that were used when the target was built. Any change effectively causes the target to be rebuilt from scratch (ie, rm -rf / untar) to ensure any dependencies are accurately reflected in the built library.

Altered the build system to place all temporary files used during the build process, including the untaring of the distribution, in to ${PCRE_TEMP_ROOT}, which translates in to build/RegexKit.build/CONFIGURATION/PCRE.build/pcre-VERSION/. CONFIGURATION being either Debug or Release.

Build System Changes

Numerous build system changes, too many to list and nearly all are not important to end users. Examples are the more robust PCRE build subsystem, clean up of the documentation build system, clean up of the aspell spell checking of documentation, clean up of tidy checking of the HTML output, etc.

SVN at SourceForge.net

This release represents the first import to the SVN repository at SourceForge.net.

File Encodings

Updated/converted all appropriate files to Unicode UTF8 encoding and updated the Xcode reference to reflect the new encoding.