The past few days I've been learning how to write bindings for Perl using XS so that I can use the many great libraries out there that I normally use in C or C++. Native bindings are very magical things because they glue together different languages that often don't have a direct mapping of semantics with respect to each other. XS is a bit quirky in that, while most language binding APIs require writing calls directly in C or C++, it is actually it's own DSL for making bindings. There is a preprocessor called xsubpp that generates the actual API calls to glue the Perl interpreter with the native code.
I actually wanted to start learning XS a few months back. In the past,
I would put together rudimentary bindings using SWIG,
but the results weren't very pleasant to use. It ends up creating bindings that
look very much like calling C code and force you to deal with pointers and context
directly. That pretty much defeats the purpose of creating a binding! So now that
I have a bit more tuits, I started looking
around for documentation on using XS. Coincidentally, I found a
project that gathered many of the same
notes I was using. Seems that I timed my learning process just
right and I've
been learning a great deal about Perl internals from the newly relaunched
channel on irc.perl.org.
As I usually do when I'm learning something new, I jump right into making something as I'm picking things up. I chose to work on something that was both simple, but non-trivial. Years ago on Freshmeat, I came across a project called libuninum that converts different number system strings into integers. Once you have these integers, you can use them in operations for arithmetic and sorting. Pretty useful if you have to deal with data in different languages.
Before I actually hack on the bindings, I need to think about how I'm going to
distribute this code. Most people's systems aren't going to have access to the libuninum
source code to build these bindings, so I'll need to somehow get the source
code and build it on those systems. That's where Alien::Base
comes in. It's a neat module that will download a tarball, extract it, build
it, and place the dynamic library and headers in a place that can be accessed
by other modules. I made a subclass of Alien::Base called
Alien::Uninum that will do just
that for libuninum. I even got a small patch in to
Alien::Base to fix some issues I had. All I needed now to start hacking on the XS code is a
way to tell the compiler where all the libuninum files are. With Alien::Base,
I just send those to the package build process using the
libs methods which is pretty much like using
I got to hacking and started on the simplest task: getting the list of all the
number systems. I first approached this by just making a list of hashes that
contained the name and ID of each number system
Not too bad. I then added caching of that list by storing that as a private
attribute of my
Then I built on that and created a
Unicode::Number::System class to
store the number system name and ID so that I could return instances of that
I then moved on to to the actual main function of the library: converting a Unicode number to an integer. This was a bit tricky because Unicode comes in many different encodings (e.g. UTF-8, UTF-16, UTF-32) and these encodings can also have different endianness. Since the libuninum library expects all strings to be in UTF-32, I converted Perl strings from UTF-8 to UTF-32 and sent them to the XS code, but the library was giving me an "illegal character" error. To debug this, I grabbed some of the data from an example file that came with libuninum and put it in my XS. Still not working. This didn't make sense because I could get it working in plain C, but not in the XS. So I put together a small script using Inline::C that let me call the libuninum function directly.
It still wasn't working. So, as you can see above, I grabbed a function from
uninum.c and renamed it to
MyLaoToInt and called it directly. Still wasn't
working. Only when I started to print out the contents of each character did I
realise what was happening. In libuninum's
UTF32 typedef is
defined as an
unsigned long, however
sizeof(unsigned long) is 8 (64-bits)
on my system, not 4 (32-bits).
That means that as the library iterates over each character, it is actually
looking at two characters instead of one and of course, none of the comparisons
were working. What it actually needed to use was a
However, even though this typedef is in the C99 standard, there are some portability
issues with using it. Instead, I used the integer type that Perl detected to be
32-bits wide and patched the code when I built it using Alien::Uninum
(code). Now the file looked like this:
Yay! Now the XS code was working on the test data. All I had to do now was get my string to libuninum and pass the result back. I tried that and libuninum was giving me errors again. Now what?! I decided I need to look at what the C was accessing, so I grabbed a hex dump routine from here and looked at it:
00 00 fe ff ...
As soon as I saw the first character, I knew what was going on. What I was looking at was the byte-order mark or BOM. Remember, I had converted the UTF-8 string to UTF-32 in Perl before sending it to C, but I never specified the endianness, so Perl used big-endian as the default endianness. Well, since the C code was using the native endianness of the machine, I needed to find the machine's endianness and encode either a little-endian or big-endian version of UTF-32. I just had to ask Perl the byte order it detected at compile time and use that (code).
Once I did that, my code was working and all my tests passed! There are still a couple of things I need to do in order to clean it up, but it's mostly done for now.