Nick Bastin

My frustration with the performance of SMB is really starting to push me over the edge (oh, so that’s what’s doing it, you say?). SMB, sadly, appears to be the only widely available and supported network filesystem left. You might argue that NFS is still relevant, but it really isn’t – even on *nix it’s hell to use if you aren’t using NIS, and windows client support is downright disgusting (both expensive and lousy – the double whammy of quality software). So sadly we’re left with SMB – the only network filesystem well supported on MacOS X, Windows, and *nix.

Unfortunately, as luck with have it, the performance of SMB on high performance networks is downright awful when it comes to bulk transfer. I have plenty of occasion to move around multi-gigabyte files on a rather regular basis – to the tune of around 100 GB a day on a regular work day, on average. My first attempts to offload large files to a separate server were lackluster at best – 20MB/sec over a dedicated gigabit link to my server. A little analysis offered that ethernet frame size was actually a significant problem, and I switched to using 9000 byte jumbo frames. This increased my SMB copy performance to 30MB/sec, which was “sort of acceptable”, but hardly good on a full duplex gigabit link.

So where was this bottleneck? It seemed sensible at this point to test other transfer options, as well as the speed of the local disks to make sure I wasn’t barking up the wrong tree. (A small note – I’ve since had to change the MTU on the gigabit LAN to 7000 bytes to accommodate some new hosts with less-than-fabulous ethernet chipsets – RealTek 8111C to be specific – which top out at MTUs around 7200 bytes).

The file server in question is an Athlon 64 X2 5600+ (dual core 2.9Ghz), 3Ware 9650SE-12ML with dual 3TB Raid 5 arrays, 6GB ram, and some sort of nForce ethernet chipset that turns out not to be too bad. The clients are all AMD Phenom II X6 1055T (6-core 2.8Ghz), 8GB ram, 2×1TB 5900 rpm SATA disks, and the aforementioned horrible RealTek 8111C ethernet chipset. Each test was performed with a 40gb file.

File server read (local): 189.3 MB/sec
File server write (local): 97.9 MB/sec
Client read (local): 96.8 MB/sec
Client write (local): 75.3 MB/sec
SMB copy (server->client): 29.2 MB/sec
SCP copy (server->client): 58.8 MB/sec

For the SCP copy, the (underpowered) server CPU was absolutely pegged at 98%, so the encryption overhead was likely a significant limiting factor. As you might expect the client disk performance can’t compete with the 3ware array, but both are significantly faster than what we’re getting out of the network. I could probably draft up some more experiments to squeak the last bits of performance out of the network, but this doesn’t seem important – SMB is significantly behind even SCP, which has tons of built-in overhead already due to the encryption. Thus leading me to the conclusion that, at least for my use case, SMB sucks

It turns out that SMB sucks due to the way that it ACKs data messages, and how large those block sizes are allowed to be (64KB), but that’s a long discussion for an entirely different blog post, methinks. That being said, this story has, I believe, two morals – a) how do people deal with SMB and why don’t we have a better network filesystem, and b) maybe I really do have to set up NFS. What happened to things that just work?

I have submitted a proposal for a talk on micro-optimization for PyOhio, and you should check out the CFP yourself (ok, probably not you, mom…). I had to write a little bio and I put a link to this blog in it, so I figured I’d make sure I still had the login information to allow me to post.

Seriously though, I know myself, so it’s very likely that I will continue my practice of not actually blogging here. However, should my disassembler enhancements generate sufficient interest that I keep hacking on them (and ultimately turn them into a really nice app?), I’ll probably write about it a bit here.

If you try to build the lastest pycurl (7.19.0) on MacOS 10.6 you may get the following errors:

i686-apple-darwin10-gcc-4.2.1: /usr/lib/libcurl.a: No such file or directory
powerpc-apple-darwin10-gcc-4.2.1: /usr/lib/libcurl.a: No such file or directory
i686-apple-darwin10-gcc-4.2.1: /usr/lib/libcurl.a: No such file or directory

The problem is that setup.py is calling curl-config (as it should) and requesting the static libs as well as the shared ones. MacOS X doesn’t *have* the static libs by default (and you don’t need them), so you can comment out that line from setup.py and everything should work (in add_libdirs()).

It turns out this issue had already been reported, at:

http://bugs.python.org/issue4366

The initial (unaccepted) patch in there did the same thing I did in my source tree to make it work, but it was clearly the wrong thing (it had the “wrong thing” smell from the start, really). After a bunch of digging, the reality is that there are a bunch of posix-like platforms for which if you have Py_ENABLE_SHARED set in distutils, put -L. into the LDFLAGS, just FreeBSD isn’t one of them.

So anyhow, as the thread on the issue now states, the seemingly proper fix is to extend the list of platforms which add the current working directory to the list of directories searched by the linker during a build. Barring any last minute complaints, I’ll probably commit this to 2.7 trunk, 3.2 trunk, 2.6 maint and 3.1 maint early next week.

Ok, really that should be 2.6.4 and 2.7.1a1, but I’d imagine this problem might exist for more than just those versions. I haven’t tried to build 3.x.

I don’t quite know the exact steps that produce this problem (maybe any/all configure option?), but if you do:

./configure --enable-shared

as is my wont to do, Python happily builds and then barfs all over building modules, something like:


building '_struct' extension
gcc -shared build/temp.freebsd-5.3-RELEASE-i386-2.6/u1/Python/Python-2.6.4/Modules/_struct.o -L/usr/local/lib -lpython2.6 -o build/lib.freebsd-5.3-RELEASE-i386-2.6/_struct.so
/usr/bin/ld: cannot find -lpython2.6
building '_ctypes_test' extension
gcc -shared build/temp.freebsd-5.3-RELEASE-i386-2.6/u1/Python/Python-2.6.4/Modules/_ctypes/_ctypes_test.o -L/usr/local/lib -lpython2.6 -o build/lib.freebsd-5.3-RELEASE-i386-2.6/_ctypes_test.so
/usr/bin/ld: cannot find -lpython2.6

….ad nauseum….

This is because the compiler isn’t looking for where your python2.6 library currently is (even worse if you have a 2.6 in /usr/local/lib and it has different options than the one you’re currently building) – which is in your current working directory (‘.’). The hackish fix to the problem is to edit setup.py in the build root, and add in detect_modules:

add_dir_to_list(self.compiler.library_dirs, '.')


Do this *after* the line to add /usr/local/lib, so that (perhaps non-intuitively) -L. comes before -L/usr/local/lib on the compile command line.

Now if you rebuild everyone will be happy. There may be some sufficient CFLAGS or LDFLAGS jank to set in your environment that has the same effect, but you shouldn’t have to do this – Python ought to know where to find the library it just built if it needs it for further build machinery.

Slashdot simply reposted this blog entry at:

http://developers.slashdot.org/story/09/12/20/1433257/The-Environmental-Impact-of-PHP-Compared-To-C-On-Facebook

Passing on an opportunity to bash slashdot for lacking any sort of edited treatment of these stories (if they advertised themselves as a curated RSS aggregator, I wouldn’t even care about this, but they don’t), the source entry itself is not much short of a marketing pitch. It’s written by the people who write (and sell) a C++ web toolkit known as wt (memories of reviews written by game publishers, anyone?).

They obviously have an agenda, and they get a bit overzealous in their pitch. The fastest C++ is almost certainly faster than the fastest PHP – you probably won’t find anyone who would argue with that. However, they make a couple of rather outlandish claims:

• “assuming a conservative ratio of 10 for the efficiency of C++ versus PHP code” – using a general computing benchmark which contains no code that spends any time doing things you’d find on the web.
• “As they only say that “the bulk” is running PHP, let’s assume this to be 25 000 of the 30 000.” – Even if 25000 servers are in fact running PHP, it’s ridiculous to assume that 100% of their CPUs are consumed running PHP scripts. One would hope, certainly, that while PHP is used to generate dynamic data, that a lot of that data can be forward-cached and served again.

Certainly without more detailed data from Facebook, all of this is speculation at best. We have no idea what the mix of objects served by Facebook is between static/dynamic/cached. I have no doubt that if you started a reasonable cap and trade system for emissions, data centers might start caring a lot more about how efficient their code is, and this is a discussion that needs to be had. However, it needs to be had responsibly, bounded by the data we have, and not filled with conjecture created merely to sell a competing product.

I generally abhor wasted space on web pages, but some situations are less of a problem than others. As with all things, you should consider your audience. For example, if you’re the Mozilla wiki, you probably ought to consider that your users are likely reading your pages while trying to do something else on their computer, and you should be as considerate as possible when consuming their screen real estate.

It’s not that I mind a wiki-wide table of contents (although when compounded with an article-wide table of contents, it does seem a bit ridiculously placed):

It’s that you’ve now created a huge margin of wasted space below that table of contents:

And the wiki is designed such that the margin is fixed regardless of the width of the page:

Resulting in a workflow where I can only make the content smaller and small on the page in an effort to also see my terminal window, instead of eliding the (now-giant) margin.

Let this serve as a lesson to future authors of websites tailored towards software users and developers – your users are probably trying to use their computer while reading your content.

Children, when writing Python code, remember, bare except clauses are bad. BAD.

I know I haven’t written in a while. I might start again. I might not. I’m really unreliable like that.

However, I would like to take this moment to state that I really, really, hate Trac. Possibly I hate how people deploy it, but the common deployment for the wiki is quite annoying. The search goes to issues and changesets (and the wiki) by default, so then you have to remind yourself to uncheck those boxes and run your search again. Also, it seems not to rank results by anything meaningful – at least, if you search for the exact name of a page title, that page is NOT guaranteed to be the top result. Why!!!

I’ve been poking around using dtrace on OS X (Snow Leopard) recently and have some interest in using it with Python. Apple provides probes in Python (although they have yet to make it upstream, for reasons that I can’t fathom, given that they’ve been available for years) but I’m having some issues with them.

There is the excellent DTraceToolkit from OpenSolaris which has some nice Python scripts (in $install-dir/Python/), but it appears that Apple’s dtrace is limited in some unfortunate ways (no #define support, for one). Also, when using ‘python$target:::’. dtrace sees probes, but fails to actually report on them – perhaps I don’t understand how to invoke dtrace properly?

I have:

python$target:::function-entry
{
@lines[pid, uid, copyinstr(arg0)] = count();
}

dtrace:::END
{
printf("\n %6s %6s %6s %s\n", "PID", "UID", "FUNCS", "FILE");
printa(" %6d %6d %@6d %s\n", @lines);
}


But all I get is:

sudo dtrace -s test.d -c "/usr/bin/python test.py"

dtrace: script 'test.d' matched 1 probe
dtrace: pid 20908 has exited
CPU ID FUNCTION:NAME
7 2 :END
PID UID FUNCS FILE


Which is troubling…. Even if I use python*:::function-entry, I still can’t seem to use -c:

sudo dtrace -s test.d -c "/usr/bin/python test.py"

dtrace: script 'test.d' matched 3 probes
dtrace: pid 20900 has exited
CPU ID FUNCTION:NAME
2 2 :END
PID UID FUNCS FILE


Scripts which use python*::: and don’t attempt to attach to a given process (via -c) seem to work fine, but that’s less than optimal.