Adèle Blanc-Sec

We're back. Don't expect any long posts about the trip just yet; I'm off on another conference in a couple of days and it's going to be a while until I've scanned or processed the pictures I've taken. And of course we timed the trip to one of the most interesting events in recent Japanese political history, with Ozawa finally making a direct run for the prime minister post, possibly breaking up his own party in the process. To me it feels like a combination of an old warhorse doing a final charge, and a cornered politician trying the Italian gambit of getting immunity from his crimes by assuming high office.

The flight to Paris is about 13 hours, and the flight back is a bit less than 12. An eternity when strapped into an airplane seat. I feel like I've been stuck in a sack and beaten with sticks for a whole day. The Air France flight was a bit of a letdown; you'd think that if any airline would get the food right it would be this one, but frankly, none of it was very good. Both Finnair and Lufthansa are much better in this regard.

On the other hand, they have a good selection of French movies, including Adèle Blanc-Sec. I'm quite ready to forgive them the food just because of this. Adèle is a French adventure comic set in the early 20th century - steampunk before the term was invented, with a cynical and dark tone. A few of them have been translated to Swedish, and I read them over and over in our city library as a child. It was turned into a movie by Luc Besson recently. Normally I'd be quite wary of such adaptations, but hey, it's Luc Besson - how could it go wrong¹? And he pulls it off.

It's a film adaptation, so of course there's less material there than in a comic. But the lead character is spot on, and the story and cinematography has much the same blend of the mundane and the fantastic. There's less of the grotesque darkness than in the comic, but there's enough of a hint to make it work without (I guess) falling afoul of film censorship laws and making it an adults-only movie. Fun movie, if a bit uneven. Worth seeing.

#1 OK, so Le grand Bleu could have been cut down to about ten minutes without missing anything of the actual story, but he's improved since.

Terabyte

We're still in Paris for another couple of days. But to tide you over:

I had one of those "whoa" moments a while ago. Ars Technica (good site; I read it often) had a write-up about the latest generation of hard drives being too big for current computers. They've reached 3Tb (terabytes), and current-generation computers can't handle more than 2.1Tb per drive.

I hear that and I think that "oh, that's interesting I guess, but nothing to do with me". After all, we're talking Terabytes here; who would ever need a terabyte or more on their own machine, right? It's a near-mythical size. This is a problem for big datacenters, not for ordinary people. I mean I only have a 250Gb drive, which is.. a quarter of a terabyte.. in my laptop.....

A quarter of a terabyte, in a laptop the size of an A4 paper.

My current laptop. I know I've used the picture before, but I like it.

And 1/4 Tb isn't even very big, come to think of it. My desktop has a half-Terabyte drive, and so does our networked backup drive. So "terabyte" has silently moved from being near mythical to become humdrum and I haven't even noticed. Between my laptop, desktop and network drive I have a terabyte and a half of personal storage. 1.5 Terabytes. It's a magical number to me - and now, pedestrian. Ordinary. What's next - Exabyte? I didn't know, honestly; I had to look it up.

It's not all empty either. I use about 39% of this laptop drive at the moment. When I got this computer, Ubuntu (try it; it's good) and all my data from my previous machine took about 35%, so I've used up another 4% (about 6 gigabytes) since May.

Where does it all go? I have a lot of work-related data, research papers and so on, but a lot of my data is pictures. I store more pictures over time of course, but the size of each image increases too. My first digital camera gave me jpeg images less than a megabyte in size. The K10 gives me raw images of about 10 megabytes. And a good-quality 24-bit color scan from my Pentax 67 takes 400 megabytes. Needless to say, I don't keep the scanned files (I can always rescan the negative after all), but even the processed and compressed Jpeg files takes about 10Mb each. And while I process such an image I may have a couple of gigabytes worth of intermediate files laying about.

But even with this growth of data, disk space seems to grow even faster. I used to be really careful about saving space on my computers. I'd clear out old data, uninstall applications I don't use, and make backups to free some extra space. And still, when I got a new computer the disk on the old one would be almost completely full.

This is no longer the case. When I got my current computer, the disk on the ond one was only 2/3 full, and I had never once made a real effort to clean out old data. Abundance has crept up on us. When was the last time any of us worried even for a moment about the size of a software package, and how to clear out disk space for it? Storage has become unlimited for most of us today, and nobody's even told us.

Dusk

Everyone loves summer sunsets, with their riots of reds and yellows flowing across the sky. But when the sun has set comes dusk, that dwindling twilight as evening turns into night. It's never attention-grabbing the way the sunset is, but the cool blue colors give the mundane streets an otherworldly, almost magical quality.

Sakaisuji and Honmachi, right where I take the subway every day.

Office workers are rushing home by bike as the sun disappears.

A group of friends are having a simple dinner at an improvised table outside a back-street izakaya.

An office worker on the phone while waiting for his ride near Honmachi.

Our very convenient convenience store.

A robata restaurant sign, caught just as the intense blue dusk starts fading to black.

Election - one month ahead

We're packing our things and leaving for Paris tomorrow. Meanwhile:

The Swedish election is one month away. The two major coalitions - the somewhere-around-center-right and the vaguely left - stand completely even at around 46-47% in recent polls. Neo-nazi outgrowth Sweden Democrats now hover below the 4% limit. The Pirate Party doesn't register enough support to be listed separately. Ho hum.

But 22% of voters are undecided. Three coalition parties - the Communists on the left, and the Center party and the Christian Democrats on the right - are below 6%, and have all hovered right at the 4% limit at one time or another. Failure by any of them to get into parliament would completely alter the election and probably give the opposing coalition a comfortable majority.

The past few months have seen political triangulation at its finest - to the point where, if the election rhethoric could be believed, it wouldn't matter much which coalition ends up winning anymore. The opinion polls reflect this: for all the polling hype, the two coalitions have been in a dead heat for more than a year, with nary a blip to relieve the tedium.

There's been a bit of fireworks this summer. A minister in the center-right coalition resigned when accused of going to a prostitute by Aftonbladet, a left-leaning gutter press that later turned out to have made it up. This week, two Social Democrat candidates where caught by police in an LO¹ apartment with a couple of prostitutes. Entertainment for the public, and wrecked lives for everyone involved and their families, but largely irrelevant for the election.

I voted this week, ahead of the trip. Postal voting makes it easy to vote wherever you are so there's no real reason not to. What did I vote on? None of the coalition parties. I feel no close affinity to any of them, and I have an enduring dislike for the Communists and the Christian Democrats and will not vote for any coalition if either is a member. And I would slit my wrists rather than vote for the racist bigots of the Sweden Democrats.

Instead, I vote for the Pirate Party. They focus on open government, freedom of information and communication, intellectual property reform and privacy issues. It is a one-issue party of course, but as issues go this is important, and becoming more so every year. How we resolve these issues now will help determine what kind of society we live in a generation from now. And it's of course more important than tenths of percentage points in the building tax rate or the number of ATMs in Stockholm subway, which is the kind of thing the two coalitions are bickering about. Do I agree with everything in their program? No. But I do feel strongly for many of the important issues, which is more than I can say for either of the two coalitions.

Will they get into parliament? Doubtful. But that doesn't mean my vote is wasted - voting for a coalition I don't actually agree with, on the other hand, would. And it's not like the chance of making it is zero; they managed to get two EU parliament seats against all odds after all.

If you're Swedish, vote. And if you don't much like either of the two main alternatives, you can do much worse than the Pirate Party.

---

#1 LO is the largest Swedish worker's union, theoretically separate from the Social Democrats but in practice they're two wings of the same organization.

Bertie Wooster as Batman

I'm really, really busy trying to prepare for the SAB conference next week. Then, right after returning to Japan I'm off again for NEURO 2010 in Kobe where we're showing a poster; a poster I have to finish before leaving for Paris since it's too late by the time I return. At least Kobe is close enough that I can go home every night instead of having to stay at a hotel.

Anyway, while I'm tearing my hair out¹ over this, you could do worse than visit MightyGodKing.com and especially what it'd be like if Bertie Wooster was Batman. Enjoy.

#1 What little is left of it.

To Paris

We're going to Paris next week. I'll be attending SAB 2010, a biannual conference on the simulation of adaptive behavior. It was the first real conference I went to as a newly-baked doctoral student, and it's still one of my favourite conferences. It's the only place I know where you can hear about ant foraging communication, simulation of lizard spine movement, cockroach brain implant-controlled robots, visual segmentation and human exoskeleton development, all in one conference.

A conference, by the way, is not just - or mainly - about the presentations. True, they're the meat of the event, but if they were the only reason you could might as well do a teleconference, or just email the research papers to everyone. No, the chance to meet people is at least as important.

You gather tens or hundreds of researchers from around the world to network and to socialize. You meet old friends and co-workers, and you meet new people in your field. This is where collaborations begin, where people find their next job, and where new research projects get their inspiration. Conference dinners and get-togethers aren't frivolous extras, but as important as the presentations themselves. A conference really is a dating service for research projects. You show your project through your presentation or poster, and with any luck you'll find a few kindred spirits interested in the same thing. It may lead to a brief exchange of ideas, or to a long-term collaboration lasting for years.

Anyway, while I work, Ritsuko will be doing Paris. See famous sights, browse street markets and department stores, sipping coffee at a sidewalk table watching the world pass by... It sounds fun, and I hope I'll get the chance to do the same someday. This time I'll just add to my experience of conference centers and meeting rooms around the world.

Now, what camera to bring? It's a work trip of course, but still, Paris is Paris. The conference itself is at the museum of natural history, and should be pretty photogenic. Also, I'll have a chance to do some street photography in the mornings and evenings.

The Pentax 67 is right out, unfortunately - it's just too big and heavy to lug around at the conference. My digital SLR actually takes as much space and weight as the medium-format Pentax once you add the batteries and all the other stuff you need. Besides, this is Paris - it practically screams for black and white film.

Ritsuko will probably bring her digital pocket camera and her film SLR. I could bring a 35mm camera of course, but if we already have one then a second one feels a bit redundant. Besides, I really, really like medium format film.

So I'll probably bring the Yashica. It's light and compact so I can just stuff it in my bag at the conference. It's no Rolleiflex; the lens is pretty crummy wide open and the mechanics aren't all that reliable. But the camera is neither rare nor expensive so if something would happen to it it's not a disaster.

I can't bring a tripod and I want to avoid using that lens wide open so a decently fast film is a good idea: HP5 shot at iso 800 should do me nicely. I've used it before with good results. The Yashica, three or four rolls of HP5¹ and a roll of PN400 in case I feel like shooting color. Should be fun.

---

#1 A whopping total of 36 pictures! But you shoot a lot less with film, and you take a lot more care to think through each picture before wasting a precious frame. I can take a hundred digital shots or ten film shots over a day, and in either case I end up with about 4-5 really worthwhile pictures at the end. And I'll spend a lot less time both taking and processing the film shots afterwards.

So Quiet

It's Obon, when most Japanese take time off and hit the road to go back to their hometowns, see old relatives and visit the family grave. Which means that out there is a circus of backed-up traffic, overflowing trains and frayed nerves right in the hottest part of summer, with the first typhoon of the season making landfall across half the country. But in here it's cool, it's still, and it's so very quiet.

My mornings normally look something like the shot at the top. The picture at the bottom is from yesterday. Practically empty.

The reason being, I'm the only one at the office today. There's a couple of heavy deadlines breathing down my neck, so I'm not taking any time off this year. A lone doctoral student was working over in the common room yesterday, but today I'm all by myself.

This suits me just fine right now. I like having people around, but when alone I can sit and argue loudly with myself, play music when I feel like it, and generally act out. It helps me focus when I'm stressed, and right now I need all the focus I can get.

I miss Obon, though. Last year we saw the World Press Photo exhibition, dried our homemade umeboshi and went up Rokko mountain in Kobe for sightseeing and dinner. No such fun this year.

I did manage to help Ritsuko spread out the umeboshi to dry this morning. And we'll be meeting friends for dinner tonight, so Obon's not all devoid of fun this year.

P!=NP (maybe)

Serious geekery ahead: It seems there's a new, credible proof published showing that P is not equal to NP. Of course, if you understood that, then you've probably already heard about it by now.

It's perhaps the most fundamental and most famous unsolved issue in computer science, and one of the top unsolved mathematical problems. For a really good writeup (much better than mine), check out Good Math, Bad Math. Me, I love working on brain modeling and behavior, but at times like this I kind of wish I'd stayed with Computer Science.

Complexity theory deals with how difficult problems are to solve in principle. It divides problems up into classes, depending on how hard they are to solve. P (for "polynomial") are problems that have an efficient solution: sorting a list, for instance, or finding the shortest path between two points. Since there are efficient methods - algorithms - to solve them it means we can sort huge lists, and your car navigator can find the best route even for very long trips.

NP is a class of problems that doesn't seem to have any efficient way to solve them. The best way is more or less to just try all possible solutions one after another and see which one fits. Lots of important problems, like encryption, are in NP¹.

A famous example is the Traveling Salesman problem: you get a list of cities, and you need to find the shortest path that connects all the cities and returns you to the starting point. It turns out that the only way is to try every possible route, and pick the shortest one (there are ways to trim the list of routes but it doesn't really help). For small number of cities it's easy. But as the number grows it quickly becomes impossible; the number of combinations quickly grow too large (it's called "combinatorial explosion").

Here's the thing, though: nobody has proved that those NP problems really lack efficient solutions. It's always been possible that there's some clever way of solving an NP problem efficiently. And because of a quirk in how some of those problems are structured, if you could solve one of them - any one - then they all become solvable. That includes breaking most types of encryption.

Now this group at HP seems to have a credible proof (pretty much unreadable unless you're doing computer science for a living) that P and NP really, truly are different. NP problems don't have efficient solutions². Cryptography is safe, and travelling salesmen will still have to travel the long way around³. If, that is, the proof holds up. It is over 100 pages long, and long proofs are often found to contain errors. It could be correct; it could have errors that turn out to be fixable over time; or it could have serious errors that doom this proof. But even if this proof fails it's likely to pave the way for new better attempts in the future.

--

#1 I'm glossing things over a bit; NP actually includes P, so when we talk about a problem "being in NP", what we mean is really a problem that's in NP but not in P. The question of whether P = NP is thus really if there are any problems in NP that are not also in P.

#2 How important is this? It is hugely important, but it is verification of something most people already assumed to be true. It would have been much more surprising to get the opposite result.

On the other hand, even if a problem is NP we can still often solve it quickly in practice. For the travelling salesman, for instance, it's extremely hard to find the single best path. But if you're content with just finding a good path, but not necessarily the single best one, then it becomes easy. Other problems in NP may be almost impossible for some rare specific inputs, but easy for almost all cases you're likely to actually want to solve.

#3 A problem like this may seem pointless. Who cares what the optimal city tour is? Computer science often use toy problems like this because they're stripped of any unneeded complexity, memorable and easy to reason about. And real-world problems are often equivalent to a toy problem; if you can solve a toy problem you can solve a lot of real, important problem the exact same way. So it's really not as frivolous as it may seem.

Don't Be Stupid

Time for a bit of self-flagellation, I think. Nothing like public humiliation to remind you not to do bad things. Feel free to skip this if you're not into programming. The takeaway message is that I'm a bumbling dolt - not that most you aren't aware of it already.

Here's the thing: I'm creating some simulations in Python, a common programming language. Well, the simulation itself is run in highly optimized parallel C++ code (we use Nest; it's good, take a look), but we use Python to create the model, set up and run the simulation and collect and save the resulting data.

We run quite a lot of simulations, so I want to collect the data from one simulation run into one catalog. So, towards the end of the simulation we create a new catalog:

os.makedirs(catalog)

Now, the problem is that if the catalog already exists this will fail. Python will complain and stop. We need to check that the catalog doesn't already exist first:

if not os.path.exists(catalog):
    os.makedirs(catalog)

Great! Problem solved. Job well done, beers for everyone, pat yourself on the back. Except...

These simulations are pretty heavy. They take a long time to run. To make things go faster we use many CPU's in parallel. On my desk I have an 8-cpu computer which helps a bit, and I have access to 256 cpu's in a cluster in Tokyo if I need it. Eventually, of course, this project (where this model is just one part) aims to use the new super-computer in Kobe when it comes online next year, and that one has more cpu's than you can shake a very long stick at.

 
This means that the python code above isn't run just once for a simulation, but once for each cpu we use (some of you probably realize where this is going already). Normally this is not a problem: the first process to finish will create the new directory. The other processes will find the directory is already there and skip the creation.

But here's the problem: there's a tiny bit of time in between checking if the catalog exists, and creating it if it doesn't. What happens if two processes try to do this at the same time? Both find that the catalog does not exist, one of them creates the catalog, then the other tries to create it too - it wasn't there a moment ago after all - and the whole simulation fails. That's what happens.

This is called a "race condition", because you have two (or more) processes 'racing' each other, and you get different results depending on which one happens to get there first.

The chance of this actually happening - that two processes manage to get the timing so exactly wrong - seems really slim of course. It may indeed be a really rare and unlucky event if you just run a simulation once and get hit by this. But if you're running lots of simulations, and using lots of cpu's, then you're bound to get hit by this bug sooner or later.

And I did, this weekend. I had started a long series of simulations to run over the weekend. As I came to work this morning I found that the series had failed about halfway through because of this bug. Bad programmer. Bad, bad programmer. No coffee for you.

So, instead of almost three days worth of simulation data to analyze, I now have to spend another day generating the missing stuff - a day that I can ill afford with deadlines looming like thunderclouds over my head.

What should I have done? What should you do? Something like this:

try:
    os.makedirs(catalog)
except OSError, e:
    if e.errno == errno.EEXIST:
        pass
    else:
        raise

This is Python's way of dealing with exceptions - run-time errors such as failing to create a catalog. We don't check beforehand if the catalog exists, but simply try to create it. If we fail, we don't just stop. Instead we catch the error (that's the "except" bit). If the error is that the catalog already exists we just ignore it and continue the simulation (the "pass" thing). If it was some other kind of error we send it on for the system to take care of ("raise", as in raise a flag to alert that something is wrong).

I did not do this. Which made me lose a day's worth of data analysis. And makes me a blockhead. Don't be a blockhead like me. Take care of race conditions when you program. Do proper error checking and recovery. Think of the future. Think of the children.

Neat & Grog

Our neighbourhood bar, Neat & Grog, is no more. Haru, owner for the past 24 years, decided to call it a day this summer. He closed about two weeks ago, and as of the 1st of August the place has been completely cleaned out.

Neat & Grog, 2010.

He had a number of reasons to close the bar but the main one was that, well, he's been there for 24 years and he was getting tired of it. The area has changed over time and he was feeling out of place. When he started, the area mostly attracted salarymen and office workers. Customers tended to be well dressed adults, and the area reflected that. Today it's become much more youth-oriented, with clubs and hip second-hand clothing shops. He and most of his regulars - like us - are in their 40s and 50s, while most people in the area are in their 20s.

Neat & Grog. Exterior.

He's also picked up a new interest: farming. For the past few years he's rented a plot of land outside Osaka, and spent much of his free time growing rice, vegetables and soybeans. So he and his wife (who hails from rural Kansai) want to move out of Osaka city and into the near countryside. They'll do some farming, and he wants to open some kind of pub there as well. If they're going to move eventually, they want to do so before they get too old.

Frozen Margarita.

For our part we're a bit lost right now. A bar is a second livingroom of sorts, and a way to keep in touch with people. The owner of a diner across the street was also a sometime visitor, and last weekend I stopped by with a friend for lunch. It turned out that half of the regulars have been stopping by the same diner, as if they'd been drawn to the area only to remember that the bar is closed. If we feel lost we're not the only ones.

Takeda-san, one of the regulars.

Another couple of regulars, during the last few days.

Haru-chan.