made connections

yesterday i did something that used to be impossible.

i shuffled down a line of people waiting to order food, scanning faces. i stopped at someone, head-down, tapping (a message to me, it turned out) on a white iPhone.

"Diana? nice to see you!" we hugged. to the best of my knowledge, Diana and i had never before been in the same room at the same time.

that's how i met an old friend for the first time. describing it that way makes it sound paradoxical, oxymoronic, …impossible. but in the moment it felt natural, comfortable, real. that the first words out of my mouth weren't "nice to meet you" surprised neither of us. we already knew each other. you don't need to stand on ceremony with friends.

how do friendships build to that point? the answer is a little bit of the real world and a lot of the internet. in this case, our lives both orbited around Ann Arbor, MI at roughly the same time. we acquired mutual experiences, mutual friends…spirals and concentric circles, never crossing. but we became aware of each other's existence, and followed our subsequent revolutions around the country and around twitter, the web, soundcloud. because we followed each other, we were able to realize that our paths finally, serendipitously were passing very close to each other. only a minor course correction — negotiated over twitter DM — and we landed in Cambridge, MA, at the same organic cafe, on the same afternoon.

a generation ago, ten years ago, hell, probably even five years ago, we would have just been one long string of missed connections. yeah, that person i heard about a while ago. no, dunno what they're up to now. but in 2013, we connected. frequently it's difficult to explain to people on the outside what you get from twitter. when you tell them that it connects you with people you care about, their faces usually scrunch up with skepticism. but the internet isn't just a TCP/IP connection, a handshake of packets and bits…it's also a satisfying human handshake, the kind where the webbing of your thumb presses tight and for a few seconds, subliminally, you feel the other person's pulse and another flesh and blood creature becomes part of your life experience.

regardless, i'd be lying if i didn't point out the touch of surreality around an otherwise lovely afternoon that led to wonderful conversation. Diana was the third of my twitter friends that i'd met in the course of the weekend (hi Arika and Heather!). when people i knew from words and pictures kept materializing in front of me, i got the unnerving sense that Some Magic was going on. the fabric of reality seemed to have a little pinhole in it, and i was standing on its frayed edge.

in normal circumstances, i'm convinced of the firm reality of the world…to bend and break Decartes, cogito ergo sunt — "i think, therefore they are." i experience the world, and therefore it's either objectively real or i'm inventing the whole thing as i go, and i feel that the latter is a bit beyond my skills. but words are so close to thoughts, and despite the internet's connective power, it tends to reduce to words. if i were uncareful about how i strung those facts together, i could fool myself into thinking that i was conjuring people out of nothing.

but the only magic that really happened this weekend is that the world got smaller. or, perhaps not smaller — since everyone involved had traveled distances more or less great to arrive in Boston — but denser. i've said before: the internet is like living in a big city. it pushes and pulls people together and lets us create and share and laugh and talk and become friends. it's possible.

speaking for the Lorax

a few weeks ago i needed dish soap. i was out, totally out, had balanced the bottle at improbable angles until nothing but funny noises and the occasional bubble would come out. no big deal, i was due to make a grocery run anyway.

i've used Seventh Generation brand dish soap for some time now. i don't primarily buy it for the "hoorah let's save the earth like the Native Americans would have done" business; i like how it smells, and it gets my dishes clean, and it doesn't destroy my hands even when i need to clean every pot and pan in the kitchen. all of these are good things, including the environmentalism, but i stopped in my tracks when i reached towards the grocery store shelf and saw this:

i picked it up. i put it down. i paced around the aisle. i calculated how many more dishes i could do with what i had at home. (zero.) i contemplated walking all the way to the other end of the store and buying the radioactive blue stuff that kills bacteria, skin cells, and anything else it touches.

in the end, i caved and bought the Lemongrass & Clementine Zest Corporate Shill. i wasn't going to write about this episode as it stood by itself, especially since the internet already has plenty of indignation over corporate tie-ins with the new Lorax film.

fast forward a week or so, and i was in Austin for the North American Summer School on Language, Logic, and Information. on my last morning, i got in a final bit of tourism, including going to the LBJ Presidential Library, which is right on campus at the University of Texas.

i didn't know that the museum was under renovation, and that only a handful of the exhibits were open. they still had a mock-up of LBJ's Oval Office, but 90% of the rest was devoted to an exhibit on the First Lady. the remainder was one little case of miscellany, a hodgepodge of gifts from various heads of states and dignitaries: swords, jewelry, ancient artifacts, cowboy boots, and — at the very end, almost an afterthought — this.

the entire original art of The Lorax is in the library's collection, but just this one spread was on display. the coloring is done in crayon, as if by a child instead of for a child, and the words are held on with masking tape. how humble, i thought, unassuming artwork for an iconic book, donated to a public collection. the antithesis of selling out an idealist character to the highest bidder.

i had the impression that this art had been lovingly and spontaneously donated by Theodor Geisel. i discovered later that the truth was rather different. i suppose "spontaneous" still holds, but apparently the donation was an extreme case of presupposition accommodation:

I was at a dinner for Democrats some years ago, and I sat next to Liz Carpenter, who was LBJ’s press secretary. Since Lady Bird was so interested in beautification, it seemed that environmental protection was a safe topic, so I mentioned that I had written a book on the subject. Liz seemed interested, but soon after left the room. When she reappeared she called me to the phone and said, “The President wants to talk to you.” I said “Hello,” and there was LBJ thanking me for donating the drawings for The Lorax to his library in Austin, Texas.
source: LBJ Library

so the fate of this artwork was made by executive decision. but the Chief Executive acted in the spirit of respect, and preservation. it's not so clear that any preserving force stands watch over the Seuss canon today. what's to be done? near-indefinite copyright terms don't seem to do a lot; as intellectual property, beloved characters can be passed around, traded, sold, and generally mismanaged for decades after the author, long passed away, can no longer say anything. it's hard to say whether limiting that would help, but perhaps there would be a more level playing field; the Once-ler couldn't buy an exclusive product placement in the story where he's the villain.

i don't have a perfect solution, but one thing is clear: The Lorax is an orphan with his creator gone. he's just crayon on paper, and taped-on, typewritten verse. someone needs to speak for him, and his kind, for like the trees, they too have no tongues, and everyone from movie studios to soap makers wants to put words in their mouths.

fence me in

one of the highly touted features of iOS 5 at its introduction was Reminders. the feature that made the WWDC crowd ooh and ah wasn't just an official Apple to-do list, or a fancy timed alarm system (neither of those would be particularly innovative), but the ability to set location-based reminders. buy milk next time you're at the grocery store? get there and *buzz*. or so we thought. then this came across the twitter: [blackbirdpie url="https://twitter.com/#!/waferbaby/status/124518378076520448"]

[blackbirdpie url="https://twitter.com/#!/dokas/status/124520733496979456"]

hold on. i have to go to the grocery store before i can remind myself to buy something there? what if i want to remind myself to call my family when i arrive at an airport halfway across the country that i've never been to before? well, there's one other option: you can set reminders up for addresses that are tied to one of your contacts. i guess that would be fine if it didn't impede your productivity, but even stranger, it tempts you to violate Apple's aesthetic.

the garden of Apple

anyone who's ever watched an Apple product keynote knows about the perfectly cultivated, mythical world that exists within their demo devices. every photo is in focus with smiles. basic word processing tasks become graphic design projects and suitable-for-framing posters. every item is tagged down to the last iota of metadata. everything is beautiful. in one sense, this is just to put on a good face and show the product in its best possible light. in another sense, you get the idea that Apple actually expects you to use the product this way.

 stevejobswwdc2011liveblogkeynote0674

stevejobswwdc2011liveblogkeynote0674

on e57 of The Talk Show, John Gruber and Merlin Mann talked about the similarities between Apple and Disney, particularly in re: the insane attention to detail and upkeep that's required to keep Disney World the most magical place on earth. they mentioned the fact that the Magic Kingdom is never empty; it's staffed 24/7, even if it's only open to the public during the day. keeping everything just so is a literally constant effort, and as such is only possible when done in shifts by a team of workers. Apple may or may not be a 24/7 company (at least not on the design end; who knows when you take into account overseas production by OEM manufacturers). nevertheless, they've shown us that with their own product in their own hands, they can keep up the illusion of perfection, down to a perfectly manicured garden of contacts, faux and real, social and business.

however, Apple isn't in the theme park business. they are in the computing business, which inevitably means handing over their precious product to be maintained by us, the users. we have different tastes. we want to tinker, or hack, or just be plain lazy. we would suck at running Disney World, for the most part. nevertheless, by force of implementation or by mere suggestion, we are still pushed towards The Apple Way.

breaking the illusion

the paint begins to crumble and the facades of Main Street USA fall down to reveal the fluorescent-lit office buildings behind them when the inutility of iOS 5 location reminders comes to bear. the odds of gaining value by setting up a reminder to trigger at the location you're already in, or at the home or business of a friend or colleague, are very slim; chances are a plain or timed reminder would work just as well. what's the workaround? well, you can put that address in a new contact. but then Grocery Store is in your master contact list along with Your Mom (the lack of good nickname support and the notion that you would want to list your close family members as Firstname Lastname is a separate rant).

people do this kind of stuff all the time. people create contacts with first names only; people deliberately put where they know somebody from instead of their last name; people list their best friend as aaaSteve so he's re-alphabetized to the top. but you would never, ever see such a one-off, hacky way of squeezing utility out of the OS put on stage for demonstration. it would be quintessentially un-Apple.

building a better fence

geo-fencing is nothing new, but Apple is just now dipping their toe into it. and, if you still use iPhoto (i'm sorry), you know that Apple's idea of efficient place management is anything but. still, i'm a bit disappointed that i can't set up a reminder of the following type:

remind me to buy bagels at CTB on the second Tuesday of every month only if i'm in Ithaca, NY

i could set up an iOS 5 reminder at my apartment in Ithaca, which would probably trigger on the appropriate day, but that seems like a slightly odd way of doing it. or, if i target the actual bakery, i may never stumble into the arbitrarily sized (and presumably rather small) geofence that surrounds it. i might even have to get close enough to see the sign out on the sidewalk advertising the monthly special before my reminder would be triggered. in that case, a decidedly 20th century reminder would be more effective than my 21st century one.

fortunately, there is hope that we won't have to just build kludgy reminders and faux contacts, while we wait for Apple to catch wind of the fact that we're dissatisfied with the project. the technology that underlies the location-based reminders is a new OS service in iOS 5 that does low-power GPS tracking. and, perhaps surprisingly, it's not a proprietary API call. that means that third party developers are free to use it and put apps in the official App Store that use it. Foursquare already announced a new feature called Radar that uses it to send you notifications based on friend activity in your vicinity. this means that similar functionality, including arbitrary geofencing, ought to be able to be merged into a reminder app. Remember The Milk has already added location services to their Android app; their first priority for iOS 5 and the iPhone 4S was to get Siri working with RTM, but i can only imagine that they'll be adding location support next. and then i will gladly buy their app and subscription service over using a wonky Apple implementation. will i be upset if Apple steals their feature for the next version of Reminders? no, in fact i expect it; it should have been there in the first place. if they can build a better geo-fence than the competition, they will win back my fence-building business.

building a better model

a couple weeks ago, news of some exciting new computational neuroscience research out of UC Berkeley was circulating around the internet — or, at least, credulous reactions to breathless articles cribbed from a press release were.  ("scientists can almost read your mind!", "watch your own dreams on YouTube!", "UC Berkeley invents mind-reading machine!", &c.  no, i am not linking these.  they don't deserve it.)

the research

first, i have to give credit where credit is due: the website of the lab that ran the experiment has an extremely good explanation of how it was performed, and exactly what the technology can and cannot do; it's even perfectly understandable by a lay audience, and is longer than the paper itself [university access required].  you really should go read that entire page to gain an appreciation of the project, but here's the tl;dr.

  1. three grad students strapped themselves into an fMRI machine and watched lots of 1-second clips from YouTube videos.
  2. they made a big database of the fMRI output matched to the clips.
  3. they fed new YouTube clips into a ranking algorithm that made several guesses as to which clip in the database was most similar to the new one.

that's it.  that's the really breakthrough discovery: novel video can be assigned a similarity metric to previous video, not by comparing actual visual similarity, but only by comparing a secondary measure that relies purely on the neurophysical function of the human brain.  it's a fantastic proof of concept.  but then to drive the point home, they did the following:

  1. take the best guesses based on the ranking algorithm, look up the videos, overlay them and output a composite video.

this is the thing that freaked everybody out, because taken on its own — forget what data it was based on — the images are fairly spooky.  good Halloween fodder.  take a look if you haven't seen it already.

i have several directions that i want to take in analyzing this.  i'll start with the spooky result and proceed to the bigger question: should we bother making these models in the first place?

the opaqueness of transparencies

first, consider the reconstruction method used in this experiment: the algorithm makes several similarity guesses based on on fMRI data, and then (with or without weighting, it's unclear), stacks them on top of each other, does some pixel math, and outputs new video.  we would have no idea what those underlying components of the reconstruction were, except that the Gallant Lab kindly provided another video that shows the steps of reconstruction.  let's look closely at two of the clips: Inspector Clouseau and the elephants.

in the second clip that contains Steve Martin portraying Inspector Clouseau, we can see just how much visual similarity falls flat compared to what people actually see.  the algorithm, when using Subject 3's training data, matches this clip to a ones depicting Mythbusters' Adam Savage a full 50% of the time.  why?  because he's a dude in a dark shirt standing over there.

 Steve Martin ≠ Adam Savage

Steve Martin ≠ Adam Savage

and this is actually a case of a particularly good match.  the output is completely dependent on the training data, and the sample fortunately contains at least a handful of videos with people in dark clothing standing in the left-hand side of the frame.  how about elephants?  does the training data include any elephants?

 any elephants here?

any elephants here?

apparently not.  here the best guesses run from fish to airplanes to Tom and Jerry.  the only thing they have in common is an area of contrast in a similar part of the frame.  (i think it's interesting that contrast seems to be the key criterion here; notice how the majority of the matches are light on dark, the opposite of the input.)  this makes the confusion in the Inspector Clouseau case look downright understandable.

clearly the model cannot actually show you what people see, or what exists in "the mind's eye", no matter what pageview-baiting headline writers try to sell you.  people do not look at Steve Martin and mis-see Adam Savage, regardless of whether they can identify them by name.  nobody with any sort of visual acuity mistakes elephants for cartoon mice or vehicles.  this is just not how people's visual systems work.

the visual system is remarkably complex, and understanding it would be a great breakthrough, but does this model get us any closer?  to me, it seems like a flawed approach.  say you wanted to study the technique of master painters, who, given the right time and tools, can paint photorealistic scenes.  you bring them in for a study, show them a photo and say "we want you to recreate this image using your artistic skills".  then, instead of letting them bring their paints, brushes, and canvas, you point them to a file cabinet in the corner of the room that is filled with other photos, printed on plastic transparency sheets.  "stack 'em up, and see how close you can get."  you can repeat this as many times as you like, and you will learn nothing about painting.

the stack hack for language

as a theoretical linguist, i see the fallacy that you can learn things about complex systems this way all the time.  even worse, i sometimes see the claim that as long as you can make decent-looking transparency collages, it's not interesting or relevant to ask about the master artist's methods.  i'm not trying to put down computational models for language as a whole here, nor am i saying that they are completely worthless; that would be taking the converse argument too far.  it's just incredibly frustrating that theory and practice are so at odds with each other.

one of the first things taught when introducing the concept of syntax in introductory linguistics is that there is underlying structure to phrases and sentences, and strings of words will never suffice.  stringing together words is the equivalent of making transparency collages instead of trying to replicate paintings.   yet so many computational systems that try to deal with constructing sentences (in translation or other applications), do it via ngram models: taking groups of words that co-occur, evaluating their likelihood, and concatenating them.  the parallel with the spooky movie reconstructions should be obvious.

of course, just like the neuroscientists can't peer into the exact workings of the brain, neither can linguists.  i.e., none of us could set up a painter experiment where we set up multiple cameras and record them bringing in their tools, setting up their workspace, and thereby create a log of each brushstroke in order to build a perfect reconstruction of the steps involved in crafting a final product.  but we can do the equivalent of sending the painter into the room, saying "take your time, make a good painting, we're not watching", and then submitting the result to careful scrutiny.  ah, they always establish base layers of paint in this order.  ah, for this effect, the brush strokes always go in this direction.  from that, a theory of painting, if not an exact manual, could be reconstructed.  then if we wanted to create a computer program to mimic the process, it would have the conclusions drawn from these observations in mind.  it would deal in layers and brushstrokes, not hacking up an approximation with photos.  theoretical linguists attempt to do the equivalent of this by looking at lots and lots of sentences, instead of lots and lots of paintings.

walking the walk; not talking the talk

as i pointed out in my Ignite Ithaca talk on writing, humans excel over their primate cousins most markedly in two areas: the ability to talk and the ability to walk on two feet.  the disparity in the prevailing philosophy on how to best model these two skills couldn't be greater.  how?  people love walking robots.  Honda's ASIMO was hailed as a massive achievement, not just for being adorable, but for being the first robot able to walk on two feet like a human.  projects stemming from it have continued to improve on its abilities, making it more humanlike at each step: able to walk up stairs, able to walk backwards, able to bear more weight.  at Cornell, a robotics team set an unofficial world record for longest unaided walk by a robot, around the school's indoor track.  (i would go to run laps and see the crew with their remote controls, plodding along next to this gangly computer on legs with a baseball cap stuck on top.)  at my alma mater, the University of Michigan, they're setting records for the fastest bipedal running robot:

there is a fascination with how we walk and run, and making machines move the same way we do.  it is not the absolute most efficient way to get around; NASA doesn't send bipedal robots to Mars, they send highly articulated six-wheelers that can navigate almost any rock put in their path without falling over or getting stuck.  if all we want to do is build machines to get from point A to point B, putting legs on them is foolish.  but that's not the point.  we want to understand what it is like to be human, and to build machines that are approximations of us.

i just hope that the neuroscientists, the computational linguists, and everyone who's trying to emulate humanity with software instead of hardware doesn't lose sight of this.  be inquisitive.  ask exactly who we are and what we do, not how to hack it to 75% or 90% or even 99% accuracy.  figuring out the exact ways in which the world works has been one of the pillars of scientific inquiry since the Renaissance.  by giving it up, science risks confining the master painter to sitting in a room, staring at an ugly collage that he's not very happy with.