Dwarf Fortress, Facebook, Big Data and the Search for Story


June 14, 2013 at 11:00 am (5 Comments)

Last night after driving home from the Austin PyLadies meetup, my wife sat in our driveway for 20 minutes listening to the end of an episode of WNYC’s Radiolab.  Later, after we’d headed to bed, she spent another 20 minutes retelling the story to me, minus Radiolab’s flourish and production.  The story was still interesting second hand, and comes down to this (I’ll wait if you’d like to go listen to the episode of Radiolab, I’m sure it’s excellent):

Two people discover hundreds of letters from WWII on the side of Route 101.  They’re from soldiers replying to a woman on the homefront.  The soldiers call her mom, but she isn’t their mother.  The two ask around, no one knows anything about them.  One of them, a creative writing professor, ends up using the letters as projects for his students.  He gives them a letter, and their task is to create a story around it.  A soldier, a woman stateside, an unlikely connection.  The other discoverer wants to track down relatives, she wants to uncover the truth.  She ends up discovering it, but he’d rather not know.  He wants the possibilities.

Even told second hand, the story stuck with me on a meta-level.  There aren’t a lot of things that would make my wife sit in the car in the driveway for 20 minutes listening to the radio, but a good story is one.  We love stories, we love it when they’re well crafted and well told.  But we also love the possibilities of them.  Sometimes we don’t want the truth, we want magic, we want to dream the dream of what could be.  Sometimes the truth can’t exist, and the closest we can get is a dim outline of it.  Sometimes the dream is better.

The Promise: Stories that Tell Themselves

A few days ago I ran across a blog post by Tynan Sylvester, a designer on the game Bioshock Infinite.  It’s all about the dream of simulations for game designers, how we think that by creating more and more complex systems, we might eventually build a system that is complex enough to manifest stories.  Austin Grossman’s latest novel, YOU, is about that, in a way.  The protagonist is a game designer and the antagonist is just a manifestation of some long-running game rules.  As game designers, we want to design games that surprise us.  That’s the ultimate payoff, to build a game that entertains you, and not just a twitch game that is enjoyable for its mechanics, but a game with stories compelling enough to sit in the car in the driveway for 20 minutes at 9 o’clock at night.

Lots of game designers have tried to do this. Tynan talks specifically about systems in early versions of Bioshock where the player would have to play autonomous bots (splicers, gatherers and protectors) off each other to progress.  They hoped that amazing, emergent gameplay would be the result.  In the end it didn’t work, and the game moments that they’d hoped would happen spontaneously ended up being heavily scripted.  Players crave story, but that story can’t be left up to their persistence and chance, especially when creating a commercial title.  In that environment, a great story has to be guaranteed.

Dwarf Fortress: Madness in Text Mode

There are a few notable exceptions to this principle, and they’re mainly smaller games driven by singular minded creators.  The best example of this is Dwarf Fortress, a massive and inscrutable simulation game where the the player takes on the role of an overseer, and the titular dwarves are simulated autonomous entities inhabiting the world.  Dwarves have names and hair colors, what Tynan calls Hair Complexity, things that add perceived simulation depth without effecting anything else.  (When was the last time you played an RPG where a plot point hinged on your hair style?)  They also have more integrated systems like hunger and social needs.  They have personalities, they get sad, and sometimes they go crazy.  The dwarves live in a randomly generated world, so your game isn’t like my game, and even my second game won’t be like my first.

Dwarf FortressDwarf Fortress has a very dedicated core following, and one of the reasons is that it really lives at the edge of apophneia, the experience of seeing meaningful patterns emerge from random data.  At the core of Dwarf Fortress is a collection of rules governing behavior.  A dwarf without food will eventually starve.  A dwarf without personal interaction may eventually go crazy.  Dwarves are scared of wolves.  Dwarves exist in a world generated fractally, a world that feels real because it mirrors patterns in nature.  Therefor, as more and more rules get layered on, and more and more people play more and more games and get better and better at creating experimental mazes for these digital rats to play in, stories begin to appear, or so we perceive.

Two of the most famous stories to come out of Dwarf Fortress games are Boatmurdered, the tale of an epic game played out by members of the Something Awful forums in 2007, and Bronzemurder, a beautiful infographic-style tale of a dwarf fortress and a terrible monster.  Go read it, it’s great.

Dwarf Fortress didn’t generate these stories, though.  People played the game, sometimes hundreds or thousands of times, and while gazing into the mandala of the game, they nudged and pulled the threads of the world and created stories based on the events that occurred there.  Dwarf Fortress isn’t a windup toy, it’s a god-game, and the players impact on the game world is more than negligible.  The stories generated there are as much created by the players as by the game.

I Fight For the Users

While my wife was out at PyLadies last night, I coincidentally watched TRON: Legacy.  It occurred to me as I was thinking about writing this post, that it’s a movie about this possibility: The dream of a world inside a computer, a world created by a brilliant programmer, a world that once set in motion can create stories, unexpected events and enthralling narrative.  The creator steps aside, and no longer controls the game from the top-down.  The creator becomes a god among men, watching things unfold from their level.

Tron: Legacy - Quorra

In TRON: Legacy, the magic of digital life comes in the form of Quorra, the last of the ISOs, Isometric entities that appear spontaneously from the wasteland of the computer.  Digital DNA, digital life.  Enough rules, enough circuitry, enough care and magic happens.  That premise is exciting, and to programmers it’s intoxicating.  For those of us in the digital generation, that’s the dream we live with.  That’s what we keep trying to make happen wherever we go and whatever project we work on, be it big data or software bots.

But the lone programmer, no matter how brilliant, and working for no matter how long, can only produce so much code.  Stories from one person only grow so far, only change so much, and rarely surprise and enthrall.  Dwarf Fortress as a dwarf isn’t a game most people would play.  It’s hard to see the overall story, and the game isn’t good at presenting it.  But if there were more players…

EVE Online: More Interesting to Read About Than to Play

If it’s possible (albeit insanely difficult) to have stories appear in a single player game, it must be easier for stories to manifest in a multi-player game, right?  Games like World of Warcraft have largely fixed, planned out stories.  It comes back to the challenge that Bioshock had, complex systems are exciting to designers, but players want immediate story gratification.  Complex systems take dedication to understand, dedication most players don’t have.  When new multiplayer games are announced they sometimes hint at players making a real impact on the world, but those systems usually fail to live up to the hype.  The latest game to promise this is The Elder Scrolls Online.  We’ll see if they can do it.

One game that does this and thrives is EVE Online.  EVE is a massively multiplayer online space combat simulation, one that spans an entire universe.  It’s possible to play EVE as a loner, but it’s also possible to align yourself with a faction, and have your small efforts merge with hundreds or even thousands of others to build armadas and giant dreadnaught ships, to control entire solar systems and even galaxies.  The designers and administrators of EVE take a largely hands-off approach.  They don’t want to kill the golden goose, so they design the game for balanced conflict, and let the players sort it out.

EVE-Online-Battle-of-Asakai-3Every once in a while something epic happens in EVE, either a massive fraud, an invasion a faction planned for months, or a random accident that led to a game-rebalancing war.  There are battlefield reports, and once the space dust settles, people start to put together a history, and an accessible storyline appears.  Here are a few great EVE stories.  More people probably enjoy the reports of epic battles in EVE through these stories than actually play the game.  To quote a MetaFilter comment thread: “This game sounds stressing as hell if you really play it and not just dither around. Fascinating to read about, however, almost like news from a parallel universe.

You could say that EVE is a computer program for generating stories, and in fact the’ve even made a deal to do a TV show based on player stories from the EVE universe.  Except again we find that that EVE isn’t the thing generating the stories, EVE is just a place where the stories happen.  To a player only experiencing the events inside the game it may seem mysterious and amazing, and it certainly is to those of us who read about the events afterwards, but it’s really just a sandbox.  People play pretend with enforceable rules, but you can’t separate a story that happens inside of EVE with the real life stories that happen outside of it: The scheming that happens on IRC or in forums, the personal vendettas, the flexible allegiances  and the real-world money that flows through the system.  There’s no way to watch something occur inside of EVE, and even if you had perfect clarity on everything that happened inside, have any way of knowing for sure what really caused it.  If you take away the players, the legions of dedicated fans scheming and plotting, you just have an empty universe.

Facebook and the Timeline of Truth

I think a lot of web developers secretly wanted to be game designers.  Becoming a game designer is difficult, there aren’t as many jobs and the hours are terrible.  Instead we build web sites, but we’re building systems too, and we want to tell stories.

I joined Facebook back in April of 2006.  I had a @swt.edu address from Southwest Texas State (now Texas State University) from an extremely brief stint (sub 1 day) as an IT staff-member, so I got in a few months before they opened it for everyone.  Getting into a new, exclusive social network is a bit like finding a new simulation.  We hope the software can tell us new stories, that it can make some sense of the data it has.  With Facebook the promise was that if it collected enough information about us, it could tell us that magical story.  That’s what Timeline was supposed to do.  Give Facebook enough photos, enough checkins, enough friend connections, enough tagged posts and it would be able to tell the story of our lives.

Facebook Timeline

In the end, though, Timeline doesn’t tell you a real story.  It reminds you of stories you’ve heard and experienced, but Facebook is only a dumb algorithm working with imperfect data.  It’s smart enough to target ads, but it can’t understand the meaning, and it can’t remix the data in really compelling ways.  It can’t be Radiolab.  Most of the time the prioritization it comes up with I just want to turn off.  Its attempts at story are so bad I’d rather use my own organic cognitive story filters.

With every new Facebook feature announcement, with Google+ or the next thing that processes all your activity, the promise is that the system can get better at telling those stories.  We want to believe it will happen.  We want to believe that a couple thousand web developers and a couple billion dollars could create a story machine, but I’m not sure it can.  I was reading an article about HP’s R&D budget the other day that said Facebook invests 27.5% of revenue in R&D, a larger percentage than any other company they tracked.  You can bet a good chunk of that is going towards the search for story, in some form or another.

Weaving a Web

I’d be remiss if I didn’t mention Weavrs are designed specifically for apophneia, they produce content one step up from random, and rely on our desire for patterns to throw away the things that don’t fit.  We project stories on to them, and for a project with the limited resources that it had, it’s exceedingly good at it.

My weavr twin is posting about HP Moonshot servers.  That’s almost eerie, but it’s also posting about hockey tickets.  The story makes sense if I’m picky about the things I include, but it isn’t an internally consistent narrative.  The narrative is impressed on it by the people who see it, like reading digital tea leaves.  Your story of my weavr is different than mine.

With enough resources and time, weavrs might become a real story machine.  That’s a moonshot program, though, and I don’t know who’s going to step forward and make that happen.  Investment follows money, and right now the money is racing towards big data.

Autonomy: Billions and Billions

The lure of story, the promise of meaning from the chaos of data isn’t limited to games or the social web.  It’s the romantic beating heart of big data.  It’s the stories about Target knowing you’re pregnant before you do.  It’s what lured HP to spend $8.8 billion dollars more than it was worth to acquire Autonomy.

Autonomy’s main product is called the Intelligent Data Operating Layer, or IDOL (symbology, ahoy!).  They call the processing of information with it Meaning-Based Computing.  From what I’ve heard it’s certainly good at what it does, but while it promises Meaning from Data, and that promise separated HP from 9 Instagrams or 2,500 Flickrs, there has to be some apophenia at work here.  Just like watching solar system battles inside of EVE gives you a piece of the story and playing hundreds of games of Dwarf Fortress will result in games worth telling stories about, the system data is never the entire picture.

Screenshot_6_13_13_11_52_PMI really like Stephen Wolfram.  Stephen believes in the fundamental computability of everything.  While I love reading his blog posts, and I am interested in and admire his idea, I have to wonder how far the hyperbole is from actual execution.  Given enough computable facts and enough understanding about the structure of narrative, a perfect Wolfram|Alpha should be able to tell me stories about the real world.  But it can’t.  They aren’t even trying to approach that.  Wolfram|Alpha isn’t creating Radiolab.  They want answers, not stories.  You know what tells stories? Dirty, messy, all-too-human Wikipedia.

A Different Kind of Magic

My friend Librato.  Librato is a big data startup, having pivoted from some other work to running a service that collects vast amounts of metrics and provides dashboards on top of it.  With Librato Metrics you can feed data points, set alert triggers, create graphs, and watch activity.  It’s big data without the prediction.  It promises no magic, but relies on our own.  It optimizes data for processing by human eyeballs.

The 3 pounds of grey matter between your ears is still the best computer we have, running the best software for deriving stories and making sense of data.  Librato works because it doesn’t try to be what it can’t.  Google Analytics tries to offer Intelligence Events, but more often than not, it can’t offer anything more helpful than that visits are up from Germany 34%.  You would think that by combining traffic source analysis with content changes and deep data understanding Google would be able to tell you why visits are up from Germany, but most of the time that basic percentage is the best it can offer.  It still takes that 3 pounds of meat to pull together the data and interpret it into a story.  While computers may be generating articles on company reports or sports games, they’re not creating Radiolab.

Wrapping Up

I think there’s still a lot of room for innovation here.  The Archive Project I dreamed of long ago is essentially a system for telling stories and discovering meta-stories.  Maybe someone will finally build it.  Maybe the next Dwarf Fortress will be a world that runs persistently in the cloud, a world where our games interact with other people’s games, where crowdsourced Hair Complexity snowballs until you can get lost in the story if you want to. A game where if you want to turn off a random path and follow it down to the river you’ll find a fisherman who will tell you a tale interesting enough to make you sit in your car for 20 minutes, enthralled by a narrative.

Maybe the framing of a story is what big data needs to become personally relevant.  Maybe that’s its magic trick.  Maybe narrative is the next great big data frontier.

Future Past

iPad

I sometimes wonder about the generation of kids growing up today, in this big data, analytic-driven, always-on world.  I wonder how they will embrace it, like we embraced computers and connectivity.  I wonder if they’ll have the ability to hear the prognostications of the computer, to listen to the story from the machine, and consider it a kind of truth.  To internalize it, but also keep it separate.  To know the machine knows a truth, but not necessarily the absolute truth.  Maybe that will be their power, the thing they can do that those of us from the generation before can’t. Maybe that is where the dream finally comes true.

Means of Prod-Sumption: The Samsung Chromebook


June 10, 2013 at 10:33 am (3 Comments)

Four years ago I bought a Toshiba Portege M200 off eBay.  The Portege was a neat little machine, one of the early twist-and-flip Windows tablets.  It was relatively light, relatively fast, and was well built.  Unfortunately it didn’t have a CD-ROM drive, would only boot from certain select external USB CD-ROM drives (which I didn’t have) and needed a specific Windows disc (which I didn’t have either).  After a while I ended up wiping it and loading Ubuntu on it.  I wanted to use it as a simple Web terminal, especially for National Novel Writing Month.  It was passable, but difficult to use and maintain, and eventually was lost underneath a pile of junk.

The Samsung Chromebook on an 8.5" x 11" sheet of paper.

The Samsung Chromebook on an 8.5″ x 11″ sheet of paper.

Ever since, I’ve been looking for a simple, lightweight machine to write blog posts with.  I use a MacBook Pro for my workhorse computer, but it lives tethered to a big monitor and big keyboard on my desk.  Disconnecting all those cables, dealing with a half dozen app windows that suddenly resize and are in the wrong place is enough of a pain that I just don’t do it.  I wanted something lightweight, something with a great keyboard, ok screen, wifi, easy maintainability and great battery life.

Fast forward a couple of years, and Google has introduced Chrome OS, a managed Linux OS built specifically for running Chrome and accessing Google services.  It’s designed to run on minimal hardware, and a couple of manufacturers have put together stripped down machines with Chrome OS on them.

Samsung put out one last year, an 11.6″ machine that runs $249.  It’s designed to be disposable, but it’s been the best selling laptop on Amazon since it came out.  A month ago I broke down and bought the first one off the truck at my local Fry’s.

The Hardware

Samsung ChromebookThe 11.6″ Samsung Chromebook weighs in at a svelte 2.5 lbs, and looks like a plastic MacBook Air.  It uses the same dual core ARM processor that powers the Nexus 10 tablet.  It seems snappy enough for web browsing, which is essentially all the machine does.  It boots from a 16 gig SSD, which means there aren’t any moving parts.  The machine has 2 gig of built in RAM, with no expansion option.

The screen is… not good.  It’s a matte 16×9 LCD, and has a crazy low resolution.  The pixels are so big that it’s almost… retro, and you can see faint lines between them.  It makes me feel like I’m living in a late 90’s romantic comedy when I use it, which has a cool vintage sensibility, in a way.

The keyboard is where this machine really shines.  It may not be as good as the keyboard on my Macbook Pro, but it’s big enough and good enough for serious typing.  This is the third blog post I’ve written on it, and haven’t had any complaints or issues at all with 2,500 word posts.  The caps lock key has been replaced by a search key, but you can apparently switch it back in the OS control panel.

Samsung Chromebook SD Card Slot, Headphone Jack and KeyboardThe sound isn’t great, it has some small speaker vents under the palm rests, but that isn’t what you buy this machine for.  The chipset’s graphics are fast enough to decode high def video, so YouTube works just fine.  I haven’t watched any TV episodes or movies on it, but since you have Chrome OS you won’t be running Quicktime or VideoLAN Client.  It’s only going to support HTML5 video.  No Flash.  There’s a combo microphone/headphone jack on the left side, so streaming video and audio should work just fine.  There’s a simple webcam above the LCD, but the machine doesn’t really have enough horsepower to run it.  It would work for a simple Google Hangout, but in the little testing I did, the frame rate is low.

Samsung Chromebook RearThe computer also has a pair of USB ports (one USB 3, one USB 2), and an HDMI port, though I’m not sure why you’d use it.  It has an SD card slot, so theoretically I suppose you could plug your camera’s memory card in and upload the photos to Picassa.  It has built-in WiFi, and there’s a slot for a SIM card, as the laptop comes in a model with 3G wireless.

The Experience

It’s a strange thing, using a computer that only runs a web browser.  Most people would consider me a power user.  I have a pretty customized setup, I use Terminal on my Mac all the time, I write little shell scripts to test things, I’m regularly in Photoshop or Illustrator tweaking this or that.  I use a lot of non-web applications, and I’m always flipping between them.  Not so with the Chromebook.

Samsung Chromebook Google HomepageWhen you startup the machine it asks you to login to your Google account.  That’s it.  You can login as a guest, and access the web, but it’s very, very, very tied to your Google account.  The mail button opens Gmail, the files button opens Google Drive, you get the picture.  The integration if you’ve bought into the Google ecosystem is pretty amazing.  Just type in your email address and password and voila, all your stuff is there.

You can’t install Apps on it that don’t come from the Chrome store.  There are options for installing an entire chrooted xwindows environment with something called crouton, but you have to switch the machine over to development mode.  Using development mode isn’t as simple as a key press.  When you switch from managed to development mode the entire machine is wiped and the development image is downloaded.  Any tweaks you’ve made, gone.  You have to hold down a special key combo when you start the machine, and I’d be leery of just handing the machine to someone else.  I want the Chromebook to be essentially disposable, so no dev mode for me.

Samsung Chromebook SSHThere’s a pretty decent SSH client in the Chrome OS store, so you can open multiple SSH tabs, and do whatever development work you need to do on a server elsewhere.  It’d be nice to have a really limited shell to test bits of python code, but I see how that’s a slippery slope.  Give them an inch, and suddenly it isn’t a fully managed experience anymore.  The SSH client is nice, supports colors, and is supplied directly by Google.

The machine comes pre-loaded with a cloud-syncing Google Drive app, and you can run Gmail offline.  It’s really built to be connected, though, so this isn’t a machine you’d want to take on a long trip with no internet access.  The WiFi seems good in my limited testing around the house.

As a web browser, the machine works well.  The 16×9 screen isn’t really well suited for reading long articles, but two finger scrolling with the trackpad works well enough.  You can run Chrome full-screen, which gives you enough space that it doesn’t feel cramped.  I’ve heard that if you open more than 20 or so tabs the machine will run out of RAM and you’ll need to reload those tabs when you go back to them.  I haven’t run into this problem, but I’ve read that they’re considering turning on disk-swap, which will help alleviate those issues.

The Competition

This particular model runs a dual-core ARM processor, while other models come with Intel chips.  HP, in particular, makes a 14″ model with a Celeron.  I played with one, and it feels much more like a real laptop.  In the end I wanted an ultraportable, so the Samsung was the obvious choice.

When comparing the Chromebook to other computing experiences you inevitably end up looking at cheap Windows 8 laptops, Android tablets and iPads.  When I was standing in Fry’s there were 3 options under $300, one an AMD 15″ Windows 8 laptop, one an Intel-based HP Chromebook, and then this ARM based Chromebook.  The Nexus 7 is the same price, the iPad Mini’s less than a $100 more, and the iPad Retina and Nexus 10 run $200 more.

I’ve found tablets to really shine as media consumption devices.  That’s what I use my iPad for.  The retina screen is great for reading articles and news.  For my money, you can’t really beat it for that purpose.  Windows 8 machines, and the cheaper OS X machines higher up the price ladder are way better at media production.  You can run Word, or Google Docs, or Write.  You can install software on them, you can customize your experience.  You can run databases, instant messenger apps, chat clients.  Switching between apps is easy, as is pounding out an essay or blog post.

The Chromebook really sits in the middle.  It’s way easier to write a blog post on than an iPad.  Web browsing and tab switching is probably faster, if not as fluid.  Copy and paste is a little easier.  There aren’t nearly as many native Apps as the other two options, but if you just want an ultraportable machine to write with, something that forces you to focus on the writing, and minimizes the distractions, it’s a $249 dream.

I was really hoping that Samsung would update the 11.6″ Chromebook at Google I/O, maybe bump it to a quad-core processor, but that didn’t happen.  In the end I decided that for $249, I could deal with obsolescence.  If something better comes out that I just have to have, I can easily pass this machine down to a cousin or kid.  It’s the kind of machine I’d give my mom.  There just isn’t much you can break, and it’s great at what it does.  Plus, at $249 I don’t feel like it’s a sacred object, like my Macbook Pro.  In fact, I fully intend to plaster stickers all over it.

The Personal Cloud: Innovation Happens at the Edges


June 3, 2013 at 8:19 am (2 Comments)

Personal CloudA couple of days ago I was snuck that laptop into an MIT closet.  In my case there was no login or private, privileged access, and I was running all this stuff in the middle of the night as to not inconvenience anyone else, but the problem remained: If I’d followed the rate limiters desires, it would have taken weeks or months to grab the data.

I ended up getting around the rate limiter by using something called Tor, The Onion Router.  Tor works by sending your traffic through a distributed network of hundreds of other participants computers, anonymizing your physical and digital location in the process.  For me, that meant that I could download all the files in 15 minutes or so in the middle of the night.  For other people it means posting to Twitter or accessing dissident web sites from Syria or China or where-ever.

Running across these files reminded me of something I’ve been thinking about for a while: That what the Personal Cloud really needs to take off is an immediate problem-solving use case, and to find useful examples, we might want to look in the grayer areas of the internet.  We can talk about companies bidding for our orders VRM style or Internet of Things devices dumping metrics into our personal data warehouses, but both of those things are going to require a lot of supporting infrastructure before they’re really viable.  If you want to get a lot of people excited about something today, you solve a problem they have today, today.  And that brings me to the edges…

The Edge: Where Innovation Happens

One common aphorism shared by those in technology or innovation is that new things develop at the boundaries.  Change and chaos happens at the edges, it happens at the borders, where things mix and intermingle.  Here’s the MIT Media Lab’s Joi Ito talking about it, for instance.  MIT is a large, stable organization.  Businesses are large, stable organizations.  The Media Lab is where they meet, where they cross-populate and where the friction in developing new ideas is reduced as much as possible.  The same can be said about border towns.  New York City is an American border town.  It’s the edge between our country and a whole bunch of immigrants, both old and new.  The mix of ideas and talents and experiences creates new things.


Joi Ito: Innovate on the Edges and Embrace… by FORAtv

A lot of the innovation that happens on the edge happens in the gray area outside the strictly legal, or deep in the illegal.  Across our southern border we have very advanced drug and gun smuggling tunnels, complete with ventilation and electricity.  Neal Stephenson’s last book REAMDE was largely about northern border smuggling.  Chocolate, toy filled Kinder Eggs are illegal in this country, so people smuggle those in.  I brought some back the last time I went to Mexico, and some friends brought back a whole carton when they went to Germany recently.  In Gaza they even have KFC delivered by tunnel:

Given that innovation happens at the edges, that people solve their problems at the edges using interesting methods, and that the Personal Cloud needs some need-driven use cases in order to flourish, I think it’s useful to look at some of the ways people are using things like the Personal Cloud already for dubiously legal purposes (though the legality they’re avoiding isn’t always our own).  Perhaps by digging into what makes them compelling, and how their developers have solved those problems, we can learn something about developing Personal Clouds for everybody.

Some Personal Cloud Definitions

When looking for products that fit the Personal Cloud mold, I’m specifically looking for interesting uses of on-demand computing and networking.  Especially things that don’t inherently scale beyond the individual, either due to privacy concerns, the need to be distributed, or some other unique aspect of the approach.

A job that only takes 10% more time to run for another person isn’t a good candidate for a personal cloud, because the economy of scale is going to keep it expensive.  Running your own mail server is a bad idea these days, because your data and address can be portable (with IMAP and a personal domain) and running the spam filtering and staying on whitelists is hard.  It’s a lot better to register your own domain and let a trustworthy third party do it.  I should mention that Phil Windley has a good post about IMAP being a proto-Personal Cloud protocol, if you haven’t read it.

So, with that said, let’s look at some examples…

Tor: Anonymize All the Things

Tor OnionSo back to Tor.  Tor is built as a distributed, self-organizing network.  There are Tor nodes that you connect to, the address for which you get either by getting passed an IP address on the side, or by looking one up publicly where that won’t get you thrown in prison.  Once connected to the Tor network your public internet traffic is bounced through the network of Tor nodes in a randomized, encrypted way, and eventually finds its way onto the public internet through Tor Bridges.

The people who run Tor Bridges are paying for your traffic twice, because your connections come into their machine and then out again.  Running Tor Bridge is a labor of love, done by people who believe in anonymity and freedom of speech.  It doesn’t pay, but knowing that a political dissident somewhere can speak freely about an oppressive regime has a karmic payoff.

A few years ago Amazon’s EC2 cloud computing service started offering a free micro level of service.  You could sign up and run a really small cloud server for development or testing without paying.  It didn’t cost Amazon much to run them, performance wasn’t really great, but it got people onto their platform.  Usually people start up Amazon provided server instances to install software and play around on, but the folks behind Tor realized that they could create a pre-configured server image with the Tor Bridge on it, and let people spin those up in Amazon’s free usage tier.  They call it the Tor cloud.  You still pay for bandwidth, but if you bridge 15 Gig of bandwidth a month, your bill will only be around $3.  It’s less than the price of a latte, and you do something good for internet freedom.   You don’t have to know a lot about the cloud to set it up, you just register for Amazon Web Services, pick the image, and hit Start.  The images are pre-configured to download software updates and patches, so there’s virtually no maintenance work.  Just the kind of simplicity you need for a Personal Cloud feature.

Back It Up Or Lose It: The Archive Team

Archive TimeI’ve harped on our tendency to Jason Scott called Archive Team.  Archive Team scrapes sites that are destined for the Internet trash heap, and uploads the data to the Internet Archive.  So far they’ve archived sites like Apple’s MobileMe homepages, Yahoo Groups, and are currently trying to grab as much of Posterous as they can before Twitter drops the axe.  This may sound pointless till a few years after a company acquires and then shutters the site your mom or sister blogs at or posts family photos to, and you realize there’s no way for you to get that stuff back.

Archive Team runs into a lot of the same issues I had around rate limiters.  Yahoo! and Twitter don’t want them slurping down the whole site, they want to take the engineering resources off those projects and let them die a quiet, cost-cutting death.  To get around this, Archive Team offers a virtual machine, the Archive Team Warrior.

The Archive Team Warrior is a distributed but centrally managed web spider.  The Archive Team central server slices the archiving work up into little chunks, and the Warrior on your computer asks the server for some work to do.  The central server gives it a small to-do list of URLs to fetch, and the Warrior starts downloading those until it hits the sites rate limit.  Any data it can download, it sends back to the Archive Team server for bundling and uploading into the Internet Archive.  Then it waits and retries until the site will let it back in.

Warrior ScreenshotThe Archive Team manages the projects, and the Warrior presents a simple web interface where you can tweak a few settings and track how you’re doing.  Most importantly, it’s hands-off.  You can set it up once, and let it run in the background forever.  It manages its own software updates, and you can tell it to work on whatever the Archive Teams priorities are, and ignore it from then on.  If you have a PC sitting around that you don’t use a lot, running the Warrior is a nice way to give back to the Internet that’s given us so much.  It’s good karma, and it’s easy.

Pirate All the Things: Seedboxes

So far we’ve talked primarily about projects which give good karma, now let’s talk about a project that is often used for… not so good karma.  In 2001 the BitTorrent protocol was introduced, allowing for a (then) secure way to share lots of files in a bandwidth-optimized fashion.  Users get pieces of a file, trackers know who’s downloading the file at any one time, and clients cooperate to distribute the pieces as widely as possible.  When you’re downloading a file from BitTorrent it’s entirely likely you’ll be downloading chunks of it from people who don’t have the entire file yet, and likewise you’ll be sharing parts of the files you’ve downloaded with other people who don’t have those pieces yet.  By working this way everyone gets it faster.

Not Sure if Network Is Busy Or If They're On To MeWhile BitTorrent might have been secure once, it’s now entirely likely that your ISP knows what you’re downloading, who you’re downloading it from, and what you’re sharing back.  They can look at payload sizes, the trackers you’re talking to, traffic bursts, and pretty reasonably reconstruct your activity.  If they’re the MPAA or other pirate-hunting groups they can even run their own clients and integrate themselves into the network.  Running a BitTorrent client from your home computer and downloading anything remotely illegal is like asking the bagger at the grocery store to help you out with your shoplifted goodies.

So let’s say you’re sharing something that you think should be legal but isn’t, or you’re trying to use BitTorrent for a legal end, like sharing a bundle of book materials or distributing an Operating System or a big chunk of GeoCities and don’t have the bandwidth at home to support it.  (Or, sure, you could be downloading Iron Man 3.)  This is where something called Seedboxes come into play.  A Seedbox is a server at an ISP somewhere that just runs a BitTorrent client.  You can use them to get your torrents out to a bunch of people really fast, or you can use them to download files that you wouldn’t be comfortable with downloading to your home IP.  You can even buy them in another country, increasing the difficulty of tracing the traffic back to you.

Seedboxes are managed servers, you don’t install software updates on them, the provider does that, but they likely won’t give you much in the way of customer support.  Lots of them use a Web UI called ruTorrent, an open source frontend for the rTorrent BitTorrent client.  You don’t SSH into these machines, you probably don’t even have a server login, but you can use the web UI, and conduct your business in the cloud.

In this way ruTorrent Seedboxes are a perfect prototype for our Personal Cloud.  The providers don’t watch the servers or monitor their quality.  Privacy is implicit when you’re doing something at the edge of legality.  What they don’t know won’t hurt them as much when Interpol comes calling.  The web UIs are built for self-service.  You have a login, but the web UI is your entire management plane.  rTorrent has an Android front-end, but most people likely manage them through the web.  There isn’t any software on your home computer, just a username and password to a web site somewhere.  The data’s yours, and if you wanted to shove it sideways into a cloud storage provider, you probably could.

Points of Presence: The Personal VPN

SpoilersAs an addendum to these offerings, a sort of post-script on the idea of exploiting technologies at the edges for personal gain, I’d be remiss if I didn’t mention personal VPNs.  Tor’s good for anonymity, but what if you just want to appear like you’re somewhere else.  Say, for instance, somewhere the new season of Sherlock, Doctor Who or Downton Abbey is available for streaming 6 months or a year before it comes to your country.  (Or vice versa, where we get new episodes of Mad Men a year before they do.)  What do you do then?

The same technology that your company uses to securely connect you to your corporate network can be used to make you appear to be in the UK, or the US Midwest, or Japan, or wherever else the content is region-limited.  You run the software (likely built-in to your Operating System), and connect somewhat securely to a computer in some other country or even continent, and all your internet traffic appears to come from there.

A few years ago I was in Mexico over Christmas, and there were some really good deals on Steam’s Holiday Sale.  I have a US account, with a US billing address and a US credit card, but I couldn’t buy anything because my computer was with me in Mexico.  I ended up installing a bunch of software on one of my servers and setting up a VPN to it, just to buy some cheap games.  These days I could just plunk down a few bucks and be good to go, and a lot of people do.

A Few Learnings Lessons Learned

Users have problems, and will go to considerable lengths to solve them.  None of these services are as easy as they could be, either because they’re niche offerings (Tor and Archive Team) or because of their dubious legality (Seedboxes).  ruTorrent is a lot easier to use than it probably was, but it still isn’t as easy as using the Netflix or iPlayer iPad apps.  The Warrior is a 174 meg download that requires installing Virtualbox on your computer.  The Tor Cloud Bridge requires signing up for Amazon Web Services, and navigating their UI.  To get a VPN provider or Seedbox requires research, dealing with a company that might not be entirely legit, and really falls in the class of early adopter technologies.

Even though all this stuff is hard to use, people do it.  Seedboxes and private VPNs give people things they want.  You may not have known that you wanted to watch the new season of Dr. Who before it comes out in the US, but once you know you can, you’ll go to some pretty extreme lengths to make that happen.  Motivation can be powerful, and people will overcome serious technical hurdles if they’re properly motivated.

So looking at these examples, we can see that a Personal Cloud app really needs to offer 3 things:

1. Motivation: It needs to solve a real, immediate problem.

2. Self-Service: It needs to be super-easy to start using and offer a familiar, understandable interface.

3. Hands-Off: It needs to have software updates and easy maintenance built-in.

Any Personal Cloud offerings that don’t check these boxes may get some niche use, and may excite developers, but they aren’t going to start climbing up the adoption curve.  As you build your Personal Cloud app, keep these things in mind.  Users have needs we can solve, and we can empower them, but our solutions need to be compelling, simple to use, and simple to maintain.

Security Through Isolation


May 21, 2013 at 9:54 am (One Comment)

Yellow SubmarineThey say no good deed goes unpunished.  In internet hosting, that’s almost always the case.  For the last fifteen years I’ve had servers that I’ve given friends accounts on.  At first they were co-located machines I built by hand, then leased servers, and now cloud VMs.  I hosted friends and family’s blogs, sites for activism causes that I or friends believed in.  I’ve even had a web site of a well known Silicon Valley venture capitalist on there.

Unfortunately whenever you do that, especially whenever you hand out accounts or host web applications that people were once enthusiastic about but then moved on from, security is going to become a problem.  A few months ago while doing maintenance on the machine I noticed that an account for someone had been logging in, except it shouldn’t have been, since I created the account while trying to troubleshoot a problem that we solved another way.  But there it was, in the last login log.  Digging into that directory I realized that the password had been simple (in the heat of troubleshooting you don’t always make the best decisions for security), and someone had brute forced the SSH login.  The machine had been compromised and used as an IRC bot host.  How very 1997.

Today I decided to migrate between cloud providers.  While I could host all this stuff for free at HP Cloud, but I do some dangerous stuff in the context of my account, and it’s nice to have a stable VM elsewhere.  It’s also nice to see what the competition are up to.  I’d been hosting this VM at Rackspace ever since they bought my preferred VM provider, Slicehost, but while poking around pricing I realized I could cut my monthly bill in half if I migrated to Linode.  I’ve admired Linode’s geek-friendly control panel, ever since I tried it while designing the beta versions of HP Cloud’s.  I was also on an ancient Ubuntu 9 version at Rackspace, and this would be a good opportunity to upgrade the OS and software.

After copying all the home directories and web sites over, I did a last pass to pick up any straggler processes.  These usually live in cron jobs, so that’s where I went looking.  Lo and behold, an account I’d setup for a friends mom to host her business web site had been compromised at some point.  Another bot.  Joy.

Fortunately her web sites had long since been migrated off the server, so I was able to disable her account (and remove the stashed authorized_keys file with a bot installer in them) and a bunch of others I knew wouldn’t be used, but it really goes to show how vulnerable these machines can be.  Who knows how they got her password.  It might have been an easily crackable password, it might have been a web script compromise, it might have been an email exploit.  More than a few of these usernames and passwords are sitting in mailboxes or in saved FTP connection files on easily crackable machines.

Two weeks ago I got an email from a former client.  We built a pretty complex web site for her in 2003, lots of bells and whistles.  It’s held up pretty well, but it hasn’t had any serious maintenance work in a lot of years.  She’d gotten a call from the FBI, saying that data from her web site was circulating in Russia.  Fortunately it was just an exported mailing list, not encrypted passwords or other secure data.  In her case I think one of her employees had either an easily guessable password, or a trojan was installed on her computer that logged her keystrokes.  How do you guard against the guardians?  Nobody was thinking of two-factor authentication for small business web sites in 2003, but the next time I build one, I sure will be.

Sharing accounts on a machine, or having admin accounts into a web based system is an inherently insecure thing.  The more keys there are to a lock, the more likely someone you don’t want to have one will get one.  I created user accounts on our shared server because that’s how you did it, back in the day.  Create a user account, setup a directory for the web site, add a database for them, and let them go.  Now we have linux kernel exploits that let anyone with user level privileges become superusers.  Adding random accounts to systems and handing out the passwords is an insane thing to do.

So the only hope we can have of having any kind of security is by shrinking the permissions scope down.  When everyone has user accounts on a machine, that entire machine is vulnerable.  When everybody has a small VM, only that VM is vulnerable (usually).  Even better, give them a single-process Linux Container, like those managed by docker.io, and suddenly they don’t even necessarily need to manage dependencies anymore.

I’m sure docker has its own set of security issues, but hopefully we’re more cognizant of them now.  Don’t create unnecessary user accounts.  Use password protected SSH keys.  Don’t re-use ssh keys.  Keep your dependencies up to date.  Watch the security mailing lists.  It really starts to sound like something the hosting provider should be doing…

So I think there’s a real opportunity here for a trusted IaaS operator to create a generic Linux Containers As A Service offering.  Push down one level from VM into Process.  Bring your own docker image, buy a set amount of RAM (say, 128 meg for a big PHP or Python process) and bill by the minute. Route them inside of the machine through some kind of nginx or go based proxy, like CloudFoundry does, but a little less specialized.  Something between CloudFoundry and an IaaS VM.  Upgrades in CloudFoundry are a pain.  If I could just shuffle a docker.io image around, that’d be way easier.  Oh, and don’t sign up for too many hosted services.  Each one of those you use is like another shared account, and the more you share your data with, the more likely it is it’ll be exposed.  Build small, build focused.

So back to security.  This is a plea to all those who have friends who’ve given them accounts on servers, or people who run servers and create accounts for friends.  I know the complex password requirements are a pain at work or on bank web sites, but they’re really even more important in less maintained environments.  Nobody’s watching that shared server, keeping it water tight is a shared responsibility.  If someone creates an account for you, change the password immediately.  They won’t remember to go delete it if you end up not using it, and if they set it to something simple, there’s a good chance someone will be able to brute force it.  Don’t store passwords in plain text anywhere.  Don’t email them to people, or have people email them to you.  You’ll be happier in the long run if you don’t.  Use best practices, and save us all some heartache.

Book Review: The Rapture of the Nerds by Cory Doctorow and Charles Stross


May 10, 2013 at 9:22 pm (No Comments)

The Rapture of the Nerds, CoverI hate to admit it, but The Rapture of the Nerds is a book I thought I wouldn’t like.  It should have been a must read for me, but I waited quite a few months before picking up a copy.  It’s by two of my favorite authors, Charles Stross, creator of the Laundry books and the excellent Accelerando, and Cory Doctorow, author of Little Brother and generally good guy.  Cory’s even shown the good humor to let me bot-ify him, which is a project I need to get back to.  I was just… worried.

It may just be my religious upbringing, but the title of Rapture of the Nerds carried a ton of baggage with it.  When you couple the singularity, which has gotten beaten up a lot lately, with a religious concept like the rapture, from these specific authors, it seems like a recipe for some lets-make-fun-of-the-utopian-nerds riffing.  That’s kind of in vogue these days, so it isn’t too much of a stretch to think you could make a novel out of it.  Reading a whole novel of that really didn’t appeal to me, but it turns out the book isn’t about that.

While there’s undoubtedly a subtle undercurrent of it in The Rapture of the Nerds, what we really have is a tale of a luddite’s gonzo journey to the heart of the post-singularity, complete with mommy/daddy issues.  You could call it Boy Meets Post-Singularity World, and that would probably be more accurate.  There’s some gender morphing, militant deep south isolationist conservatism, hyper-intelligent ant farms, and bio-tech viruses.  There are also a lot of scenes in courtrooms.  All in all, par for the course for a world where technology makes anything that can be imagined happen.

The Rapture of the Nerds really reads like a looser Charles Stross novel.  His space opera titles like Saturn’s Children are usually really tight, this one’s more loosey goosey like a Laundry novel, probably the result of bouncing back and forth with Cory.  If Cory’s written much beyond-the-horizon sci-fi, I haven’t read it, so this novel seems more Strossian than Doctorowian to me.  I think some of the flavor may have bled from or to The Apocalypse Codex, as well, given that novel’s bad guy.  This book seems more brainstormed over lots of pints down at the pub than carefully planned.

There’s a lot of the third act of Accelerando here, or the first bits of The Quantum Theif, if that makes sense.  A good chunk of the novel takes place in… well… cyberspace.  There’s a love story, and a happy ending, both things I appreciate (I’m looking at you, Paolo Bacigalupi.).  It’s a lot better than I was worried it would be, though it probably isn’t either of their best.  There’s a post-singularity Lovecraftian dread throughout this book that Stross has really nailed with the Laundry novels.  In this book it isn’t so much defeated as just… survived.

If this sounds like something you’d enjoy (and I’d certainly recommend it if you like gonzo post-singularity fiction), you can pick up a copy at the usual suspects.

HP Tech Con `13: The Magic Kingdom


May 9, 2013 at 3:33 pm (No Comments)

After I joined HP two and a half years ago I started to hear tales of a magical event.  Tech Con, a technical conference for the top technologists in the company, showing off the best and brightest innovations of the year.  Moonshot servers, 3D displays, that kind of thing.  One guy on our team had gotten in on an honorable mention a few years before.  Nobody ever went to present.  Not from our small corner of HP, anyway.  But we could always dream.

To get in to Tech Con you have to write up an innovation you created, and then compete against nearly two thousand other proposals for one of less than 150 poster slots.  Out of those poster slots, fewer than 50 get speaking slots.  To put this in perspective, HP has 320,000 employees, more than 70,000 technologists, less than 1,000 get to go to the conference and fewer than 50 get to stand up on stage and talk.  HP Labs is always well represented, innovation is their lifeblood.  HP Cloud… had never sent anyone.

PodiumLast year I started working on something pretty cool, so a logical step while applying for a patent was to submit the innovation to the conference.  Fast forward a few months to late February.  While sitting on the couch at the end of a long day, I was checking my email on my phone and a message popped up.  It was an invitation to Tech Con.  Even more mind boggling, after I re-read the email for the third time, I realized I’d been invited to speak.  Queue the montage of presentation creation, practice, tweaking, throwing the entire presentation away, starting from scratch, practicing, tweaking, etc.

You can read about what other HP People higher in the company than me have to say about Tech Con: HP’s CTO Martin Fink highlighted it on the HP Next blog, and HP Fellow Charlie Bess has posted about it.  For those who haven’t had a chance to go to an event like Tech Con, a big, technical company event, I’d like to give a brief rundown of what it’s like, since it can be a once in a career experience.

Room KeyTech Con this year was in Anaheim.  The location is a closely guarded secret before the conference, because it’s chock-full of company trade secrets and unreleased products.  We were at a hotel a few minutes from Disneyland, and HP had booked nearly the entire thing for the conference.  There were a few vacationers, but pretty much everyone you saw was wearing the HP Tech Con lanyard.  Even our room keys had the conference logo on them.

LunchTech Con is a conference where you work hard and play hard, in as compressed an amount of time as possible.  We all flew in Sunday afternoon, and then had a reception and dinner that evening.  Being a teleworker I don’t get to see HP at it’s most mind-bogglingly huge, but this conference went all out.  HP logos on the walls, HP banners, even HP napkins.  When you work in a little 350 person department in the cloud it’s easy to forget that the company makes paper, ink and printers of all kinds.  (Not to mention servers, PCs, storage, networking, software, services…)

The conference is a mix of formal presentations and academic style poster sessions.  During the poster sessions my co-authors from HP Cloud and I got to stand in front of our poster and talk to technologists and executives from all over HP, discussing how we could work together, explaining our product and getting some really great feedback.  HP’s COO (and our boss’s boss’s boss’s boss) even dropped by our poster and talked with us for about 20 minutes.  It was amazing to get that level of recognition, and I think I managed to not completely embarrass myself.  Meg Whitman was on the poster hall floor, but unfortunately didn’t get to where we were.  She did talk to all of us, though, which was great for those of us who aren’t from the bay area and don’t see her regularly.

Tech Con Dinner

There were two blocks of scheduled activities, the first an outing to the Queen Mary that I missed to spend more time with some fellow HP Cloud folks (though everyone I talked to said it was really fun), and second an activities slot with things like whale watching, art walks, and the Warner Brothers studio tour, which I went on.

Nerd Herder

The highlight of that tour might have been seeing a real Nerd Herder from the TV show Chuck, but we also got to see the main set for The Mentalist, which was really cool.

Speaking at Tech ConI gave my talk Wednesday morning to a nice, big crowd, had some great QA, and then talked to a bunch more people at our final poster session.  It was great to hear from people who’d had 30 year careers at HP working on amazing products that they totally got our thing and thought we were on the right track.  That’s the kind of affirmation that can only come from a smart, diverse group of people like those at Tech Con.

Tech Con Room

Tech Con is a highly confidential conference, and they take IP violations really seriously.  I wanted to bring my poster home, since it was really pretty, and would go really well with my Whole Foods Market Milk poster.  I had some people ask about the possibility on my behalf, but apparently after the last session they were going to lock the door, ship them all back to Palo Alto, show them for a few days, then cut them up, boil them in acid, burn them, and bury them in the desert.  Oh well.

The conference wound down Wednesday afternoon, but I managed to walk over to Disneyland with one of my coworkers from HP Cloud to get souvenirs for the family, having a nice chat on the way.  The highlight of the conference is really those times, getting to talk to people you work with, or people from other parts of HP that you never have contact with.  I had conversations with people from Australia, the UK, Germany, India, China, Italy, Israel and Brazil.  It was really, really cool.

Thoughts on Digitally Native Artisanal Artifacts


April 29, 2013 at 2:34 pm (No Comments)

Tucked away in a corner of a bustling storage depot in the halo of the San Francisco bay area is a metal shipping container.  Nestled inside that container, surrounded by the furnishings and bric-a-brac of a contemporary urban life, is a small, silver figure.  It is the only one of its kind in the world.  It is the result of a nearly 20 year design process, distilled through three minds from a short collection of descriptive words.  It is also the future.

A Little Backstory

It’s good to have friends who understand you.  I like to try things.  I like to pick up new technologies and roll them around, get a feel for their heft and texture.  This often involves doing a project, but these projects can sometimes be a little weird.  Finding good natured co-conspirators really helps.

Matt & Jeff Party Hearty

For the last few years, Matt Sanders has been my target guinea pig.  Matt is perfect, because we’ve diverged paths enough he doesn’t know everything I’m up to.  However, we spent years in the trenches together, so I know him pretty well.  If we saw each other all the time there might be an obligation tied to the things I come up with, and that would make it weird.  Matt also happens to enjoy the artistic and technological, so I know the fundamental concept of the attempt will be appreciated.

In 2010, I got Matt DJ Brixx, a friend I met through an electronics comparison shopping project wrote and recorded it.  Brixx lives in the Philippines, and once professed a desire to eat at the Cracker Barrel.  He’s a crazy guy.  You meet crazy people by trying to do crazy things and finding the people who are willing to go along for the ride.

Ideas

The rap song set a high bar, but sometime in 2011 I was driving up Loop 1 with Irma, and realized that I could top it by making the virtual real.  I could create a small figure of one of his RPG characters.  Years ago, starting in 1994, Matt and I spent a lot of time together in an Internet-based text-based role playing game called Ghostwheel.  I’ve 3D models of the Dust Bunnies characters, and knew that there were freelance 3D modelers out there who were experienced at character design.  Shapeways let you print things in metals, including a very nice sterling silver.  So the pieces were there, I just needed to get it done.

Production

SketchI ended up working with a 3D modeler named Bhaskar Rac.  He had worked at a video games studio, and had a good feel for character design.  We did a contract through oDesk, which handles payment and taxes and whatnot.  I sent Bhaskar Matt’s character’s description, some photos of him for general reference, and some sample pictures of the things he had on his person.

After a few days of discussions about Fallout and thematic inspiration, Bhaskar sent back this sketch.  A few days after that he followed up with a draft 3D model.  I thought it was awesome, so we tweaked a few small details, and I uploaded it to Shapeways New York manufacturing facility.

Shapeways prints their sterling silver models in a three step process.  First they print the model in wax using a high resolution 3D printer.  Then they submerge the wax model in liquid plaster to make a mold.  The wax is melted out and molten silver is poured in, resulting in the final piece.  It produces a very high level detail, and is a process often used for jewelry.  It also works really well for anything small you intend to last for a long, long time.

A few weeks later, this appeared in the mail:

MattFig 1 MattFig 2 MattFig 3

Reflections

The little 2 inch high figure is now in transit with the rest of Matt’s stuff as he moves to San Francisco.  The digital model exists, but no other physical traces grace the earth.  There is only one.  Unless Bhaskar sketched out something on paper, it’s the only physical manifestation of this entire project.  That’s a pretty weird thing.

If Matt were to somehow lose the figure, if someone broke into his house or if there was a fire, or someone unleashed a bio-engineered virus that only ate silver, we could print another one.  As part of the “gift” I sent him the 3D file, so if he wanted he could populate his house with tiny Matt figures in every size and color.  He could open source it, upload it to Shapeways and let anyone print a tiny Matt figure for their Monopoly set.  It’s a present that comes with it’s own infinite digital reconstruction blueprint.

But what if someone stole the digital one?  What if it leaked out, and people liked it so much they started printing their own?  How would that make Matt feel?  How would it make me feel?  Does he have “the original” even though there is no original?  Is it a “first” like a blog comment?  Is there still something unique about the one that arrived in his house packed in a tiny little box for his birthday in 2011?  I think so, but it lives in a weird space.

Implications

I think this kind of gift, the present deeply rooted in the past, in a shared history and experience, but interpreted by skilled artisans into something new, is going to be the new normal.  While hiring artists and 3D modelers is challenging now, there’s nothing stopping someone from creating Photoshop or Maya as a service.  Perhaps Shapeways will even evolve in that direction.  Supply the talent, ship the product.

We’re surrounded by mass market objects.  Books, movies, furniture, even sometimes what we consider to be art.  We collect it and we arrange it, but it isn’t truly unique.  The hand blown glasses at Ikea say they’re made by hand and each is unique, but you’re buying them from Ikea, so the really weird ones probably get tossed in the recycling heap.  Sometimes we may shop at craft fairs, but even crafters will reproduce an item if it sells.  It’s hard to create things from scratch, and producing one-offs is expensive in a traditional model.

But now that the means of production are so cheap, and the training to use them is largely free and open, we can truly have unique things without spending a lot of money.  We can create home movies that are beautiful, we can hire artists to create beautiful things just for us or the ones we care about.

It’s possible we’re just setting ourselves up for a backlash.  The figure I made for Matt isn’t a Warhol, and while we both get it and enjoy it, I’m sure some would argue that we’d be better off with a good reproduction of something truly important instead of a meta-reference.  But Warhols are meta-references, so maybe we’re just becoming hyper-personal with them.  In the end that’s what we get to weigh.  Is the quality of the work more important, or the personal connection you have to it?

Book Review: YOU by Austin Grossman


April 22, 2013 at 9:39 pm (No Comments)

YOU Novel Cover

Austin Grossman has a new novel out.  It just hit last week, and it’s called YOU.  YOU is like Ready Player One and Fight Club having a baby while making an Ultima game.  If you’ve read both of those books (or watched the movie, in Fight Club’s case) and liked them, do not pass go, do not go to your cave and find your power animal, buy this book and read it.

YOU is a book about making computer games, about making the Ultimate Game, a game where you could be anyone and do anything and the world would still work, the story would still unfold completely naturally.  Austin has worked as a writer and designer on some of my favorite games, including Deus Ex and Thief: Deadly Shadows.  His experience in the games industry really shows, and you can see bits of real games peaking through the imaginary ones.  There’s a section in the book about a demo at E3, and it sounds exactly like they’re playing Thief, scrambling over rooftops, firing flaming arrows at torches, evading the city watch.  Austin’s latest game is Dishonored, which is sitting on my shelf, and has now risen much higher in the next-to-be-played list.

The story of YOU is told through a prodigal protagonist.  Out of options, he returns to the game company his friends started after high school, after they all built a pair of RPGs together.  He gets a job as an entry-level game designer, and proceeds to unravel a mystery about friendship and adolescence and being a nerd.  The game shifts between its present day of 1997 and the 80’s years of high school, the story unfolding through flashbacks and dives into the games the company created.

If you read REAMDE and enjoyed the parts in the MMO, or if you enjoyed Daemon, or Ready Player One, or Tad William’s Otherland books, you’ll like this book.  It’s obvious he’s writing from experience when he introduces a game, and while some of the details may be embellished from what was possible then, they play like we want to remember them.

The ending of this book doesn’t land as well as it could, it doesn’t leave you with a particularly warm sense of accomplishment, but it isn’t bad.  The macguffin is resolved, but the mystery sort of peters out.  This isn’t a book you read for the ending, though, it’s a book you read for the journey, for the time warp back into high school, into games on floppy discs and BBSes and a million possibilities inside the magical machine that no one over the age of 25 understands.  As an ode to that bygone era, it is unmatched.

VallisMOO: A Game Designer Is You!


April 22, 2013 at 11:49 am (No Comments)

The world is a scarred shell of wind and sand and heat.  Whoever had their finger on the button finally pressed it.  There’s only one safe place left, a tiny, sheltered valley between two giant mountains.  To the south is the ocean, to the north, the wasteland.  A magical gate keeps the monsters out, and keeps the valley safe.  We live well here, in our little sea side town or deep in the forest.  We roam the grassy plains, dotted with bald hills.  We hunt, and forage, and build.  Sometimes we fight, because what’s worth fighting for more than the last good place on Earth?

YOU & Me

A couple of days ago I read a review of Austin Grossman‘s new book, YOU, by Cory Doctorow, and decided to buy a copy.  Austin’s a game designer, and he’s worked on System Shock, Clive Barker’s Undying, Deus Ex, and Dishonored.  He’s now an author, with his first book SOON I WILL BE INVINCIBLE, and now YOU.  YOU is a book about a directionless 27 year old who gets a job at a game company started by some of his high school friends.  While designing a new role playing game, he delves into the mystery of what happened to his friends and their dream of the ultimate game.  A game where you could be and do anything, and the story would unfold before you naturally.

I haven’t finished YOU, but it’s been bringing up all kinds of memories.  I got my first internet account in early 1994, dialing up through Real/Time Communications in Austin.  R/T hosted a game, a text-based Zork-ish virtual world game called Ghostwheel, or GhostMOO.  I’ve talked about GhostMOO before in my Pocket Worlds post, but I had forgotten an interesting chunk of GhostMOO history, and my own stab at multiplayer game design.  My own dream of the Ultimate Game.

Ghost^2

In 1997 GhostMOO was on the decline.  We’d had a big rush of users in 1995, but many of them had graduated from college, and user numbers were down.  The two main drivers of the game, Quinn (lead programmer) and Razorhawk (content creator) were busy with other projects, and without strong direction, GhostMOO was stagnating.

scribbleA couple of GhostMOO programmers, including Matt Sanders (who I’d go on to start Polycot and then join HP Cloud with) and I decided we’d start a spin off.  Quinn had been gracious enough to release the core of the game, the bits and pieces that made up combat and non-player characters and monsters, out into the open as the GhostCore, so we had a good place to start.

We were thinking of creating a Ghost^2 or Ghostwheel 2.0, if you will.  Similar core concepts, but different execution.  Ghostwheel was all over the place thematically, we had monsters cribbed from Princess Bride, Alien, a whole community of Dragons straight out of Pern, a quaint japanese island, basically whatever a programmer was really into, they built.  Even the name was cribbed from Zelazny’s Chronicles of Amber.  In retrospect what we had was a mashup, we were just ahead of our time.  Ghost^2 was an attempt to wipe the slate clean, to start with a core concept and theme that would be internally consistent.

Like all groups starting with a blank sheet of paper, we wanted to create the Ultimate Game, and we had the audacity to think that we could create a better game than the ones that had come before.  We wanted to create a game where if you wanted to play a blacksmith, you should be able to do that all day, role play with other people, and generally succeed and feel progress.  If you wanted to fight monsters, great, but that wasn’t the only path.  Years later the first Star Wars Galaxies game would do this in MMORPG form, only to be neutered and turned into more of a combat grind in an attempt to compete with WoW.

Terrain Zones

One of the really interesting things that Quinn built at GhostMOO was the Terrain Room.  GhostMOO was a MOO, a text based game, kind of like a multi-player Zork.  In a game like this as you walk around you’re presented with room descriptions, which include the objects in the room (furniture, people, monsters, etc) and the available exits.  Like so:

The R/T Round Room
Eight walls for each point of the compass, each with an open doorway.  The
 floor is tiled with checkered perma-linoleum; little matching octagons.

                    N   
        NW        Lounge       NE
      Austin        |       Hot Tub             UP
             \      |     /                     Helipad-
               \        /                         Austin
    W              YOU             E              Jizo Island
 Library -----     ARE   ----- Infirmary          Ghostwheel Plain
                  HERE
               /   |    \                       DOWN
        SW   /     |      \    SE               Ground floor-
     Obsidian               Greenroom             Exit to Wasteland    
                   S                               Guest Chamber
            PX/General Store
The center desk is empty of all personnel.  Someone must be on an extended
 coffee break.  An electronic sign-in pad is bolted to the desk.  A monitor is
 bolted into the desk.  Nobody's sitting on the floor.  Alongside the east
 wall is a queer little potted tree, a tallish leafy husk with a, uh, snout?
Contents:
 Bulletin board                                             
Obvious exits include down (d, trapdoor) and Up (Helipad, u).

Each of these rooms is dug by a programmer from an existing room, like mining through the digital aether.  That new room then has an exit back to the room you were in before.  (Here’s a map of the main house and grounds of LambdaMOO, for comparison.)  That style of design makes for very detailed, interesting areas, but the overall area tends to be small, because every room needs to be described individually.

Quinn created something called a Terrain Room, which lived inside a Terrain Zone.  In a Terrain Zone you insert a little ASCII map, something like this:

        @qedit me.tmp
        # ############
        #    # #     #
        ###~~#~#~~~#~#
        #          # #
        ############ #
                     #
        ##############
        .

This map defines what kind of rooms there are and what the layout is.  In this case the pound signs (#) may be walls, while the tildes (~) may be rivers, and the blank spaces grassy fields.  With this lovely hack Quinn transformed the Zork style MOO into a player-perspective Rogue-like.

The cool feature of the Terrain Zone is that it only creates the rooms if they need to exist.  In a MOO everything takes up memory space, and back in the mid 90’s memory wasn’t as plentiful as it is now.  If you’re standing in a terrain room, the terrain zone can look at the rooms around you on the map, and tell you what’s there.  “To the north, east and west are open fields.  To the south is a stone wall.”  Those rooms don’t actually exist.  Once you decide to walk north, the Terrain Zone creates that room, moves you into it, and destroys the room you were standing in before (unless you dropped something, or there’s some other reason for the room to still exist).

By operating like this you could create huge areas without actually digging and describing every room yourself, and the memory consumption would be a lot smaller.  Your monsters and other non-player characters could also know about their “home” terrain type, so they wouldn’t stray from the grass or river.  You could also let the player “travel north” and they would keep walking through rooms until they hit a room that was different than the type they were already in, or contained something unique.

In Ghost^2, we decided to use a Terrain Zone for our world, since it let us create an overall map that had consistent distances and spacing.  I sketched the map out on a sheet of paper (above), and then started drawing it in a paint program on my Mac (below).  I had a program that let me convert graphical images into ASCII art, so I was able to go from a map directly into the MOO.  This is the last version of the Ghost^2 map, created in October of 1998:

terrainimage5

The map is 1,000 pixels by 1,000 pixels, so our game would would be 1,000 rooms by 1,000 rooms.  In the orange spots where there were towns we could dig out special rooms for houses and buildings, we could dig out special dungeons from the fields or forests.  In this map light green is grassland, dark green is forest, gray-green is hills, blue is water, yellow is roads, and grey is rocky terrain.  The outlines around large sections are edge types, like the edge of a forest, a beach or the base of a hill.

It was a really cool concept, and thinking about it still gets me excited.  The idea of adventuring in that world, exploring the bustling cities, verdant fields, dark forests and dimly lit caves sounds like a lot of fun.

VallisMOO

Game design isn’t easy.  It’s easy to dream about, but it’s hard in practice.  I’m about 1/3rd of the way through YOU, and the main character’s starting to realize that he has to enumerate every kind of object in the world, in every state it has.  We reached a similar place with Ghost^2, which eventually became VallisMOO when interest waned among the other developers.  We decided not to create a sister-MOO to Ghostwheel and I kept working on my own, and that MOO became VallisMOO.

I appreciate the stamina exhibited by the Adams brothers, the team behind Dwarf Fortress.  They’ve been working on that thing for years, but that level of dedication is really hard to maintain.  I got to a place with VallisMOO where I needed to begin populating the combat system with weapons.  I heard that Steve Jackson games was working on a Low-Tech book, due out “any day now”, and instead of forging ahead (ahem) and making do, I decided to wait till it came out, and use it for reference.  That was 1998.  GURPS Low-Tech finally shipped in 2002, and by then I was on to real paying projects and VallisMOO was only a memory.

Legacy

I went digging through my project archive and found a directory full of VallisMOO code and to-do lists and graphics.  It even had some logs of conversations where I discussed the ideas for Ghost^2 with some friends, things I’d long forgotten.

I thought I’d share the design documents with the world, so I’ve uploaded them to github with a Creative Commons license.  There are files of character types, races, locations, maps and all kinds of crazy things.

If you thought this post was interesting, and enjoy 90’s era game design, you’ll probably really like Austin Grossman‘s YOU.  I finished it last night, and once I get a chance, I’ll write a more complete review.

Meditations on Sleep


April 5, 2013 at 4:11 pm (2 Comments)

How you sleep is a secret.  You can’t tell from walking next to someone on the street if they can drift off in seconds while the lights are on and the TV’s blaring, or if they need a sealed cocoon of solitude kept at an exact temperature with no piercing noises or lights.  Well, sleep doctors might.  I’ll ask mine the next time I see him.

Based on the totally non-scientific notes I’ve taken from since I’ve started dealing with my sleep issues, I can say that a lot of people can’t sleep at night.  Some people have full blown sleep apnea, and I really feel for them, but I think there are a lot more who just can’t shut down at night and drift off.  Maybe ten thousand years ago we could have, but now that we’re in the big cities with the job stresses and the always-available entertainment and the ten thousand projects going at once, we’ve lost that ability.

My journey towards trying to get better sleep started when I moved out of my parents house after high school.  I’d gotten a pretty good gig doing consulting development at Whole Foods, and had some money to spend.  I figured that since I’d probably spend at least a third of my life in it, I went a little crazy, and bought this:

The Bed

This is actually the short version of The Bed (note the capitalization), the first version had full height box springs.  I’m 5′ 10″ and I felt like I needed a ladder to get into it.  Our house in San Marcos at the time was a 800 square foot (if you measured the outside of the building) 2 bedroom.  There was barely enough room in the biggest bedroom to walk around it.  It was the most wonderful thing ever.

I’m one of those people whose brain refuses to shut down.  Given normal everyday stresses, I will lay in bed, awake, till 3 or 4 in the morning before finally falling into a half-sleep.  If there’s something urgent happening early, or I’m wearing something scratchy, or there’s a strange noise, or there’s too much light I might not sleep at all.  I want to, by body likes to be in bed, but when I close my eyes it feels like someone hits fast-forward on the VCR in my brain and I can’t slow it down.

Acquiring The Bed was a great first step in solving some sleep problems.  It was a Serta double-sided pillowtop, and I think it set me back $3,500.  It was probably the best $3,500 I’ve ever spent, but even it wasn’t enough to guarantee sleep.  I still have problems, and between the time I started writing this post and when I actually finished it, there were a lot of nights of no sleep.

So in the interests of helping out someone else who might share my problem, here are a few things I’ve tried, and some comments on them:

Environment

I like it dark and cold.  The colder the better, really, but usually in the high 60’s.  This can be expensive in Texas.  If I were clever I’d have an in-room AC unit and only cool off the bedroom.  White noise is good, but those tinny little white noise generators just don’t do it for me.  I need base, I need rumble.  The circulator fan in our AC unit is right next to the bedroom, and for years I slept about 4 feet away from it.  Now I have a small air purifier next to the bed that I run on medium, which does a decent job.

For a while when our daughter was small and in the same room with us my wife wanted to have a night-light, which drove me up the wall.  Fortunately now our daughter’s in her own room.  The baby monitor we use throws off a really bright light, but some things you can’t avoid.

The Bed

Finding a really good bed can be hard.  The market is designed so you can’t do real comparisons, and even with a 30 or 90 day money back guarantee, who wants to go to the trouble of returning a giant mattress?  We bought The Bed at a dedicated furniture store, and paid a lot of money for it.  We bought our second bed at Sears, and got it on a big sale.  The first bed was great, the second bed isn’t great at all.  You can’t compare list prices between stores, sales don’t really matter if they’ve just marked the cheap one up a bunch.  Go to a reputable place, get some advice from friends, and lay on them for a while.  I like really soft beds, they’re pretty hard to find.  Don’t give up, you’ll be living with that thing for the next 10 years.  Keep your purchase and warranty information.  Scan it, take a picture with your iPhone, whatever.  In 8 years when the springs start popping out the side, you might be able to get a new one free.

I’ve heard that foam beds are hard to… procreate on.  I’ve never bought one, I don’t really know.  I like giant, squishy but not too squishy pillowtops.

Doggie SleepHaving a good bed is a big deal, and having a good bed-frame can be important, too.  Right now we have a barely-held-together craigslist bed frame, and probably need to trade it in for something more supportive and stable.  If I feel like I’m on an incline or if there’s a weird aberration in the bed, that’ll keep me up.  Like I said, a princess.

The Pillows

Having a good pillow is key.  I like big squish-able down pillows, because I sleep on my stomach or side.  IKEA has a great selection for cheap, so you can try them and not feel guilty.  Buy extras.

Preparation

My sleep doctor’s advice is to keep the bedroom as the place you sleep, don’t watch TV there or play on your devices or read books.  I’m not good at following advice, and pretty much do all of these things.  If you’re having trouble, it could be something to try.  Meditation and breathing exercises work for a lot of people, I haven’t had success with it.  The force is not strong with this one.

Support

Having a partner who gets that you’re lying in bed going crazy every night is a big deal.  For a lot of people sleep issues aren’t something that ebb and flow, every night is a battle.  It can be really hard to be responsible for stuff early if you tossed and turned for hours the night before and only got 3-4 hours of unhappy sleep.

My wife has the ability to fall asleep anywhere.  She’s fallen asleep during conversations with me, while playing games on her iPad, I think she could probably sleep standing up.  I’m incredibly jealous of that ability.

Some couples who aren’t able to deal with sleep problems, especially snoring, sometimes sleep in separate bedrooms.  I get that it’s a solution, but it seems kind of unfortunate.  When we travel I find that we only really fight over the covers, so I always call downstairs and get an extra comforter sent up, and we use two on our bed at home.  The bonus there is that hers has a warmer fill and mine is lighter, so we’re both happy with temperatures.  This isn’t the TV sitcom ideal of mom and dad in the perfectly made bed straight out of the catalog, but it works for us.

I usually get my best sleep, almost lucid-dreaming style, after my wife’s gotten out of bed in the morning.  That means that most of the time she takes care of the baby and gets the household going.  That’s a big deal, and I really appreciate it.

Pharmacology

I was raised to hew to the rugged individualist ideal.  Drugs are a crutch, you came out of the womb the way you should be, etc.  Eventually my wife convinced me to talk to my general practitioner about it, and she prescribed be something called clonazepam.  It’s a great little drug, and for the first time in forever, I felt normal.  I’d just sort of drift off to sleep, no muscle twitching or anything.  Alas, it’s habit forming, so it isn’t prescribed for long-term issues.  Eventually when I started going to the sleep doctor he had me try zolpidem (you might know it as Ambien), it’s probably the most popular option and is available in a generic.  It has a long half-life, though, and it made me feel groggy in the morning.  Eszopiclone (Lunesta) gave me a weird aftertaste.  Eventually we settled on zaleplon (aka Sonata) which has a much shorter half life and is lighter.  My doctor says he uses it if he wakes up during the night and can’t go back to sleep.  The only downside is for those days when your brain’s really going and you have a big presentation or early meeting, it’s pretty much useless.

Food

Food is a killer, and late night eating from sleep issues is a big reason why people gain weight.  It’s really comforting to have a filling snack after laying in bed for 3 hours, your blood goes to your stomach, you get the itis, but it’s terrible for you.  Late night eating is one of the things that concerned my doctor the most.  I’m not sure if I have much advice here, except that we don’t eat things we don’t have.  That leftover fried chicken isn’t lasting till morning, but if the chicken isn’t there, it doesn’t get eaten.  I don’t drive to the store at 3am, I just go back to bed.

Vacations

That Bed is Broken

That Bed is Broken.  Like, literally.

Travelling can be tough when you can’t sleep, but often the increased activity can make it easier to fall asleep.  Don’t forget your meds, like I did on my trip to PyCon.  I felt like a zombie all weekend.  If you don’t know about the hotel chain and have space, pack a pillow.  Hilton’s pretty good at it, as are most boutique hotels.  I always crank up the AC when I go to sleep, both for the noise and for the cold.  I’m lucky enough to be able to afford nice hotels.  When I was in Paris several years ago we booked the cheapest hotel we could find, and when we got to our room the bed was actually broken.  I think we’ve stayed in a few places in Mexico where the beds were almost literally an exercise mat on top of concrete.  Vacations are expensive, book a nice hotel if you can.

In Conclusion

If you have sleep issues, you’re not alone.  There are sleep doctors, and some good options.  There’s no magic bullet, though, so if your partner has sleep issues, give them a little slack and support them.  It’s a tough thing to live with, and it can make living a normal life really hard.

If you have some tips or tricks, please share them in the comments.  If there are a few, I’ll add them to this post, if there are a bunch I’ll do another one and link it from here.

Sweet dreams.

Post Script

This post originally had a different opening and title.  I was curious how people would react to it, as I’m stretching myself more as a writer.  I think the general reaction was ‘”That’s probably spam,” or “Good grief, I really don’t want to read that.”  Whoever would have gotten the joke probably has, so I tweaked the intro to be more on point.  In any case, here it is:

I’m a Little Princess

I’ve always known I was different.  I know regular people.  I see them all the time on TV and in movies.  They talk about their normal lives, their simple problems that seem so solvable, and I just can’t relate.  I wonder if they know how lucky they are, to be exactly who they need to be, to not be plagued by this.

I’m lucky too, I guess.  I have a job that lets me work from home and pass for normal.  My wife’s very understanding, probably more than a lot of women would be.  I know other people share my secret.  I read between the lines in their tweets, or see their living situations.  I’ve seen doctors, and taken medications, and it helps a little.

When I finally moved out of my parents house, I made some bad decisions.  I thought I could get a cheap fix, just some slapped together left over parts, but it was awful.  Finally I had a run of good paychecks, and I went to the store and plopped down more money than I’d paid for anything but a computer or a car to buy the ultimate solution… this: