Unauthoritative Pronouncements

Subscribe About

End of an Era - Follow-up

The other day, a friend emailed me to let me know that my post about Imageworks closing in Culver City was up on the VFX Soldier blog. I didn’t know it would find its way there, and I had not talked to Daniel about any of it. The last we spoke was when I ran in to him at a mall, when he was working for Digital Domain years ago. My blog is mostly for personal things, and isn’t a call-to-action, or anything. There was some argument in the comments over on VFX Soldier about pro-LA, anti-Canada, anti-Tax, pro… Etc. That wasn’t what the post was about, it was just about my feelings, and definitely not deep analysis. I winced at the thought of coworkers reading my piece, but I guess that’s just self-doubt about my writing. My blog is basically about nonsense, so please don’t stick around for any further emotional outbursts. Just for the really bad puns, and sarcasm.

💚

2014-06-07 12:45:00

Category: text


Form and Function

My favorite, cantankerous snowman wrote about design last night. In Design is How it Works, Dr. Drang argues that people misuse design to talk about how something looks.

Steve Jobs’s [Joe’s note: Take a shot.] “design is how it works” gets a lot of lip service, but when most Apple bloggers and pundits say design they still mean how it looks. Flat design, skeuomorphic design, “clean” design1 these generate millions of words of heated discussion, but they have little to do with how your computer operates2. You could go to the Iconfactory and change every icon on your machine, but that wouldn’t change how you or it work.

He managed to have two footnotes there.

  1. “Clean design = Helvetica + white space.”
  2. “I’m not arguing that how something looks has no effect on how we use it. There’s no question that things like layout, shape, and color can have a profound influence on user interaction. But the features have to be there to interact with.”

I don’t feel like those should be bunched up in a pop-over footnote. By shooing away how things look, he is downplaying an integral component of design. How something looks, and how something works, should not be disclaimers, they should work together. Also, his example of changing the icons is not a good one. Change every icon on your system to be exactly the same icon (pick your poison!) and then tell me it has no baring on how you interact with it.

There are a myriad of different kinds of design. Many shorthand the artsy-fartsy part as ‘design’, which should be corrected to include the engineering, but it shouldn’t exclude the people responsible for the look, because really, both parties are as responsible for the design.

When the doc decries ‘clean design’ he’s right to do so. In almost every case, it is used to describe vast tracts of emptiness padding the elements you read, and interact with, so that the overall look is ‘spacious’. It is more often, cold, spartan, and out of proportion. Google is especially bad at it in their web apps these days. That is bad design, and it’s the fault of someone with an art degree. After all, we all know engineers have beautiful, functional web sites that can be used on any mobile platform.

From Yesterday’s episode of The Prompt podcast, Stephen Hackett, Myke Hurley, and Federico Viticci discuss the necessity of some of the visual changes in iOS 7. Without them, the new mechanics of iOS 8 would be very strange.

Myke: We got iOS 7 to enable iOS 8, and that the design of iOS 7 was done in such a way that iOS 8 would make sense. Because if you think now, like, what they’ve done this year, is so much bigger than what they did last year. What they did last year was just change the way everything looked. In regards to functionality, there was some cool stuff, but it’s nothing like what they added this year. It feels like this is two years of functions. Like if you look, 4,000 new APIs. It seems like too much. Like it’s so much stuff. So I feel like iOS 7 was designed as a groundwork for iOS 8, which Apple probably should have made clear, I think.
Stephen: Well they’re not going to make it clear on the front end. They’re not going to say — Because what are you going to say? Last year, hey we redesigned it but we’ve got a lot of cool stuff coming next year? They’re never going to say that. In hindsight, it’s easy to connect the dots. And I have a couple points to make about OS X when we get there.
Myke: I’m sure that you do.
Stephen: But absolutely, [iOS] 7 was a stepping stone to this. And you see the way these things work, and parts of the interface make a lot of sense. Even down to, and it’s probably dumb, but, like, even trying to get developers go to an interface that’s white, that’s very flat, that makes pulling elements of different apps in to each other, easier, if apps kind of look like they’re in the same family.
Federico: I mean, can you imagine if, this stuff was done in iOS 6? You’d have, like, a metal calculator on top of a wooden, library bookshelf —
Myke: [giggling]
Federico (Continued): — I mean, that would have been crazy, right? Where as the new look of iOS 7, and iOS 8, allows apps to not only be more consistent with themselves, but also with each other.

That is, of course, assuming a lot about the intentions last year. It’s hard to ignore that thought exercise though. You’d double-tap to go to your multi-tasking card view, with a linen background, and linen would still slide down for notification center. ‘Well, you’re being too literal.’ I know, it’s my blog, go get your own blog.

That’s not really worthy of a footnote. People should still talk about the design of what’s changed this year when it comes to functional aspects, and not exclude that important work, but that’s because it is form and function that make up design.

Let’s say you need to design a chair. Sketch out a chair, I’ll wait. No matter what you draw, it will have components of form and function. It will have a supporting structure, part of it will be for resting your tuchus, perhaps it will have back support, lumbar support? You can go on. Whether or not it works is the functional aspect of the design. What it ends up looking like is it’s form. There is no way to completely abstract away form from your chair, it will have it. The aesthetic value of the form speaks to how well the form was designed. A well-designed chair will meet not only functional standards, but aesthetic ones. Shorting either will give you a poorly designed chair. Frank Lloyd Wright designed some beautiful-looking chairs. They are great to sit in.

Even in computer programming there is form and function. An engineer might think he has distilled his well designed program down to just code with a simple command line interface, but it’s still an interface. A text prompt is still a form. Don’t believe me? Go play with a bunch of command line applications and look how they choose to print information back in to the shell for you to read. That is an readability is an aesthetic concern. Even the code for the program has a form. Tabs or spaces? Line breaks between your blocks? Where should your imports go? Functionally, the program will work, form-wise, it’s gross and people will not like you as a person.

I work in a field that requires technical work, like setting up a ‘rig’ of ‘joints’ that need to rotate a certain way in order to function. They need to bind to a rig so they can actually be a character, and they need to render out, all-pretty-like, for people to see. Form and function throughout the pipeline.

Can’t we all just get along? We can all be designers, guys. No need to downplay what each of us contributes to design processes. Surely we can all come together and agree the real enemies to design are the managers.

2014-06-05 12:14:00

Category: text


One More Concern…

While I am excited about Swift, I am apprehensive about the frequent use of the word ‘seamless’ during yesterday’s WWDC presentation to describe many other things. Apple is notorious for being really bad at services — or I should say, “not as reliable” as their competitors. They have improved the reliability of these services, but even with the very latest iWork update, they continue to miss the mark. From David Sparks:

Yesterday I had a small writing project with a Mac-savvy client and I decided to do it collaboratively with him using iCloud Pages. I figured that if the application can support 100 collaborators, it should be pretty solid with just two. It still isn’t.

That’s the most recent release of iWork, just days before the WWDC announcement about more, seamless workflows. Color me skeptical.

Every CloudKit Has a Silver Lining

Perhaps what should be encouraging is that CloudKit, and other iCloud services, seem to be different from the infamously bad iCloud implementation of CoreData. Not very many details are out, but a few tweets from programmers attending CloudKit sessions seem to indicate they at least believe it is possible for the service to function appropriately. Perhaps this indicates that some future version of iWork will be updated to include some of these other functions, that it will continue its slow shuffle in the direction of improvement?

“I would only agree that a symbolic [cloud service] is as nourishing to the intellect as a photograph of oxygen to a drowning man.” — Dr. Manhattan

Photographs

Part of this overhaul includes rethinking the storage, and sharing, of photos on your devices. Mac OS X.X and iOS 8 will have some more persistent available storage for your photos, instead of the weird, busted, Photostream system that no one has been happy with. This is good, in theory, but they didn’t go in to a lot of detail about the tools to manage the storage, or to archive, old photos. I am hopeful that they learned their lesson from Photostream, but the fact that there is another new Photos app that won’t ship until “early next year” should be a red flag that this idea hasn’t really been fully tested. This was not an announcement they were making for a project that’s been under development for years, this was something that was far more recent. Let’s focus on the positives… I’m sure it will seamlessly move photos around just in the way that I both want and expect it to. I’m sure.

PhotoKit, however, would seem to solve a problem for some photo apps that want to ingest your photos in to their own storage facility to use their app. Things like VSCOcam. There really isn’t a reason to have a duplicate collection of photos that you need to keep importing from your Camera Roll and exporting to your Camera Roll.

Dropkick

Another strange development in the rat-king that is iCloud services, is that they’ve added some support to directly access other cloud storage solutions. Notably absent were the two biggest, most widely-used cloud storage solutions on the market: Dropbox and Google Drive. Will this mean that there’s going to be a bifurcated experience going forward? If I have data in one of those buckets, like Dropbox, then will I have separate storage widgets popping up using Extensions? That would seem to make for a less than ideal user experience! I’d hardly apply the word ‘seamless’ to that. While Extensions can solve many problems that users experience, they can also create new problems if every developer chooses to carve out “me-me-me” space for themselves. A year from now, when people go to save files, are they going to get a list of 10+ services they can save to? When they open a file, are they going to immediately remember what service it was saved on? When they go to save it, will they know to return it to the same place? Particularly if some services choose extensions, or built-in Dropbox browsers?

No One Will Ever Need More Than Five Gigabytes

They have three tiers.

  1. 5 GB for free.
  2. 20 GB for $.99 a month.
  3. 200 GB for $2.99 a month.
  4. Plans up to 1 TB for ? a month.

The top end is a good value, relative to Dropbox, but the low end is laughable. I am fine paying for it, but as Bradley Chambers notes on his blog:

While people reading this article understand that $.99 is cheap for 20 GB, a lot of regular people will just not pay any amount of money for it. They will just use up their 5 GB and then nothing else will be uploaded or backed up. When they drop their phone in the pool, they will still be upset that some of their photos aren’t backed up. Google offers 15 GB for Google Drive as the starter plan. We can argue about business models all day long, but the bottom line is that things like automatic backup with a ton of storage for free helps sell devices.

Preach, Brad. Like I said, I am fine with it, but I am unlikely to run in to normals that will put down money for this. We exchanged some tweets about this the other day, and I noted that if Apple makes it sufficiently frictionless to enter, and sufficiently frictionless to auto-renew, then perhaps people will participate. Here’s an article with data from a Rutgers University study on pricing.

Maybe They Got the Message?

iMessage is one of those services that has steadily improved over the years. It continues to have hiccups, and rough patches, and it’s never really clear when an iMessage will just bounce off one device, and hit another one you aren’t using. It’s ‘magical’ that way, I guess. Still, very reluctant to call this ‘seamless’. When I’m at my Mac, and the Messages app is closed, I will get a new iMessage that will show as a red badge. If I read, but do not reply to it on my phone, the red badge persists. That’s not accurate, let alone seamless.

While they announced many features yesterday for Messages app, some of them are using new techniques to piggyback on your iPhone to get audio, and text data on to your Mac. No mention is made of progress on the other weird bugs in iMessage during the WWDC keynote, but perhaps by consolidating it all to go through your iPhone it will be less buggy? I am given to understand that part of the problem with synchronizing the notifications was the way every device deals with encrypted connections to the server. If your phone is a single conduit then it may increase reliability and presence awareness because it is dealing with one channel instead of two? The last thing I want is for EVERY thing to ring like in the bad, old days when every single device you owned always got a notification.

Having said all that, I will give Apple the benefit of all of my doubts and download Yosemite when it is available.

2014-06-04 14:50:00

Category: text


Swift Excitement

Of all of the many announcements from Apple’s WWDC yesterday, the one that seemed to pique the interest of the largest number of people in attendance, online, and in my living room, was the premiere of Swift. Apple has been the subject of some ridicule for their Objective-C language for many years. A lot of blog-ink has been spilled, and a lot of audio has been recorded, with people arguing for and against adopting a new programming language. I happened to be in the camp that thought they needed develop one, but that was purely for selfish reasons. I’ve wanted to make applications — nothing profitable, marketable, or really all that interesting. Just to put together a little something-something for myself to make task X, or Y easier. One might say, “to scratch an itch.”

Objective-C melts my brain. It is a hideous monstrosity of a language, covered in warts. I know why it is that way. Many consider that a feature, and a testament to the original design. As someone that didn’t grow up using C-based languages, those reasons mean very little to me. I have none of that development baggage. All I want to do is get in, and get out. Have my tactical strike run of a few lines of something I can read, and be done with it. This isn’t just a barrier to adoption for me, but for many. The reason to learn Objective-C is to do something with Apple’s platforms that require it.

As I’ve said before, my ability to do anything at all stems from using graphics software that uses embedded Python interpreters to provide scripting, or expression, functionality. That’s why I know enough Python 2.7 to configure a Twisted settings.py file for multiple, virtual hosts, and to write a stupid static blog engine, and other proprietary pipeline scripts. I don’t know enough to do anything super-important, because what I consider to be super-important is programming a GUI, not CLI stuff. Tons more people know JavaScript because they’ve had to tinker with it for the web. These things are not completely dissimilar to one another.

That’s what’s so valuable about Swift. It integrates all the things I want to make with a language I can read and understand. I don’t have an iOS, or Mac Apple Developer account yet, but I probably will. It is too enticing to think about writing little pet apps for myself.

Just how easy is it to understand it given my limited background? Well if you understand:

print('Hello World')

Then you’ll understand:

println('Hello World')

Blocks aren’t exactly the way I’d like them ({} cruft), but they’re easy to interpolate between the two in your head:

mydict = {'apple':0, 'blackberry':0}
for key,val in mydict:
    if key == 'apple':
        val = 1

Compared:

var mydict = ["apple":0, "blackberry":0]
for (key,val) in mydict {
    if key == "apple" {
        val = 1
    }
}

If you already know one of these similar languages then this is approachable as fuck. If you don’t know one, then at least it’s equally cryptic! The only part I foresee being potentially confusing for me is the difference between var and let which I’m sure I will get wrong 50% of the time.

Next Generation of Programmers

I don’t recall seeing many Apple developers maintain a personal site where they publish any information about how something internal was developed, but Chris Lattner has done just that. Excerpt from his site.

I started work on the Swift Programming Language (wikipedia) in July of 2010. I implemented much of the basic language structure, with only a few people knowing of its existence. A few other (amazing) people started contributing in earnest late in 2011, and it became a major focus for the Apple Developer Tools group in July 2013.

The Swift language is the product of tireless effort from a team of language experts, documentation gurus, compiler optimization ninjas, and an incredibly important internal dogfooding group who provided feedback to help refine and battle-test ideas. Of course, it also greatly benefited from the experiences hard-won by many other languages in the field, drawing ideas from Objective-C, Rust, Haskell, Ruby, Python, C#, CLU, and far too many others to list.

The Xcode Playgrounds feature and REPL were a personal passion of mine, to make programming more interactive and approachable. The Xcode and LLDB teams have done a phenomenal job turning crazy ideas into something truly great. Playgrounds were heavily influenced by Bret Victor’s ideas, by Light Table and by many other interactive systems. I hope that by making programming more approachable and fun, we’ll appeal to the next generation of programmers and to help redefine how Computer Science is taught.

A beautiful sentiment. It made me feel bad for laughing when I read this.

Cons

There are some not-great things about Swift that are immediately obvious right out of the gate.

  1. It was not, and will not be, developed in the open so that people will have advanced knowledge of new features in the language, or be able to propose, or respond to, features in the language.
  2. It can’t be run on any other operating system. (You could write a web service in Swift, but it would need to run on Apple hardware.)
  3. The development IDE, Playground, looks great, but that will be the IDE.
  4. Security. Apple is the only one that will really be able to do anything with the guts of the language, and the compiler. Not that anything would happen.
  5. It is not one of many, similar, existing languages.

Those are ALL the things I expect of any Apple language. I don’t expect them to start a standards body where people can weigh in on this stuff. Pfft. What possible incentive is there for them to do that when the goal is not to increase adoption of the language on competing platforms, but to increase adoption of their platforms.

A few, long-time Objective-C developers seem to be averse to Swift. Todd Ditchendorf, former Apple developer, and developer of commercial software, like Fluid, was beside himself on Twitter yesterday. Todd loves Objective-C, and he sees Swift as inferior. I mentioned to him that I was excited about it, because it would help get more people to write software for Apple’s platform, to which he replied:

@joesteel I’m sorry, you must be looking for rational @iTod. You’re currently speaking with emotionally violated @iTod.

Todd’s been linking to complaints against Swift, so anyone interested in seeing this from another angle should definitely be following him. He linked to this piece by David Flanagan, a Mozilla programmer.

The fact that Apple treats a new programming language as a WWDC surprise, says a lot about Apple’s culture, I guess. If I wrote software targeting Apple hardware (or managed people who did) I don’t think I’d be happy at all about having a new language dropped on me from the blue—here’s the new language you’ll be using from now on… go learn it right away!

Naturally, I think David is overstating the urgency to adopt Swift swiftly. Its very design allows it to run alongside Objective-C code — of course it would. It is in Apple’s own interest not to invalidate the a language that is used for everything in both of their operating systems. The reason it was unveiled was to increase excitement about future development. To give guidance that there is a direction things will go eventually. Apple famously had Carbon and Cocoa frameworks for doing things in OS X for a really long time before they deprecated Carbon. There’s no way they expect an overnight switch, as David argues.

Back to the security concern: Unicode. There is a delightful part of Apple’s Swift book that shows unicode characters, and emoji, being used in variable assignment. This is neat, and great for developers of other languages. Python 3 also lets people do this and Armin Ronacher, a prominent developer, has pointed out that it is very possible to include backdoors in code if you substitute certain, lookalike unicode characters. I have not been able to test a similar situation under Swift, but I’d be curious to see how they avoid the same pitfall.

Who Else is Pumped?!

I am! Are you?

2014-06-03 14:50:00

Category: text


Contributing to Film Development

UPDATE: Rob Bredow, the Chief Technology Officer at Sony Pictures Imageworks for many, many years, is moving to ILM instead of staying with Sony Pictures Imageworks after the Vancouver move. I didn’t interact personally with Rob ever, but I would not be breaking any NDA to point out that he was crucial to the development of everything I mention below. Not just the development, but of the open source contributions that have benefited so many companies, large and small. More in David S. Cohen’s Variety article.

Original Post:

In my last post, on the loss of Imageworks as a local institution, I was pretty focused on my personal feelings. There are some practical things to consider though. Imageworks was staffed not only by artists, but also with brilliant software engineers. Every visual effects studio has a budget for software engineers. You just do. They set up a pipeline where resources can move between multiple software packages, and through the hands of many people. When I started in the industry, companies were starting to open up to open source, free software. Industrial Light and Magic (ILM) distributed OpenEXR so that all studios could handle passing around image files without resorting to ancient, limited image formats.

Imageworks, similarly, started to release tools as open source and they’ve been adopted across the industry. Not just with other studios bolting on the tools to existing software, but with third party software vendors integrating them. The effect is so large that the Sony engineers have won Academy Awards for their technical work.

  • Alembic - When animating geometry, every software package handles not just the geometry data differently, but the way in which the surfaces deform. Imagine a puppet rig inside a character, and you want to hand it off to another vendor. Their software isn’t the same, or its a different version, and that delicate puppet rig breaks. Alembic lets you bake out, on frames, sub-frames, everything before it is handed off. Then the other software loads only this baked geometry, no delicate rig that can break, or have precision issues.
  • Field3D - A common format for storing voxel data. Things like smoke clouds, explosions, etc. Like Alembic, this is baked per frame, or sub-frame.
  • Maya Reticle - Autodesk Maya pretty standard for animation and layout work these days. This improves looking through the camera in the software, so it’s a little more niche.
  • OpenColorIO - This is huge. Color is fucked up. Seriously. If you’ve ever worked with images, and thought “gee, that looks different on my computer in Photoshop” you might be able to empathize with the problem. Imagine a world where color management was totally unified, and actually implemented. No more ‘let’s just assume this is sRGB’ in software. Jeremy Selan, the chief developer, won an Academy Award for this in 2014.
  • Open Shading Language - OSL is an ambitious project to create an open standard for how to write shader code for any render package. Pixar recently announced that they would add support OSL in to Renderman.
  • Pystring - Python string handling in C++. I don’t do anything in C++ but I always thought this looked cool.
  • Scala Migrations - I don’t want to use this one, but I’m glad it exists.
  • Pyp - This one is fun. I’ve posted about it before. You can do all kinds of neat stuff on the command line. It is most useful if you are working with lots of directories, or need to transform file paths. It’s something you can run right there on your Mac at home.

That’s a lot of really useful open source, totally free software. Most of it is MIT licensed, so it can be put in commercial projects, like The Foundry’s Nuke node-based compositing application (the current industry standard for compositing).

Katana

One of the things that I love the most is Academy-Award-winning Katana. I am under a boring NDA so I can’t talk about how I’ve used Katana, but I can point to other people that have been authorized to talk about using it (video). Like Nuke, it is a node-based system. For those unfamiliar, it looks like a flowchart, but none of the nodes in the flow chart can loop (boring wikipedia page on DAG). This provides a powerful, modular way to filter through data. Katana is also based around the idea of deferred loading. Instead of loading everything in the file in to memory, it keeps track of where those things will go. This means you can do things like render all of Manhattan.

The reason why Katana is a big deal is because it was sold to The Foundry, makers of the aforementioned Nuke, and now studios can buy Katana licenses to use it in production. Indeed, Pixar used it for their short film. The Foundry has a user guide for Katana as a PDF, and a more detailed technical guide if you’re interested in learning about any particular details.

This stuff took years to make at Imageworks. You can’t just whip it up overnight. As you can see in part of that video, Tippett Studios did not have the resources to work on a system like this. Even though this is commercial software, and not open source, its wide spread use (much like Nuke, which originally started at the studio Digital Domain) is still a valuable contribution to the industry. All of Sony’s open source projects are also inside of Katana, Alembic, Field3D, and OpenColorIO.

With Imageworks shuttering in Culver City, where all this development took place, it’s another reason to be sad about these tax subsidies.

2014-05-31 12:13:00

Category: text


End of an Era

In May of 2005, I graduated from art school. I was terrified that I had not been recruited before the end of the semester, like many of my graduating classmates were. I had my demo reel (a series of clips demonstrating what you can do), and it was specialized around lighting and texturing surfaces. In those days, you needed to burn DVD’s to mail to recruiters, so I made sure I bought slim-line DVD cases. I printed color DVD labels, and made a DVD sleeve for the slim case. When you send things out, to be judged based on your schoolwork, it can really put you ill-at-ease about yourself. After all, it was a personal effort. It was very hard not to take the rejection letters personally. Even more painful than rejection: complete silence.

I had an interview with Electronic Arts in Orlando, Florida (Tiburon, they called it). They were primarily working on EA Sports titles. Every year there was a new Madden, etc. They started their first non-sports game, a Superman title, and a few of my friends were already working on it. I really wanted to get on that. Instead, in the very large conference room, with 12 people in it, I got questions about what my favorite sports team was, and since I was from Tampa, they thought I must surely like The Bucs. I do not like football, and hold The Bucs, and Tampa, in contempt for all the tax shenanigans they used to extort financial backing for their state-of-the-art stadium. Remember the tax theme, it’ll come up again.

It was very clear to everyone that I would not fit in well for what they wanted me for. I bought a ticket to fly to SIGGRAPH in LA, a convention that’s held for computer graphics, which also has a job fair. No dice. After another month of sulking (a total of four sulking months) I got a call from a recruiter for Sony Pictures Imageworks. They wanted to conduct a telephone interview with me, provided I was immediately available to start work in Los Angeles (I was in Tampa).

My stomach was inside out, I was full of nervous energy. Now that I was getting hired for something, and moving somewhere, I was going to have to be in the real world. There wouldn’t be time to hem and haw. There is a tightness in your chest with this kind of anxiety, and tingly numbness in your extremities. Uncertainty sparks it. I flew out to Los Angeles with nothing but a large suitcase. I stayed at my Aunt’s in Altadena, and drove the 1.5-2.5 hour drive to Culver City in complete stop-and-go traffic. I was even late for my first day of work. Back in 2005, you couldn’t check your phone for directions.

I worked. I worked, and I worked, and I worked some more. In those days, it was project after project. There wasn’t really a life outside of work. There are so many stories like that in VFX, and tech, jobs that it’s kind of banal and gratuitous to dwell on it much more than that.

The proudest moment I had was working on the Watchmen movie. A lot of people were disappointed in it, but I liked it. (Anyone who says they should have kept the squid monsters is plain wrong.) The work was incredibly challenging for me, and it made the reward of seeing it on the screen all the better. I got to blow up four people in two shots. There weren’t a lot of those exploding people shots, so getting the opportunity to work on something so memorable was beyond my wildest expectations.

I am under NDA for the rest of eternity, so I can’t say anything really crazy, but there are public things that happened in the news over the years. They opened a New Mexico facility to take advantage of tax rebates in 2007. They bought a facility in India to handle some types of technical work in 2007. They opened a Vancouver facility when the tax rebates were better in Vancouver. They closed the New Mexico facility in 2012. They closed the India facility in 2014. What happened in the industry-at-large was basically the same, satellite facilities moving to where financial offsets were favorable.

The first time I was impacted by the shifting landscape in VFX was when I was laid off after Hotel Transylvania. I had a lot of that same anxiety from before. I had convinced myself that there were still other opportunities out there. With a new demo reel (things were all online now, no more DVDs) I applied to places, and once again went to SIGGRAPH (only it was a much smaller job fair, and everything was for Vancouver work). There weren’t as many rejection letters this time, it was mostly silence. After three and a half months, I got a call from Rythym & Hues, another large VFX house in the LA area. They had relocated their facility from Marina Del Rey to El Segundo, CA for more favorable economic reasons. They just finished Life of Pi and they were staffing up for several projects. Right before my interview, I was contacted about coming back to Imageworks, right away. I let R&H know this and they hemmed and hawed on making a decision so I had to go with Imageworks. Only 2 months later R&H filed for bankruptcy. (They also won an Oscar.) The short-term hire turned in to another long gig, and I was there until the end of Cloudy With a Chance of Meatballs 2. There wasn’t anything for me to work on again, and it was getting pretty scary that there wasn’t anything to work on outside either. Digital Domain, another LA company was in trouble.

Where was all this work going? It was going abroad, for tax rebates on labor costs. Hollywood, like any industry, feels that labor is always too expensive, and can be obtained at a discount. Unfortunately, as DVD sales (remember my demo reel DVD?) declined they needed to reduce their risk but subsidizing the production of the movies they were making. You’ll recall seeing articles about why there aren’t as many middle-budget films these days. That was also part of the impact.

I was hired back to Imageworks again, only two months later this time, to work on Amazing Spider-Man 2 for a few months.

Before the end of Spider-Man, a rally was planned by VFX artists outside of the Academy Awards to draw attention to the labor situation, and what it was doing to families. It got a good deal of press, but I could tell that it wasn’t going to last. The event wasn’t really close to the Awards. No one was really going to cover very much of it. No stars would stop on the red carpet and stare at the people wearing greenscreen-green when they weren’t even on the same street block. Besides, I had to work. I went to work. The rally would have been cathartic. The news articles circulated until the next morning.

I wrapped up, again, in March. By this point the anxiety was less. I had survived this twice, remember? It would be a good “vacation” in a horrible sort of way. This time, however, I suspected Imageworks would not be calling me back to start right away. Something about it felt very permanent.

Today, it was made permanent. In Variety, The LA Times, the CBC… In a cruel twist of fate, the banner image they selected of Spider-Man with the manhole cover is something I did stereo work for (not the 2D, just the stereo). I have no credit on the film, but these things happen. It doesn’t really matter where the story ran, no one reads that stuff except the dwindling coterie of VFX artists. It’s not like Motorola closing a plant in Texas for making phones no one would buy. That’s news. It’s related to electronics, and tangentially to Apple, so it’s worth dwelling on. How entertainment is made is so dry and boring it quickly gets flushed off the home page of every site that ever posts it. Stars and directors are almost the only part that people track. People like to fixate on Kubrik, and other auteurs. That’s not to sound unappreciative, or spiteful, it’s just a fact.

People that have either already accepted positions in Vancouver, or are considering them. People that have moved to the bay area to try and make it in VFX there. People that went to commercial houses in New York to get out of film. Even people looking for work in Vancouver, where SIGGRAPH is announcing an unemployment discount for the first time ever.

I did my first stint at a small company in LA, and it was an interesting experience, for a month. I will look for other, tiny places, and tiny places after that. When I work on something, I go all-in, but if the work isn’t here to do, I don’t feel the same way about moving to chase it. International migrant work is not appealing to someone that wants certainty, that wants stability.

I hold on to those moments, like Watchmen as the reason why I have pride in what I do. I like to hold on to those moments where I worked crazy overtime for pride in my ability to do a task set before me. However, without more of those moments I fear I’ll lose momentum.

Truth-be-told, that’s why I’ve been writing so much recently. It’s why I tinker with bad code. I need something other than my career to think about or I’ll go nuts. I’ve even toyed with the thought of writing, or with web dev crap. Not because I think I have the same aptitude, nor because I think it would be profitable, but just because it would be something else. I’m even posting really awful jokes on Twitter more often because it distracts me from peering in to the abyss.

We now return to our regularly scheduled snark fest about things that don’t matter.

2014-05-30 15:07:00

Category: text


Mac App Store Pro-vides Poor Pro‑Prospects

When was the last time you bought anything in the Mac App Store? Was it longer than 6 months ago? It has been for me. In your recent recollection you may be confused, thinking you had paid for something, when it was a free download, or an upgrade. Did you used to download from the Mac App Store more often? I did. When it came out I thought it was the bees knees. I thought that finally I could do away with entering serial numbers, and holding on to digital receipts, and software updates that you had to go get manually — five miles uphill, in the snow, both ways. What a relief it was to have it. Indeed, initially developers seemed to be relieved as well. It solved many things for them. The Mac App Store was our last, best hope for peace.

Then Apple just decided to bulldoze over fucking all of it. It served their purposes to enforce a sandboxing requirement. Then, as time went on, it became increasingly clear that wasn’t going to work so they just kept extending the deadline. And then they extended it again. And again. Then they said, “this far, no further!” and everyone bailed on their fucking shit. I mean, seriously, how many apps have bailed to go back to the shitty way things were before? How many stories have there been? SPOILER ALERT: A lot.

I open the Mac App Store today because there’s a red badge-thingy on it. “Update this number of things!” It cries out, in its circle of crimson blood. “Sacrifice your time to me!” It commands. The majority of things I’m even downloading updates for are Apple’s OS X fixes. Of the last 13 (ominous number choice) updates it displays, just five aren’t OS X updates. Shouldn’t I be using this store so often that I barely see Apple listed anywhere at all? Shouldn’t this be a profitable system for developers to get people in to buy things in volume since they can no longer get me to buy expensive things? Wasn’t that the whole point of this system?

Indeed, even Apple has applications in the store that predate the store. They are still selling software that they made over 4 years ago. Aperture — the dead horse I like to beat every now and again — was released February 9, 2010. I gladly bought it on DVD, which was SHIPPED to me. For several of Apple’s apps, Aperture included, Apple just authorized it to appear in the Mac App Store, eventually, for updates. That was the grand solution they had for their pro developers. Just suck people in to a nightmare world of incrementally bug-fixed software for the low-low price of “we’ll never do anything good with this again because this doesn’t even make business sense for us and we made it.”

Currently, on the “Featured” pane of the store, there is a game that was released last calendar year for iPhone, a RSS reader that is the editor’s choice, an application that lets you bookmark things on the internet, a game that looks like it would never make it on the Xbox or PlayStation stores, and an application for planning what you need to do in your day. I do not want to belittle those applications any further than that. They are fine at what they do, and they are very useful to people (with the exception of the one game, that’s a head scratcher). Where are the big apps? Surely, OmniFocus will let you do things, and Reeder will let you catch up on your subscription services, but where is the BIG stuff? And of those things that have some use to you, how much time would you say you use them for? Hmm? I certainly don’t have much practical need for anything they are featuring, but that might just be me. Even contrasting the apps on the Mac with the apps for the iPhone or iPad, there’s no comparison that there is an overall dip in desirability between these apps.

In the Best New Apps, the best app is Pixelmator, which isn’t really new, it’s an updated version of Pixelmator. I love Pixelmator, and I love the updated Pixelmator. There are things for timers, home inventory, notes, little things you need a small solution for. All the big software you need, all the powerful stuff, isn’t here. Nowhere to be found if you search hither and yon.

The Mac App Store offers very little promise to developers, which in turn, offers very little promise to me. I’m still using web downloads, serial numbers, and — for fuck’s sake — Creative Cloud. I hate that. Words could not possibly do justice to the caustic bile that wells up from within my blackened soul when I think of Adobe’s Creative Cloud and it’s update for it’s update process. It spews forth excrement in to the world, and I gladly sup it, because I need it.

I hope that after Apple’s WWDC event next week they release an “App Store”.

2014-05-29 23:58:00

Category: text


Using iOS for Eeeeeeevil

I felt like being annoying (what’s new), and I decided to take advantage of an ASCII twitter meme from last… uh, summer? Matt Alexander had harnessed this meme for the patented “Super Fave” already, but I needed to make it even more ridiculous. Easy to do with a simple text substitution. Just replace all the same characters with a different character. Simple find and replace stuff. You can do this on your iPhone or iPad so easily. If you don’t have Pythonista, you should, because: PYTHON WORKFLOWS. It seems so intimidating to “program” something if you don’t do it often, but the function is literally called replace and you give it the text to replace, and the text to replace it with. 'this string'.replace([original],[new]) That’s not fucking Scheme, it’s not Objective-C, there are no End Tells. That means you can bug the shit out of people without breaking a sweat!

Start with whatever ASCII meme you want to butcher, like a “Super Fave” if it’s on multiple lines, use three quotation marks in front of, and behind it. This lets Python know it should respect all the lines of text.

superf = u"""
☆。 ★。 ☆  ★
 。☆ 。☆。☆ 
★。\|/。★   
  SUPERFAVE
★。/|\。★ 
 。 ☆。☆。☆ 
☆。 ★。 ☆  ★
"""

Easy, right? The “u” at the start of the quote lets it know you’re using fancy, unicode characters. No big deal really, you just put a “u”.

Now that you’ve assigned the text to a variable, it’s super easy to manipulate it. Assign some other variables for the text you want to replace. like

a = u'💩'
b = u'🌹'

Boring, I know. Now let’s do the replacement:

r = superf.replace(u'★',a).replace(u'☆',b)
print(r)

Notice that I assigned it to a variable again so I could reuse this. I’m also chaining the two replace functions to each other. You could put every one of these things together without defining any variables, but it’ll just look like a train-wreck, so skip it.

If you want to make this an interactive application, there’s a feature in Python called raw_input() where you can let the user enter text at run time. Inside the parentheses, you can specify text to put at the prompt line to give people some kind of instruction. Like so:

a = raw_input(u'★  = ')
b = raw_input(u'☆  = ')

There are also some fluffy things I added to the script that are Pythonista niceties, and not things that work on the desktop. Pythonista helpfully includes a console module to control the way text is printed out when the script runs. I want to set it to look like Helvetica Neue so it’ll match closely to watch iOS Twitter clients will display. Naturally, these are not monospaced fonts, so you’re going to get variable-widths to the characters that will deform the overall shape of the text, to some degree. The clipboard module just helps to copy the text output right to the clipboard when it runs. No need to do it yourself. We live in The Future. You can run without any of those extra things just fine on your desktop.

I initially sent this to two, very responsible people on Twitter. I admonished them not to use it for evil. Then I used it for evil, like, uh, 20 times since then.

Go forth, my minions, and annoy the shit out of everyone!

2014-05-28 22:34:00

Category: text


Really Senescent Syndication

The other morning, I was neurotically reading all the tweets I missed while I slept, and I came across a mini tweet-storm from Federico Vittici of MacStories.net and The Prompt fame. He was expressing frustration with elements of his posts not being appropriately captured, in the RSS feed, and presented in feed readers. This, of course, isn’t really the fault of the feed readers, they are conforming to a specification. It’s the fault of the positively antediluvian specifications for RSS 2.0 and Atom 1.0. Yes, that is 2003, and 2005, respectively. This is in no way, shape, or form keeping current with what “The Web” is these days, what is in a web page, these days. I joked about in the previous post, the W3C Atom validation widget said that <figcaption> was not a valid HTML tag. Ahem.

The modern web uses <script> tags for everything. There is JavaScript all around you (like roaches) and it does all kinds of neat tricks on the pages you visit. Sure, there’s the scary, ad-tracking, privacy-invading kind, but there’s also stuff like pretty animations, and swell graphs. Federico wanted to return to truncating his feed for this very reason. He went through so much trouble putting together his site, and the features, that it was an anathema to him to have it neutered by the requirements of feeds.

Days before that, Casey Liss (Who the hell is Casey?) complained on Twitter about the bits and pieces of tags that feeders wouldn’t render either, of course it was also the <script> tag.

Why are things this way? How can this specification be so criminally neglected, and misunderstood? I posit that it’s precisely because of “Web 2.0”, and more recent, events. Small startups with publish/subscribe features appeared to push and display content for us. Things like Blogger, and Google Reader fell out of fashion for things like Facebook and Tumblr. After all, RSS and Atom are files, not protocols for synchronizing data. They have no social features whatsoever. You can’t click a button in your RSS reader of choice and the feed owner gets a ‘like’ back. That’s the kind of vanity people expect these days. It’s better to be in a walled garden than to be a wallflower. While Twitter and Tumblr initially supported exporting flattened data to RSS feeds, the use of the features declined because users could not interact with the services. Then services started to phase out RSS (Tumblr still has some, limited RSS support).

The whole reason I moved off of Tumblr was because they were truncating posts in my feed. Not just a little truncated, mind you, but there usually wasn’t even a full sentence in the entry in the feed. When I contacted their support, they plainly told me they had no interest in doing anything about it. I can see why they wouldn’t, advertising revenue comes from the dashboard, so everyone should read Tumblr blogs on the dashboard. Feedpress even ran in to RSS support problems when Tumblr quietly nixed the ability to redirect to another feed. This particular frog was starting to feel the water boil.

Dining on Ashes

Much like an old house, there are layers that have been patched, ripped off, gutted, plastered, drywalled, and rewired. RSS really started as the project of Dave Winer (a person that has managed to stumble backwards in to nearly every controversy you can imagine) when he was at the company UserLand. Netscape co-opted his work with some custom headers and called it RSS. Dave and Netscape went back and forth about what to do with it for a while. Eventually, RSS 2.0 was released. Then, that was it.

Atom was a project started to fix quirks of RSS that had arisen because of its percolating development. It was going to be the new, hot thing. No restrictions to the past, a clean break and a path to the future!

Only problem is that no one has touched either in aeons. Indeed, their web sites are the very model of inactive development. A part of the website for RSS, hosted by Harvard Law, boasts how it was converted to static files so it was easier to keep serving the files forever. I mean, that’s nice and all, but shouldn’t that be a red flag that nothing is happening?

Alpha Google and Omega Google

Way back at Google’s inception, they used to let engineers run amok and make novel things. One of those things turned out to be Google Reader. It used the Atom format internally, converting incoming RSS to Atom, and using private APIs to manage things. People tapped in to those over time. When Google killed it, there was a great cambrian explosion where people took their feeds off to other services. Many bloggers hailed this is as renaissance for feeds. How exciting to think of managing feeds! (I guess?)

I largely ignored all of this because I didn’t like stripped-down formatting in feeds, and unread badges. I was more than happy to just go to each bookmark, look for new stuff, and read it on the site. Later, mixing links from Twitter in to the process. Indeed, to this day, I read most of the material I see online from links that other people post on Twitter.

Where the hell is Donatello?

Where’s the fucking renaissance? I see none of that early exuberance yielding any advancement of any format for streams of syndicated content. Whatever classification you want to assign to it, RSS, Atom — anything else — nothing is happening. What I do see, and hear on podcasts, are people complaining about improperly formatted XML that doesn’t conform to 2003 and 2005 specifications. Indeed today I listened to Myke Hurley’s excellent CMD+Space with Padraig Kennedy and Oisin Prendiville of Supertop. Padraig and Oisin lamented that the best influence on podcast feeds was iTunes guaranteeing some lowest common denominator of conformity. That is hardly a ringing endorsement of Apple’s influence on the market, but it is a realistic assessment that without Google, it’s basically just Apple’s managed feeds of audio holding on a line in the sand on bare minimum.

And yet, there are still services built to synchronize these outmoded, meticulously-manicured menageries in perpetuity. There is no life left in these formats. They are operating on support systems pumping social-air in to them, carrying read-waste away.

These standards are no more. They have ceased to be.

United Federation of Something–Something

From the above description, the angry, pitchfork-equipped villager might incorrectly infer that the onus is on a singular party to step forward to fix all this mess. That some stranger will wander in to town, the ‘sheriff’ badge shall be affixed to him, or her, and they will right the wrongs. That won’t solve anything, then you just have another Twitter, or another Facebook. Even when the goals are lofty, like they were for Diaspora (Oh look at me, remembering things) or for App Dot Net. There needs to be a body of people interested in moving things forward for many players in the market. Singular sources of propulsion are simply not going to get anyone anywhere. We need federated protocols and services (it’s just as exciting as it sounds!) where no singular entity can absorb the entirety of the market (Google Reader, iTunes), but not so Linuxy that they can’t get anything accomplished. That is easier said than done, however, because people are horrible. Look at you! You’re already thinking this won’t work so don’t judge me for also thinking this won’t work.

A Serious Sketch

Moving on with A Serious Sketch, or an ASS, let’s break out a few requirements that we may feel are obvious these days but were not obvious when these specifications were engineered.

  1. Data for a reader is synchronized across all of a reader’s devices.
  2. Updates from writers/podcasters/artists can be pushed incrementally, rather than in one, monolithic feed refresh. People could still host static feeds that get polled for changes, but the option to push changes would benefit many. Think about push and non-push email on your phone. Both can co-exist, one is for serious business.
  3. Anonymous reader statistics can be part of a platform because the reality is that it is important to know, at the very minimum, reader counts for ad-supported content. (And vanity.)
    1. Update: Dan Benjamin had a tweet about RSS after this originally posted, stating that unique subscriber counts were important. I was speaking broadly above, but there’d need to be some very specific metrics provided in order for any system to not only be useful, but also uniform.
  4. Social features, like assigning some gratitude-based star, or heart, widget to incrementally praise someone and to show other potential readers that they might also find the content interesting based solely on the amount of aggregated interest in it by other readers. (And vanity.)
  5. Real support for every HTML tag, CSS selector, and every feature. Sure, there will always be room for people to use mobilizers, to make certain sites legible, but we should not start from the assumption that no one should be given the chance to show a full, “web” experience.
  6. No more handwritten XML. Make serious, fucking tools for generating feeds. No ambiguity about what a field is for, or about which optional tags can be used. Atom is a decent start in this regard but there is no “make an Atom feed” kit.
  7. Break the “feed” up in to multiple files by default. No more singular file feeds that will hit a 150 kb wall of sadness. The web exists as standalone XML (HTML == XML) there is no reason XML can’t be served, as needed, on demand.
  8. Do not rely on timestamps to update the content! Especially if content will be rich, and not the plainest of plain text. We have so many diverse mechanisms to tell if a file is different from another file, or for a file to report it is different by sharing very minimal amounts of data. There is no reason to rely on someone updating a totally optional field, or expecting a reader’s client to go above and beyond the expectation of the specification.
  9. Real, functional, individualized password protection. There are some loopy, buggy ways to password protect feeds right now. This has certainly played a part in shaping content on the internet to be primarily ad-driven instead of pay-wall. If the feed is in files, then it’s even easier to allow incremental access, or to allow other kinds of access. Think about Dropbox shared URLs that get generated, and revoked, at a user’s whim. No one can really host a feed through Dropbox though. Even if you could you wouldn’t have any authentication portal that RSS readers, or podcast clients, would recognize. You’d need to have some way to say, “Hey, here’s some fucking password protected shit.”

See that stuff? It’s not crazy stuff! The word “computer” in my degree is followed by “animation”, not “science” or “engineering”. Surely someone with actual brains could do reason things through better than I. Think of all the services that we use today that do that stuff! Think about the fact that when you read a RSS, or Atom, feed you’re basically doing one of the most archaic things you can do with digital devices. Even email support for rich content is 500,000x ahead of what your feed reader of choice can do, and that’s email. You might as well subscribe to newsletters! You’d see feature development occur at a faster pace. (Anything’s faster than 0 though.)

2014-05-28 00:52:00

Category: text


If Wishes Were Yaks

Last night, I had a big post about Yak Barber, and how it was to a point that was functional-enough to post about it. However, after I had posted it, I kept fiddling with it anyway. My milestone of ‘well the Atom feed is broken’ wasn’t good enough for myself. I ended up staying up several more hours trying to do the thing I said I would wait to do. Even after that, I couldn’t fall asleep immediately because I was thinking of all the spaghetti code I want to yank out (yak out?).

Self Critique

Importing the settings from the settings.py file relies on the imp module. This is so I can have a command line argument to specify a different location. If I just hardcoded a path, then it would work, but you’d always use the same path. Python will gladly import the settings.py file. It’ll also gladly complain that it’s not really how that’s supposed to work. So kudos for whining.

The settings that do get read in are then piped to local variables. This is is in the event that I need to override something locally with future command line args. It will also throw a warning if these required variables are missing.

root = settings.root
webRoot = settings.webRoot
contentDir = settings.contentDir
templateDir = settings.templateDir
outputDir = settings.outputDir
sitename = settings.sitename
author = settings.author
md = settings.md
postsPerPage = settings.postsPerPage
typekitId = settings.typekitId

Then, because of the way python is structured, the main function is called. This triggers some steps to make sure directories are there, and if not, create them. It in turn triggers start() which handles the broad strokes of processing the pages and moving files by delegating out to other functions. Nothing bad here, really.

Then, unfortunately, you get to the part that renders the pages:

def renderPost(post, posts):
  metadata = {}
  for k, v in post[0].iteritems():
    metadata[k] = v[0]
  metadata[u'content'] = post[1]
  metadata[u'sitename'] = sitename
  metadata[u'webRoot'] = webRoot
  metadata[u'author'] = author
  metadata[u'typekitId'] = typekitId
  postName = removePunctuation(metadata[u'title'])
  postName = metadata[u'date'].split(' ')[0] + '-' + postName.replace(u' ',u'-')
  postName = u'-'.join(postName.split('-'))
  postFileName = outputDir + postName + '.html'
  metadata[u'postURL'] = webRoot + postName + '.html'
  metadata[u'title'] = unicode(smartypants.smartypants(metadata[u'title']))
  with open(templateDir + u'/post-content.html','r','utf-8') as f:
    postContentTemplate = f.read()
    postContent = pystache.render(postContentTemplate,metadata,decode_errors='ignore')
    metadata['post-content'] = postContent
  with open(templateDir + u'/post-page.html','r','utf-8') as f:
    postPageTemplate = f.read()
    postPageResult = pystache.render(postPageTemplate,metadata,decode_errors='ignore')
  with open(postFileName,'w','utf-8') as f:
    f.write(postPageResult)
  return posts.append(metadata)

The markdown module generates a Python dictionary from the MMD-style metadata. That’s just the crap at the top of the file with colons. First empty line, or line without colons, seperates the metadata from your text in your document. I’m overriding several values in the dictionary with the variables we’ve pulled from settings.py.

If you think that’s not so bad, just wait until you see the the part that makes index pages and the atom feed! It’s like opening a wall in your house and finding black mold. Oh boy!

def feed(posts):
  feedDict = posts[0]
  entryList = str()
  feedDict['gen-time'] = datetime.datetime.utcnow().isoformat('T') + 'Z'
  with open(templateDir + u'/atom.xml','r','utf-8') as f:
    atomTemplate = f.read()
  with open(templateDir + u'/atom-entry.xml','r','utf-8') as f:
    atomEntryTemplate = f.read()
  for e,p in enumerate(posts):
    p[u'date'] = RFC3339Convert(p[u'date'])
    p[u'content'] = extractTags(p[u'content'],'script')
    p[u'content'] = extractTags(p[u'content'],'object')
    p[u'content'] = extractTags(p[u'content'],'iframe')
    if e < 100:
      atomEntryResult = pystache.render(atomEntryTemplate,p)
      entryList += atomEntryResult
  feedDict['atom-entry'] = entryList
  feedResult = pystache.render(atomTemplate,feedDict,string_encode='utf-8')
  with open(outputDir + 'feed','w','utf-8') as f:
    f.write(feedResult)

def paginatedIndex(posts):
  indexList = sorted(posts,key=lambda k: k[u'date'])[::-1]
  feed(indexList)
  postList = []
  for i in indexList:
    postList.append(i['post-content'])
  indexOfPosts = splitEvery(postsPerPage,indexList)
  with open(templateDir + u'/index.html','r','utf-8') as f:
    indexTemplate = f.read()
  indexDict = {}
  indexDict[u'sitename'] = sitename
  indexDict[u'typekitId'] = typekitId
  for e,p in enumerate(indexOfPosts):
    indexDict['post-content'] = p
    print e
    for x in p:
      print x['title']
    if e == 0:
      fileName = u'index.html'
      if len(indexList) > postsPerPage:
        indexDict[u'previous'] = webRoot + u'index2.html'
    else:
      fileName = u'index' + str(e+1) + u'.html'
      if e == 1:
        indexDict[u'next'] = webRoot + u'index.html'
        indexDict[u'previous']  = webRoot + u'index' + str(e+2) + u'.html'
      else:
        indexDict[u'previous'] = webRoot + u'index' + str(e+2) + u'.html'
        if e < len(indexList):
          indexDict[u'next'] = webRoot + u'index' + str(e-1) + u'.html'
    indexPageResult = pystache.render(indexTemplate,indexDict)
    with open(outputDir + fileName,'w','utf-8') as f:
      f.write(indexPageResult)

You’ll notice that there’s a lot of repeated keys and values from the renderPosts() function. In fact, it even starts with the output the previous function generated. The way that the Python version of Mustache was handling lists in the dictionary resulted in an Atom XML file with Python list syntax being wedged between all the XML tags. That wasn’t cool. It needed to be looped over separately in just the right way to make a dictionary pystache wouldn’t barf all over. Because it needs to have things handled in a special way, and because the index pagination needs to happen in a special way, I have to jump through these hoops to make these separate, looped dictionaries just to make the “simple” template engine happy. That is a lot of code duplication in there setting dictionary keys.

You’ll also notice the feed has date string shenanigans in order to meet the UTC (Greenwich Mean Time) requirement of Atom, the tantalizingly named RFC-3339 specification. Turns out, Python doesn’t really have a singular module that does this. So I get to use time, datetime, and pytz to parse the date and time, convert it to 2 different formats, stick an origin timezone on it, and feed it to a third party library that actually understands what timezones are. To say that this is a deficiency of Python would be an understatement.

def RFC3339Convert(timeString):
  strip = time.strptime(timeString, '%Y-%m-%d %H:%M:%S')
  dt = datetime.datetime.fromtimestamp(time.mktime(strip))
  pacific = pytz.timezone('US/Pacific')
  ndt = dt.replace(tzinfo=pacific)
  utc = pytz.utc
  return ndt.astimezone(utc).isoformat().split('+')[0] + 'Z'

But don’t worry, that’s not the only thing the W3C Feed Validation Widget wanted to yell at me for! It is also considered “bad” to have <script>,<object>, and <iframe> tags in your XML, even if they’re inside of the part that’s just HTML. This means things like YouTube embeds and Twitter’s embedded tweets need to be sanitized. It is certainly, without a doubt, not even worthwhile to use embedded tweets in the future.

After spelunking through StackOverflow all night for time problems, I got to go back and look for the best way to remove tags. It is not considered wise to use regex on XML/HTML to remove tags. Fine, eggheads, what do you recommend? Enter BeautifulSoup.

def extractTags(html,tag):
  soup = BeautifulSoup.BeautifulSoup(html)
  to_extract = soup.findAll(tag)
  for item in to_extract:
    item.extract()
  return unicode(soup)

This explains all those entries in feed where the content key kept getting scribbled over. That isn’t the right way to do that. I should feed a list of tags to remove to the extractTags() function and I should write to a new key on the dictionary. If I did that, then the information could co-exist with the original content value and I could make it part of renderPosts().

Basically, there should be one, singular dictionary, or Class object, that carries all the information a post. There should be one, singular dictionary, or Class object, that carries all of the posts together. Then all the functions should yank data from that one tree of data instead of generating what is, essentially, identical stuff.

In terms of benchmarks:


1939885 function calls (1925524 primitive calls) in 2.171 seconds
1939758 function calls (1925400 primitive calls) in 3.014 seconds
1939758 function calls (1925400 primitive calls) in 2.663 seconds

Processing 122 discrete text files in 2-3 seconds doesn’t strike me as especially bad. The biggest offender is — surprise — Python’s regex substitution. A teeny-tiny part of the code that removes characters from the titles to generate filenames and URL’s. Oh regex! (arms akimbo)

I should have just gone to bed.

2014-05-21 15:00:00

Category: text