Adam Lisagor, Andy Roth, and Dan Sturm at Sandwich were gracious enough to answer my questions about the behind-the-scenes stuff on their stereoscopic livestream to their app. It’s a novel enough endeavor that I really wanted to dig into details about it, and not shy away from nerdy things that don’t get covered if a person doesn’t know what questions to ask. I hope you enjoy it too.
Category: text
This is another WWDC 2024 video that caught my eye. A few months ago I wrote about using some of Apple and Google’s features when I was traveling abroad in Japan.
One of those issues was that reviews for locations in Apple Maps were in Japanese, which is great if you’re a local in Japan, or a traveler who can speak Japanese. Data sources appeared to mostly be Tablelog. Each review needed to be tapped into to select the text to do a translation of that individual review, or you could go to the Tablelog website and use the “ᴀA” menu to select the translate page option and translate all the reviews on the page at once. You’d then have to go back up the navigation hierarchy to look at another location and repeat the steps.
So I was surprised to see that one of the demos in the WWDC 2024 translation video was for translating Japanese reviews in a the sample hiking app to English inline with the source text without having to translate each review one at a time.
I wondered if that would be coming to Maps in iOS 18 (I’m not installing the developer beta, but you do you), but when I went back to the same locations that had Japanese reviews from Tablelog, they were now being pulled from TripAdvisor or Yelp.
Huh.
I doubt that was in direct response to my kvetching, but it is certainly behaving differently than it was at the time I wrote that up. Now I wonder if Apple Maps will eventually get a feature that could translate those Tablelog reviews inline, or if the Apple Maps team considers the matter closed because they have the TripAdvisor and Yelp reviews?
Category: text
I’m still working through WWDC videos. A lot of interesting stuff is in them, even if you aren’t a developer actively plying your craft you can see Apple’s design and engineering ideas.
The CarPlay videos are kind of wild for that reason. I’m not picking on Ben Crick, or Tarnya Kanacheva who present in both the videos. They present as well as any Apple WWDC video host.
Next-Gen CarPlay was announced WWDC 2022, then Porsche and Aston Martin had teaser gauge clusters at the end of 2023 that isn’t shipping in anything, even their brand-new cars that have been released this year.
The two 2024 videos are basically sales pitches and explainers for the vague 2022 announcement. A lot of extra work has happened in two years, but … will anything ever ship with what they keep teasing?
The first video, Say hello to the next generation of CarPlay design system, is a sales pitch about how a car company’s brand will only be mildly diluted into a cobranded experience. From Ben Crick, “It empowers you in partnership with our team here, to develop a beautiful cobranded experience that celebrates both brands.”
This is … weird. Ironically car makers are teased with a level of customization that has never appeared on an Apple product in this century, but it’s when working in conjunction with Apple designers, and you apparently have to use the San Francisco family of typefaces? Wild proposition.
Android Automotive uses Roboto, but Roboto doesn’t leave the car when a phone does because it is the car. There isn’t a separate interface system, with typefaces, that needs to exist in the absence of a phone, or the use of a different phone.
This is an odd negotiation in public, since this is a business-to-business video dropped in with all the other publicly accessible videos. A declaration that they’re willing to give up a little control, but definitely not all of it, when in fact they have no control as leverage at all.
I’m not outraged, or disappointed, I’m just confused. Allaying fears that an automaker’s dashboard will look bland and generic aren’t a real concern of automakers who simply won’t adopt the system.
The automakers already have to design a system for when an iPhone isn’t connected. They have to incorporate animations, design gauges, select gradients, use 3D and 2D assets, etc. If they do all that work, and then contact Apple to apply to create a cobranded experience the best they can do is get something that’s close to the work they’re already doing.
Our Audi uses a few typefaces, one that I like on the speedometer and the heads-up display is either Microgamma, or something that’s damn similar. I like it just fine, and I’m sure Audi likes it too. However, the best Audi can do is to work with a designer at Apple to alter the variable properties of San Francisco to get it close enough. How does that help Audi, and why would Audi’s customers want the fonts and gauges to change depending on whether or not their phone is connected?
The next video on the architecture is also mystifying. It starts as a pitch to automakers that it’s not THAT much more complicated than CarPlay, and so customizable.
Then you get into how it works, which is also surprisingly high-level in the video compared to other Apple WWDC videos. There are four composited layers, where you have the top level of the stack being an overlay of vehicle telltales and warning lights. Then you have locally rendered elements from an asset package, loaded by and updated by iPhones when they connect to the vehicle. It renders real time stuff like speedometers, because it needs to be able to do that without lag or connectivity issues. Then there’s the remote UI which is all the stuff rendered by the phone for maps, and media. The optional fourth element is a punch-through to views rendered natively by the vehicle, for things like cameras, and advanced driver assistance systems (ADAS).
The compositing pipelines are also separate for each display in the car. There are time codes that are supposed to keep things in sync for buffered output.
This multi-layered approach where things are being rendered and dispatched by different systems perplexes me. Also, not for nothing Apple, but people are pretty familiar with compositing and UI errors on Apple system software. I don’t know how much faith I have in a real-time, multi-layered, multi-source compositing solution.
Not that this is the same problem set, but this was my lockscreen when I AirPlayed the video about how easy it is to composite the custom UI layers in real-time:
This is a lot of work to do to reskin an interface. Apple doesn’t supply a theme for the vehicle, and it doesn’t take over everything, it does something between the two extremes.
That means that the mix of elements requires a mix of logic and system status updates. The speedometer is already implemented in the vehicle, but now automakers can reimplement a similar speedometer. The climate controls are implemented in the vehicle, but now they can relay status updates and display UI elements that look like Control Center to reimplement all their climate control logic as close as they can get it.
Setting aside the highly polarizing topic of what should be a physical button, and what should be on a screen, there’s no reason to do all the screen work twice. Especially not if it adds to customer confusion over their vehicle controls when their phone isn’t connected to the vehicle.
Lastly, the example is given for car settings that are usually in menus, and guess what? Automakers can reimplement all their vehicle controls just for Next-Gen CarPlay. The automaker can also decide to punch through to their own native UI for the settings rather than rebuild it, but that linkage still needs to be made to get people there.
The work to do these things is depicted as being easy, trivial even, but someone has to keep every vehicle setting in sync between the car’s native system and Next-Gen CarPlay. Even if the individual tasks are easy, doing all the tasks twice for everything in the vehicle and making sure they work when a component of the system (the asset package loaded from the phone) isn’t easy or trivial.
I’m not a car enthusiast, but I am a CarPlay enthusiast. CarPlay was a requirement the last time I shopped for a vehicle, and I’d never buy another vehicle that didn’t have CarPlay support.
My boyfriend was skeptical of CarPlay, but after a few rental cars with CarPlay he very quickly got it. It’s great. However, Next-Gen CarPlay isn’t that and it’s hard to see customers lobbying for it because Next-Gen CarPlay will be different in every vehicle.
Apple assumes that all automakers make bad choices in interface design, and that Apple makes better choices. That’s not universally true.
I’d posit that a reason why people love CarPlay so much is because the media, communication, and navigation experiences have traditionally been pretty poor. CarPlay supplants those, and it does so with aplomb because people use those same media, communication, and navigation features that are personalized to them with their phones when they’re not in their cars.
No one is walking around with a speedometer and a tachometer on their iPhone that need to have a familiar look and feel, rendered exclusively in San Francisco.
As long as automakers supply the existing level of CarPlay support, which isn’t a given, then customers like us would be content with the status quo, or even a slight improvement.
How would users vote with their wallets for Next-Gen CarPlay experience when the benefits to the customer seem non-existent?
While Apple has been dreaming of controlling car interfaces, Google has actually been doing it with Android Automotive. Android Automotive stays with the vehicle, so there is no twined interface design step or synchronized phone compositing. It is the vehicle. If an automaker adopts Android Automotive they would rightly ask, why they would do the extra work for Next-Gen CarPlay? Android Automotive supports existing CarPlay, as a windowed media experience, and that’s enough for Google to Trojan Horse their way into every car.
In my humble opinion, Next-Gen CarPlay is dead on arrival. Too late, too complicated, and it doesn’t solve the needs of automakers or customers.
Instead of letting the vehicle’s interface peak through, Apple should consider letting CarPlay peak through for the non-critical systems people prefer to use with CarPlay.
One of the drawbacks of existing CarPlay is that your navigation app can only display in the center console, and can’t provide directions in the instrument cluster in front of the driver, like built-in systems can. Design a CarPlay that can output multiple display streams (which Apple already over-designed) and display that in the cluster. Integrate with the existing controls for managing the interfaces in the vehicle. When the phone isn’t there, the vehicle will still be the same vehicle. When the phone is there, it’s got Apple Maps right in the cluster how you like it without changing the gauges, or the climate controls, or where the seat massage button is.
Offer automakers the ability to add navigation waypoints for things like charging stations that are relevant to the vehicle you’re in. Like you have free or discounted charging, or certain fast-charging partnerships, tell the phone it can display, or emphasize that charging network.
The everyday irritations people have are mundane, practical, and are not related to how Apple-like their car displays can look.
If Next-Gen CarPlay doesn’t actually ship in something, or only ships in a Porsche, and Android Automotive takes over nearly every new car, the opportunity for Apple to mould anything about the car experience dwindles. The time for B2B videos about how easy it is to do work no one wants to do, or anyone benefits from, has passed.
Category: text
The keynote video was a real mixed bag. But not a mixed bag like Starburst candies where there are only a few good ones and you’ll still eat the other ones. This is a mixed bag like Starburst candies, artisanal chocolate truffles, and dog turds, and the dog turds got all over everything else.
There’s really good, clever, ingenious stuff in there, but I had a viscerally negative reaction to the part of the keynote presentation I was most trepidatious about: AI. Even in the Apple Intelligence section there were decent features that could be helpful to use in the real world, but then there was the laziest, sloppiest stuff shoved in there too.
Unsurprisingly, my negativity is nearly all focused at generative image slop. The examples they showed were akin to the results of image generators from four years ago. The appeal, that these models would personally understand us, and our relationships, made it all the more alienating when applied to generative AI, and not schedules, or directions.
The first example: Wishing someone a happy birthday, caused such an intense revulsion that I tried for a moment to convince myself that this was a joke that they’d say they weren’t doing this because this literally looked like you were insulting the person you were wishing a happy birthday to. But no, they pushed on with an image of a “super mom” and I began to question the taste of everyone from that generative AI team on up through all the executive ranks.
That image looked nothing like the photo from Contacts of the mom in question. It looked distressingly “AI”, and had that weird, smeary not-violating-copyright Superman shield. Revolting.
There are people that are entertained by the output of generative AI models, but those people have access to those models already.
The sales pitch for Image Playgrounds, where people can select from a pool of seed images to graphically build an image is just a front for having the system generate the prompt for you. As if the greatest difficulty in image generation is writing. The absurdity that a project like this gets time and resources put into it is mind boggling to me.
Image Wand depressed me. There was a perfectly adequate sketch that was turned into a more elaborate —but not better— image. Another empty area of a presentation was circled and stuff was mushed in there too. Remember that this is, essentially, a substitute for clip art. And just like clip art, it’s superfluous and often doesn’t add anything meaningful.
Genmoji is quaint compared to the rest of this mess. It generates an image made of the corpses of other emoji artists’ work. It’s likely to be about as popular as Memoji. Unlike Memoji this can be applied in the new Tapback system with other emoji, even though it’s an image, so it might not suffer the sticker-fate of other features. God knows what it will render like when you use it in a group chat with Android users.
Clean Up is the only thing I liked. See, generative image stuff doesn’t have to be total sloppy shit! It can help people in a practical way! Hopefully, when people are rightfully criticizing generative slop they don’t lump this in.
As for the rest of “Apple Intelligence” it’ll need to get out the door and survive real world testing. It seems like there’s good stuff, mixed with bad stuff. The integration with ChatGPT is what I would categorize as “bad stuff” and I don’t appear to be alone in that emotional reaction to it. I don’t care if OpenAI is the industry leader, there’s no ethical reason why I would use their products, and Apple made no case about why they chose to partner with a company that has such troubled management other than OpenAI being the leader in the segment. Their ends-justify-the-means mentality seeming having justified the means.
Speaking of those means. Apple had an interview with Craig Federighi and John Giannandrea with Justine Ezarik (iJustine) asking them questions (a very controlled, very safe interview) and The Verge was in the audience for this with Nilay Patel, David Pierce, and Allison Johnson live blogging. Nilay:
What have these models actually been trained on? Giannandrea says “we start with the investment we have in web search” and start with data from the public web. Publishers can opt out of that. They also license a wide amount of data, including news archives, books, and so on. For diffusion models (images) “a large amount of data was actually created by Apple.”
David Pierce:
Wild how much the Overton window has moved that Giannandrea can just say, “Yeah, we trained on the public web,” and it’s not even a thing. I mean, of course it did. That’s what everyone did! But wild that we don’t even blink at that now.
The public web is intended for people to read, and view, and there are specific restrictions around reproduction of material from the public web. Apple, a company notoriously litigious about companies trying to make things that even look like what they’ve made, has built this into the generative fiber of the operating system tools they are deploying. I do not find that remotely reassuring, and isn’t better than what OpenAI has said that they are doing. [Update: Nick Heer wrote up the sudden existence of Applebot-Extended. Get in your time machines and add it to your robots.txt file.]
Apple did say they’ll ask before sending stuff to OpenAI, but it’ll also suggest sending to OpenAI. I very much hope there is a toggle to turn off Siri even suggesting it, but Siri suggests a lot of stuff you can’t turn off, so who can say for certain.
The other features regarding text are pretty much what I hoped they wouldn’t be. Summarizing. I wrote a blog post for Six Colors about what I hoped I wouldn’t see AI used for from WWDC, and they did all the things. It’ll write or rewrite things. It’ll summarize things. It’ll change tone. It’ll introduce spelling errors.
David Pierce, again:
Everyone has the same ideas about AI! Summarization, writing mediocre emails, improving fuzzy search when you don’t know exactly the keywords to use. That is EVERYONE’S plan for AI.
Notification prioritization could be something, but it still can’t filter out junk notifications from overzealous marketers abusing their app’s notifications privileges.
I’m not filled with joy by any of it, and that really smears the rest of the diligent and thoughtful work done by many other people at Apple with this shit stain of slop.
So then here’s a brief bit about cool stuff: I really like the idea of iPhone mirroring. I like revising notifications. I like continuity improvements. The new Mail for iOS stuff looks good. I’m cautiously optimistic about the new Photos app, even though I almost exclusively want the grid, and the current design makes it easy to ignore the terrible machine learning suggestions.
Math Notes — despite it’s eminently mockable name inspiring people shove it in a locker or give it a swirly — seems like such a neat idea that capitalizes on “Calculator for iPad” but makes it relevant to the iPad.
I was even surprised to see that tvOS will be getting features at all. The only feature announced that seems interesting is InSight, which is like Amazon Prime Video’s X-Ray feature that they’ve had forever. This is objectively good and I’m happy they did it. Unfortunately, it’s only for Apple TV+ shows.
The other changes, turning on captions when the TV is muted, or automatically when skipping back, are … fine.
Highlighting 21:9 format support for projectors made me furrow my brow, because that seems like a feature for a very, very, very specific and tiny group. It’s not bad that they did it, but just weird to brag about.
In a bizarre twist of fate, the iPadOS section of the keynote made a big deal about how important apps are and demoed the TV app for iPadOS to show off the Tab Bar, which is literally the old pill-shaped floating bar at the top of the TV app, and how it can turn into the Side Bar, which is the new UI element Apple introduced at the end of last year.
The Tab Bar can be customized with things from the Side Bar. This is supposed to be a design feature that third party developers should use, which is why it was demoed like this, but it did immediately make me question its absence in the tvOS portion that came right before this where the poor customization and layout is a problem.
Also, this addresses zero of the reservations iPadOS power users had. No updates on Files app, or multitasking, or the new tiled window controls shown for macOS. But it gets Tab Bars that turn into Side Bars as a new paradigm.
Speaking of media weirdness, I’ll circle back to the beginning of the presentation for visionOS 2. I still don’t have any plans to buy a headset of my own, so this part really didn’t peak my interest much. This is the yellow Starburst in the mixed bag.
It seems like there’s a lot of catchup still with generating media, and proposing tools for generating media. In my post suggesting what to do for Vision Pro content, I mentioned that Apple needs user-generated video, and that the largest source of user-generated stereo and immersive video is YouTube.
Which is why Apple announced that there would be a Vimeo app for Vision Pro this fall for creators to use.
Vimeo is the artsy version of YouTube, that’s been slowly dying for forever. It’s never totally died, which is good because I host my VFX demo reel there, but it’s not growing. They even killed their tvOS app, and most of their other TV apps, in June of 2023 and told affected customers that people should cast from their iPhone or Android phones, or use a web browser on their TV.
Naturally, Vimeo is the perfect partner for growing a first in class stereoscopic and immersive experience for video creators. Vimeo doesn’t seem to want what people think of as “creators” they want filmmakers that make artsy short films.
A huge flaw in that line of thinking is that Apple marketing increasingly relies on YouTubers to promote and review their products. There is a disconnect here. I don’t want a YouTube monopoly, because it leads to things like their tacky screensaver, but this doesn’t seem like a coherent plan for video content production.
Also they mentioned partnering with Blackmagic Design (makers of many fine and “fine” media products) for stereoscopic support, and highlighting Canon’s weirdly expensive stereo camera lens that they’ve been selling for a while. Truly bizarre, and in my opinion very unlikely to go anywhere interesting.
The other kind of relevant thing is automatic stereo conversion of images. Apple did that thing where they showed image layers parallaxing inset in a rounded rectangle so there’s absolutely no way to judge what they were able to do, but it is the kind of thing that I’ve been suggesting they do instead of less-great mismatched iPhone lenses and the quality loss inherent in that approach.
Surely, many will dismiss any post-processed stereo, but it could theoretically work better for some. If they get stable generative fill some day we may see it for video, or at least stereo converted Live Photos. I’m not mad about that at all. I don’t have a burning desire to view media that way, but it’s much more practical for people buying these headsets.
The changes to iOS to move icons are good, but the dark mode icons all look kind of bad. Which is weird because I use Dark Mode on my iPhone 24/7, so I feel like I should prefer them. Odd to even think of such a thing but the icons all look so strangely ugly with the background color just swapped for black. The icons also look quite appalling in the color themes at the moment. It feels like the sort of thing you’d find on a Windows skinning site circa 2001. It’s too monochromatic, and flattens out everything instead of being an accent color. I don’t quite get the vibe, though I know the idea of doing something like this appeals to people. I’m not certain the execution will.
Concluding the video with the “Shot on iPhone. Edited on Mac.” after the last video was allegedly edited to some degree on both the iPad and Macs, is perhaps fitting, given the realities of Final Cut for iPad support (and managing files! Simple-ass files!), and is only notable for that reason alone.
Just a real mixed bag. Smeared with sloppy AI that feels as rushed, and as “check the box” as a lot of people were worried about. Like the product of a company that both learned a lesson from the Crush ad, and didn’t learn anything at all. As these things slowly roll out in betas “over the next year” I hope that people apply the necessary, probing criticism that can help guide Apple to make better decisions in the shipping products, and for 2025.
Category: text
In my last post where I emphasized (yet again) that Apple really should cut a deal with the juggernaut that is Netflix, I wasn’t endorsing the streamer’s TV app, but talking about it’s unassailable dominance in streaming. Not working with Netflix hurts Apple more than it hurts Netflix. Even if every user with an Apple TV canceled their Netflix subscription… well, Netflix would be fine.
The Netflix tvOS app is very hostile. The autoplaying video is out of control in the app. Yes, even if you follow Netflix’s instructions to disable them. I can’t get a moment’s peace in that app. What Netflix does, other streamers copy, and that includes irksome autoplaying cruft.
And yet, Netflix is testing a new home view. I don’t have it yet, but you can read about it at The Verge and see it in action.
Ironically, Netflix removing the side bar and putting a streamlined set of options in a horizontal row at the top of the interface is the exact opposite of what tvOS 17.2 did. I’m definitely not the only one who noticed.
It’s like the old saying goes, “There are only two ways to redesign streaming interfaces: moving top row navigation to side bars, and side bar navigation to top rows.”
Oddly enough, you tap the “<” back button to get the top row to appear? It’s not an “^” button.
That quirk aside, the irritation is really the dynamic, swoopy, take-over-everything previews that happen when you hover over a tile. God forbid you don’t immediately have something taking over your field of view at all times or you might have intrusive thoughts.
From The Verge:
“We often see members doing gymnastics with their eyes as they’re scanning the home experience,” Pat Flemming, Netflix’s senior director of product, tells The Verge. “We really wanted members to have an easier time figuring out if a title is right for them.”
I would argue that Netflix would like to persuade people reluctant to push a button to play a preview that a title is right for them by forcing them to watch the preview, and minimizing alternative options by shoving them off frame. The entire interface doesn’t gymnastics to make previews more prominent and discourage people from continuing to browse is not something I am a big fan of.
Netflix already isn’t a place to browse for the faint of heart —or if you are going to do it, do it on your iPhone which will be a lot quieter.
While other streamers have copied Netflix, even Apple, with autoplaying video previews, they usually have a setting that let’s you turn it off.
Apple TV app:
Settings app -> Accessibility (not the TV app settings from the Apps settings) -> Motion -> Auto-Play Video Previews (Off)Prime Video:
Prime Video app -> Side Bar -> Settings Gear Icon -> Autoplay (Off)Paramount+:
Paramount+ app -> Side Bar -> Settings -> Video -> Autoplay (Off)Max:
Max app -> Side Bar -> Settings Gear Icon -> Playback -> Autoplay Previews (Off)
It’s very likely, in my opinion, that research folks at Netflix will be able to support Pat Flemming’s redesign being a good thing. It’s about getting people to play a video faster, and spend less time browsing, which is not the same thing as satisfaction with the browsing experience. It could be attrition, or apathy that makes a person give in to whatever tile took over the screen.
Anyway, the reality of the situation remains unchanged. Unseating Netflix is a non-starter. For the most part, reservations people have about Netflix as an app give way to how they feel like they need to have Netflix as a streaming service. Giving people ways to view Netflix without this browsing experience might also be a non-starter for Netflix, but it’s one Apple should pursue.
Category: text
Apple TV is the best of streaming boxes, it is the worst of streaming boxes. After a flurry of changes early on it’s life it’s stayed kind of static. It’s more attractive in 2024 than it’s ever been mostly because competitors have junked up their streaming boxes to turn a profit with banners in the interface, auto-playing videos, and ads in their platform’s screensavers. Apple has resisted that (with one big exception being Apple TV+ promotion).
For a few years (2016, 2017, 2018) I wrote a specific post before WWDC about updates I was hoping to see for tvOS. These were never requests for those features to be built in a few days, but things I was hoping had already occurred to Apple, like the many years I put picture-in-picture on the list before it occurred to someone at Apple to ship it in 2020.
I stopped writing these posts because fewer and fewer updates were coming out for tvOS, in general, and those that were were often tied to new hardware launches usually occurring late in the Fall. The software changes were not always exclusive to new hardware, but from a marketing perspective it made it sound like the new hardware could do more stuff than the previous model.
There’s no reason to expect any announcements about tvOS, but perhaps Apple will highlight some more about Apple TV+ …at the developer conference.
That’s not to say that development on tvOS has stopped, but it’s definitely not in the spotlight. Last WWDC, there were virtually no user-facing features to mention other than the inconsequential updates to the default video player, and user profiles. I say inconsequential because —who’s using them?
Apple has the carrot and stick for new developer features, and on iOS and macOS they can wield both (mostly because they can kick your app out of the store, or they can say “no more 32-bit libraries”). On tvOS, they have no leverage because they have a small share of the media platform market. There’s no stick. Apple needs streamers to build for their platform so they can’t even enforce things like the player controls, which is why we all have Apple TV remotes with jog wheels that work in theory but not in practice. Every app I use with profiles still has their own profile system totally detached from tvOS user profiles, because they already built their profile system (and need it for more important platforms) so what incentive do they have?
The only big change in 2024 was halfway through December when Apple dumped tvOS 17.2 out for the public with its heavy changes to the Apple TV app, and Search that were half-baked. Presumably this was to meet some internal release target, and not because it was ready, as features were missing, search results were (are) wacky, the way you navigated TV shows you purchased got worse, etc. It was a messy, but large, update.
Knowing that it’s very unlikely we’ll see anything from Apple for the Apple TV this summer, I’ll offer a critique of where things are at instead, and offer some possible solutions ranging in complexity.
Regardless of internal shipping targets, fall hardware announcements, or week-before-Christmas-break releases. I’ve learned that predicting what Apple will do with the Apple TV is foolish, but these are definitely pain points.
Bear in mind that I’ll be repeating a lot of what I’ve said before.
The app still sucks butt. I’ve said it for years, even prior to 17.2, but the app’s purpose is to let people watch what they want to watch centered around media, rather than centered around apps. Apple feels like the app’s number one purpose is to get people to subscribe to, and watch, Apple TV+ -even if they subscribe to, and watch, Apple TV+ already.
Opening the app, on a freshly updated system (not a new system, but just one where the software was updated!) will kick a user to the Apple TV+ section of the app. This makes it more important than any other function of the TV app, it’s saying this is numero uno.
If you’re not a current subscriber the first item in the carousel is “Come Back for New Apple Originals Don’t miss the star-studded stories being added every month.” I love alliteration as much as the next person (I’m not a monster), but this appeal just irks me further. Associating negative feelings with a service that by all measures has critically acclaimed shows.
My subscription is inactive, but it’s not because I forgot Apple TV+ existed. Such a thing is impossible to forget when you own Apple devices, every single one of them tasked with nagging you to subscribe. The TV app on iOS, the TV app on macOS, the TV app on iPadOS — at least you can’t watch TV on your Watch or it’d nag you there too!
I will resubscribe and watch things later. There aren’t enough hours in the day to watch everything, and my priorities are my own. When I do reactivate the subscription it won’t be because I saw Ted Lasso’s grin emblazoned across the interface. I don’t hate Ted Lasso! Apple is making me complain about seeing Ted Lasso! It should be a delight!
This was formerly “Watch Now” which was never a great name for that view in the app. It’ll make more sense when the home screen dies. Because just like everything else on the Apple TV, it’s name pollution all the way down.
Apple really messed up this area of the app over the years. The simple “Up Next” view is superseded by the same garbage carousel of auto-playing video every other streaming-centric platform has.
However, unlike other platforms, where that space is for rent to the highest bidder, Apple occupies nearly all of the carousel for the purpose of promoting Apple TV+ shows, movies, and MLS games. You’ll eventually see another show from someone other than Apple in the carousel, but the priority is always Apple.
The other thing about that carousel, that absolutely kills its utility, is that it highlights shows regardless of whether or not you’ve already watched them. Right now, as I type this, the first recommendation is for a MLS game that’s live, a thing I will never watch, and the second thing after that is Ted Lasso, a TV show that I have watched, and completed its run in May of 2023.
After the carousel, and the Up Next row, we have these rows:
So, That’s carousel, Up Next, and 29 discreet “rows” of stuff, that happen to be ordered with the first third of it about Apple TV+. The rest is a mix of incredibly generic stuff, and a few personalized options.
I’m pretty salty about this because personalized recommendations are more relevant than Apple TV+ but they’re de-prioritized and very limited in scope.
Several of the rows have collections —displayed as a tile in the interface alongside the shows and movies. The tile will take you to another view, usually with a clip-arty thumbnail and header, that’s showing rows of movies selected, and grouped by unseen humans with editorial context provided in the subheads for how the tiles are grouped. This is not personalization, but it offers something that’s deeper than the surface-level overviews offered in the Home view.
I’ll highlight “Pride 2024 Living!” because it’s pride month, so I’m shoving my homosexual Apple TV agenda down your throats.
This collection is unlike the year-round “Action” or similar genre collections way at the bottom of the Home view. The majority of the screen is taken over by Bottoms to “Buy, rent or watch on MGM+” but … it’s also on Prime Video, something people are far more likely to have than MGM+. That is disclosed if you click through, and then down to the How to Watch section. Neither view discloses that I already started watching this in Prime Video or shows a progress bar with that status (I’ll finish it later, get off my back!)
The other billboard-sized tiles are Love Lies Bleeding, RuPaul’s Drag Race (yes, all of it), and All of Us Strangers.
Below that are row after row of movies and shows that use the horizontal thumbnail, with text about the media placed to the right of the thumbnail, so only two thumbnails, and their respective text, are visible at a time per row. This isn’t really used in the Home view, and I can see why because it’s not as information dense, but it avoids people having to click through into each tile. Odds are pretty good that people might not have heard of many, or most of these.
Further down, in “LGBTQ+ Visibility for Kids”, they use another element that I have seen in the Home view on occasion, and revile, and that’s three larger thumbnails with 2-3 lines of tiny preview text placed vertically under each thumbnail. Both the thumbnail and the text count as navigable elements, with separate click targets. If you click the text it takes over the full screen to show you just that block of text. If you click on the thumbnail it takes you to the season-episode view of the show with the episode thumbnail given emphasis. You can click the thumbnail and it’ll start playing the episode, or you can go down to the preview text, which is not exactly the same as the preview text you were just looking at, and click on that to get the detailed information about the episode including what services it’s available to watch on. Why the preview text from the collection view can’t take you to the thorough episode info, I don’t know.
Why not present the thumbnail and text as one click target and take them directly to the episode info page instead of any intermediary step that looks the same but functions differently?
Of the two ways to present thumbnails and text I guess I prefer the horizontal ones.
Then at the bottom of the collection is a row of more collections called Deep Dives here.
I really appreciate the thoroughness of these kinds of collections, even if I don’t love how they are presented. While I complained about the presentation of “LGBTQ+ Visibility for Kids” I love that it’s there at all, and that someone was thoughtful enough to highlight specific episodes of TV shows.
It offers a choose-your-own-adventure way of browsing that is not tailored to you at all (like it’s unlikely that a lot of this stuff going to be from streamers you’re subscribed to), but you can drill down as much as you want to.
Where does this collection go when the Home view is reorganized next week? Where does it go at the end of the month when it’s no longer pride month? If I liked the selections made by the anonymous person(s) behind this?
I can’t even tell someone to go to this collection unless I describe navigating to it in the underworld of the Home view while it’s still June. You can’t ask Siri. “Hmm, I’m not findnig anything for ‘Pride 2024 Living’.”
This is not dissimilar to the fate of a lot of the playlists that are created for Apple Music, where they might as well not exist if they’re not featured in front of your face. These are all orphans adrift somewhere on a server and they don’t link to any kind of structure.
The genre collections “Action”, “Romance”, “Comedy” etc. all have collections inside of them too, but they seem to be organized around a sort of timeless agenda, rather than demarcating when the collection went up, or was updated. Which means that there’s nowhere to find the previous Pride Month collections, which are marked by year. Let alone year-round LGBTQ+ movies and shows beyond the LGBTQ+ Romances under the Romance collection, which stops after a handful of predictable suggestions before it’s just a sea of tiles of any LGBTQ+ romance (including the movie Bros which I would personally shoot every copy of into the sun.)
Why not recommend collections to people based on their viewing history, including the nested collections, and people could browse up or down from them? Like if I’ve seen movies in a collection, show me the deep dive that goes further as a tile right on my home page. If I watch gay shit, and zero soccer, put this year’s editorial collection of gay shit up higher than MLS Season Pass.
The most far-fetched thing would be to break down the editorial wall of these collections coming from some general-purpose board. Attach names (could be made up, to avoid harassment, I don’t care) where users can realize they like Sandra’s picks, and Sandra’s collections, or they don’t and they align with Brendan’s.
Personalize based on taste. You know, like the staff recommendations of a book store, or movie rental store, that this is attempting to imitate.
This collection shows you great movies available to Apple TV+ subscribers, a new thing Apple started testing out where they’d pay a movie studio to stream a popular movie (what a concept). They do a pretty bad job at surfacing the movies relevant to you, specifically, you have to browse this as it was constructed for everyone with Apple TV+. It doesn’t tell you when a movie is leaving, unless you click on it to get to the info page for the movie.
You can browse the internet to find things to then go look for from those Apple TV+ limited releases. This is some real when is the super bowl level content but it’s not superfluous because it’s easier to look at a web page with the list of movies than it is to look at the collection on the Apple TV.
There’s a special collection under Movies Spotlight, that’s not like the other hand-crafted collections. It has the capacity to show rows of “Recently Added Movies”, “Popular Movies”, and “Movies We Love” filtered by the channels and apps you’re subscribed to already. What a concept. You used to be able to get these kinds of rows awhile ago when the Home view was less devoted to Apple TV+ but I guess they ran out of room after the Apple TV+ prioritization and created Explore Movies for the overflow.
The only problem with that is it’s 10 rows down from the top. I bet a lot of people don’t even know it’s there, or that you can click on it, because they probably think it’s just a display tile of some kind saying, “Explore Movies” because sometimes the interface uses tiles as graphical elements.
It’s just those three rows that are filtered based on your subscriptions. The rest of the collection is an assortment of popular movies mostly by high-level genre containers. They don’t link to the genre collections from the bottom of the Home view though. This is completely different “Action” from the other “Action”.
I’ve watched a lot of stand-up comedy with my boyfriend, Jason, in the Netflix app (which is not part of the TV app experience) and in the HBO Max app, but the number of hours I’ve watched Star Trek: Discovery and the recency of watching The Last Starfighter from my media library, mean that those are weighted heavier than anything else.
The real problem is that I didn’t even know those were my current recommendations because they were buried in catacombs under a landfill of garbage.
I also wouldn’t seek out those personalized sections because like the rest of the interface it doesn’t really know what I’ve watched before if it was watched outside of an app that fed data to the TV app. For example: One of the recommended movies is Star Trek III: The Search for Spock which I have seen more than a handful of times, but I have no way to mark that movie as watched.
If I long-press on a movie or show, I have just two options: “Go to Movie” or “Add to Up Next”. There’s nothing on the “Go to Movie” page to mark the movie.
You might think “Oh just add it to Up Next, because you can mark a movie in Up Next as Watched.” You’re absolutely right, but keep in mind that geographically that queue is second-from-the-top. So You can swipe, or tap, all the way back up to the top, or hit “<“ then long press on it, then mark it watched, then go all the way back down (no button for that). How often do you want to do that?
If you go all the way back down the movie you marked as watched will still be in the recommendations. You have to force quit the TV app to refresh the list, but rest assured it will be missing after you completed that simple 78 step process.
There is one row from the personalized rows that gets special treatment and that’s For You, which added a User Profile doohickey on the right side of the top of the row. So navigate to the For You row, then over three, then up one. Once you’ve entered that Konami code, the little thing will highlight with the words “Set Up” click on it for a modal screen explaining “Add or remove user profiles and get recommendations for everyone included.” Which is just about the worst copy.
You see, the first part of that sentence is targeted at the last button in the interface. If you already added all your user profiles, then it can be completely ignored. It should say “Get recommendations for the selected user profiles.” Then the profile selector, then “OK” and finally the “Add Users to Apple TV” button.
However, that’s all irrelevant, because Jason and I don’t always watch the same stuff on the Apple TV so I would never turn on a universal toggle that tied everything together. I need to be able to specify who is currently watching the TV. Jason has no interest in Scavengers Reign, Star Trek, or the movies I watch for the podcast.
If that didn’t make it irrelevant enough, none of the streamers hook into these user profiles. So if I wanted to include or exclude Scavengers Reign it’s immaterial. Either when it was on Max, or now that it’s currently in Netflix.
So functionally, the “For You” row is de facto for me, and for everyone using the Apple TV, but never tied to them.
The row for “News” with news broadcasts that have a pink label saying “Live”. These are not specific to what you have installed on your device, they’re just possible sources for live news. So if you click on an news broadcast that you don’t have installed it will ask you to connect an app on your box (where you might no longer have an active subscription) or it will ask you to install an app.
So … it’s more that it’s theoretically live, and just those specific things someone selected (which includes Fox News). It does not connect to my DirecTV Stream app, or any other over the top (OTT) “cable like” provider. It doesn’t connect to any free ad supported TV (FAST) provider.
I mention all that because live sports are connected. If you click on a game that has a “live” sticker it’ll show a screen similar to a movie or TV show with a watch now button, and information about where it’s available (like DirecTV stream). It’ll open the app, “change the channel” to ESPN, and you’re off to the races baseball stadium.
What if you want to watch live TV that isn’t news or sports from the TV app? You’re shit out of luck.
Unscripted reality TV, game shows, channels that play Law & Order: SVU for 8 hours straight, are all unknown to the interface. We have had the technology for interactive programming guides for more than 30 years, and each OTT or FAST service has one, but there’s no unified programming guide that exists for Apple TV. You must go to each silo that offers live content and browse their programming guides individually.
I have been writing about interactive programming guides since 2016, before there even was a TV app. Back when we didn’t even have a dark mode in the interface.
Two years ago, Amazon revealed their unified programming guide for live TV, which funnels all of it together. I wrote about it here. I would hope that it occurred to someone at Apple that they should be working on this, but I have seen no indication that anything other than live sports is a priority.
This is important enough that I feel like it needs its own section, probably in the side bar, if not Settings (where you can only clear your history in total). Recently Watched is really the wrong name, and it’s in the wrong place, and it has the wrong options.
It should be called “History” and it should be in the side bar (more on the side bar later). It should not be a single horizontal row that you scroll infinitely at the very bottom of the Home view.
When you long press on it you have three options: Go to Movie, add to Up Next, Remove from Recently Watched. Which are fine options, but it doesn’t display anything about when it was started or completed. If I want to know when I watched Scavengers Reign I’m out of luck.
A more robust solution can be found in YouTube, where there’s a very thorough and editable, list of what’s been watched. If Jason watched the NPR Tiny Desk Concert for Wicked, and now all my recommendations in YouTube are for pop-u-lar Wicked videos, I can quickly fix that.
Speaking of users, a missing option is to add or remove user profiles from the Recently Played items. You know, in the event the user profiles actually functioned for anything useful, then I could mark something as watched by Jason, myself, or the both of us.
Hit the “<“ in the TV app, or swipe/tap past the left edge of the screen, and you’ll get the side bar. Like I said it kicks you to the Apple TV+ section on occasion, but this is also where you find Home. Just like I wrote about back in November, it sucks.
Again, the priority is on things that Apple makes money from, not things you might necessarily be using. There is no way to reorder or remove Apple TV+, MLS Season Pass, Sports, Store, or Library. They are always there, as something you must swipe/tap past to get to Channels & Apps.
Channels & Apps has added exactly two features to help with organization — no, it’s not long-press and everything wiggles and you drag things up and down. You can hide an app, or you can “pin” a Channel or App.
“What’s pinning an app do?” You ask, because you obviously didn’t know that was event possible because only a mad man goes through every inch of the tvOS interface long-pressing on everything. Well, pinning creates another divider, between Channels & Apps and the alphabetically ordered channels and apps. The pinned item is placed there with the first pinned item in the number one spot, and any subsequently pinned items appearing underneath.
“At least hiding is pretty straight forward, right?” No. When you hide something a notification will briefly appear in the upper right corner explaining that the channel or app is hidden until you play something from it again. How do you do that if it’s not in the interface, you wonder, well you can go to your home screen, or search for something you know is in that app. There’s nothing in the byzantine Settings for this that I can find. No row at the bottom of the home screen for show hidden channels and apps. You just simply get a video to watch.
That also means that if you didn’t want Prime Video in the side bar, for example, because it’s not that useful to you there, and you want to browse the full app, then it’ll always be there. You’d have to rely on having other stuff that pushes Prime Video offscreen.
Again, I already wrote about this, but the interface for each channel and app is spartan. They all get a carousel of stuff. If there’s something you were watching on that channel or app you’ll get “Up Next on Max” where it has filtered the Up Next queue for just what’s in the app. That’s a nice, if somewhat useless touch. But then things go downhill fast with “Top Chart”. Man, give me more of “Top Chart” I love whatever’s popular without context or genre.
Things filter down through other rows that are both specific to each streamer, but also incredibly generic at the same time. These lists have no knowledge of what you’ve already watched. In Prime Video, for example, “Amazon Originals: Highlights” first recommendation is Rings of Power which I watched in its entirety. Amazon knows I watched it, and it’s own app won’t recommend that show to me. Apple TV knows I watched it because I can scroll back in my Recently Played (it was last year so this was fun to do). However the Prime Video view has no way to filter, no algorithm to recommend, nothing but the same recommendations for everyone.
I’ll reiterate that it makes anything in the Channels & Apps view useless to me, because this is objectively a worse experience for data even if it is laid out in a less aggressive way than Prime Video’s own app. I don’t want to browse “Top Chart”.
Unless you never want to browse, and you exclusively know what you want to watch before you turn on the TV, I would never recommend this kind of impersonal navigation experience.
I know people that subscribe to Paramount+ as a channel, because the Paramount+ app is buggy and awful (there were a couple weeks where if you paused the episode and then hit play, it would play the auto-playing episode preview audio from the app interface over the video you resumed and you had to exit the video. Just one particular example.) However, those people know they’re subscribing to watch a Star Trek show, and that’s all that they care about, and they’ll unsubscribe. That’s not a browsing mentality, and that’s fine, but a lot of people want to know what value a streamer offers, and a lot of streamers would like to tell you about how valuable they are to you specifically by recommending personalized content.
I don’t care what Apple needs to do to get Netflix to integrate with tvOS, and I don’t care, but whatever it is it’s past time to do it. A content recommendation system, and navigation system, that has nothing from Netflix only appeals to people that don’t have Netflix.
An example of how it hurts users is easy to find: That “LGBTQ+ Visibility for Kids” row in the collection highlighted TV shows that are available for kids to watch on Netflix, like Star Trek: Prodigy which used to be on Paramount+ before Shari started having money trouble. So it just shows people how to buy the episodes from Apple.
Apple, if people spend money they didn’t need to spend, or skip watching something because it was perceived as an extra expense, when they could watch that thing with a subscription they already had for free, they are not going to like your streaming platform, or your recommendations. That is not how you treat your customers, and saying that the onus is entirely upon Netflix so that you are excused from criticism is weak.
What’s the point of sifting through the current version of the TV app, or an improved version of the TV app, if it will be missing the biggest player in streaming?
The TV app was announced in June of 2016 for tvOS 10, and released in tvOS 10.1 late in the fall of 2016 making it almost as old as dark mode. The odds Netflix is going to reconsider doing anything, out of nowhere, unmotivated by any kind of incentive, are basically zero, so make it work.
The mess of the TV app is precisely why I primarily use the home screen on my Apple TVs. The TV button is remapped from the TV app to home screen. The way I interact with the TV app is:
Then when the TV app is in the top row (which it is by default) I can hover over it and get right to my Up Next queue from the home screen without having to deal with any Apple TV+ sales pitches, or the Apple TV+ carousel.
As you might guess from my thorough critique of the TV app, I don’t feel like I’m missing out on anything.
Having said all that, the home screen needs to go away and be replaced by a unified TV app that supports access to apps, and personalized TV show and movie recommendations. For that to be a success, the TV app needs to not suck.
Fire TV, which has both improved and degraded the user experience at a startling pace, has a top row in the interface that lets a person pin pieces of media, or whole apps. Instead of fully integrating with Netflix in their home screen view, and proceeding to fruitlessly fight with Netflix, people can pin the Netflix app right there. Things can be reordered too. Apps they seldom use can go in a junk drawer. Other rows have personalized recommendations that are just from Amazon, or mixed, or things that are free, or things that are for rent — but they’re all personalized to a degree. There’s still too much self promotion, and the terrible auto-playing carousel, etc.
There’s a solution out there, but it would require Apple to give up control over the TV app as a promotional tool for Apple TV+ to let users place other streamers in places that reflect the importance of the streamer to the specific user. Not everyone will want to put a Netflix button in the top of the TV app, but it’s going to be more than zero.
My opinion about the Apple TV as a gaming platform is unchanged from 2015. Talking about console quality graphics is meaningless when the platform isn’t a console. I know there are fans of gaming on Apple TV, and they play the assortment of iOS-ish games that are available. Every Apple Store I’ve been too recently has two Apple TVs each connected to its own TV and each with a single PS5 controller. The Apple TV still isn’t a PS5. It isn’t a Nintendo Switch. It isn’t an Xbox.
Through a combination of factors, including but not limited to persistent storage, and the lack of a game controller, it’s just not a games console.
If a hotel advertised that it had a pool, and you wanted to use a pool, then that’s great. If a hotel said that they let people bring their own above ground inflatable pool as long as it wasn’t too big, then surely someone would take them up on the offer but most people would merely use the hotel as a hotel.
Now I don’t need it to play games, so I want to stress that it has very little impact on me whether or not it’s good for gaming. I do care that games are one of the justifications for the two storage tiers for the device, and the the overall price of the hardware.
I’m in the camp of people that hates the new Settings app on macOS, so complaining about the relatively minor problems of the tvOS Settings app feels silly.
But I’m gonna do it anyway.
It’s biggest issue is, surprise, organization. Unless someone tells you where something is in Settings you might not find it for a while, or you might find it in multiple places. There is no voice or text Search for Settings, which is what we all fall back to when we try to find something in iOS or macOS.
Settings -> General -> Privacy & Security -> Apple TV Users. That’s where you go to turn on or off whether or not an app can access Apple TV users (the only one I have there is Hulu, and it’s an inactive subscription at the moment.)
Settings -> Users & Accounts is where you go to switch current user (also possible in Control Center or the top of the TV app sidebar). It allows you to specify a default user, add a user, specify a TV Provider, and Home Sharing. Each user account here has privacy and security settings that are not in the Privacy & Security settings screen, including things like iCloud and whether or not purchases require authorization, and the toggle for sharing user data isn’t here, it’s in the other place.
Turning off autoplaying video in the TV app isn’t under Settings -> Apps -> TV, it’s under Settings -> Accessibility -> Motion even though the major apps that have autoplaying video have their own ways of turning it off, so this ends up mostly just applying to Apple’s TV app, and people irked at that specific app might not assume to look here.
If you need to manage storage for some reason that’s General -> Usage -> Manage Storage, which just shows the apps organized by size and puts a trash can next to them implying you’ll delete the app. Click the can and it’ll ask if you want to delete the app, or offload the app but keep it’s data. That reduced the Reuters TV app (why the hell did I download that?) from 120 MB to 78 KB. It doesn’t offer a way to filter or rank the list by how often you use an item, or to automatically offload unused apps. It doesn’t even say how much available storage you have.
Apple’s only three options for Home stuff are the HomeKit pane in Privacy & Security, which will only have something in it if an app on the Apple TV has requested access to HomeKit data, and the AirPlay & HomeKit menu where the device’s “Room” can be changed and Home Hub can be toggled on and off.
For the Home Hub, which bridges device access, and routes automations, it sure doesn’t do very much for controlling your home. If you tap on power (remember all TV remotes on the planet earth turn off the TV when you do that, but not this one) then you get the Control Center, and you can navigate to the Home icon, which will show Scenes you can toggle on and off. Not a status board for you lights, or your thermometer or anything.
I can use the Siri button on the remote to turn on the living room light, and that works just fine, but I can’t use this to make sure I turned off the lights in my office. I need to use an iPhone, iPad, Mac, or teeny-tiny Watch making this the only Apple devices with a display that can’t show me a Home interface, and it’s supposed to be the brains of this operation.
There are two Apple TVs in my home, and both fight to dethrone the other for the role of controlling my home. I don’t know why. I don’t know why I can’t just enforce order. If the one in my office takes over the role, my Eve Weather devices fall off the network, and other weird, undiagnosable, un-bug-reportable stuff happens. I manually toggle it off of being a Home Hub for a minute and then the living room Apple TV takes back over. I don’t want to leave it excluded as a Home Hub, because what if the one in the living room goes offline (loses network connection, installs an auto-update).
Let me designate a primary, then the other Apple TV can briefly take over before passing the responsibility back to the good one so that some of the functions in the home still work. I know people with Home Pods have it so much worse with the random reshuffling, which just makes it all the more baffling that we’re still asking for this, begging for this, to come in one of these WWDC sessions.
I’m not talking about putting a camera in the Apple TV to control your TV with hand gestures from across the room. I’m not suggesting anything sci-fi like adding a UWB chip to the Apple TV to geofence the specific living room area so when people brought their iPhones into it, and they had a user profile on that Apple TV, they’d have the the viewing history for that show or movie without having to do anything. I’m not asking for always-on voice control using any Apple device microphone in vicinity.
What I’m critiquing, and asking for, are solutions to the day to day inconveniences in a household which would not be solved by microphones, chips, and cameras. Pain points could be worsened by fancy new user profile stuff if user profiles stay exactly the same. If personalization remains less important than shilling Palm Royale.
Help us find shows from any streamer, not just Apple TV+, or movies that are for sale from Apple where there’s direct profit. When we want to browse, turn us lose on collections that align with our interests. If something is newly available on a streaming platform, and it might interest us, let us know.
If we are going to personalize our experience let us mark who’s watched something, and who hasn’t. Who has no interest in something, like let’s just say sports as an example. Let us organize the interface so what we’re likely to use is at our finger tips instead of buried.
Give us a place to aggregate all the linear TV we can watch from OTT or FAST. Let us pick and choose favorites from that.
That’s why this isn’t a “WWDC 2024 Wish List” where I’m setting some sort of impossible timer on my expectations. I know Apple’s not ready to do any of this. That they aren’t hungry to do any of this.
They ought to want it though, and you ought to want them to do it too.
Category: text
Over at The Incomparable, Scott McNulty and Jason Snell just celebrated a big milestone: covering the final episode of Star Trek: Discovery a show that they talked about beginning to end.
They had a few episodes of the TeeVee podcast dedicated to Star Trek: Discovery, where they covered the weekly releases, and eventually that spun off into it’s own podcast, Vulcan Hello (which may be a reference to the first episode of Star Trek: Discovery “The Vulcan Hello”).
The podcast continues on, but the thing it’s named for is over now. That’s kind of bittersweet, but I’m relieved that I’ll still have Scott and Jason to listen to whenever Star Trek: Strange New Worlds comes back.
Every week there was a new episode, I’d watch it, then hop into a conversation about it with other Incomparable hosts, and then finally listen to that week’s episode of Vulcan Hello. Rinse and repeat.
I’m grateful that I’ve also been on Vulcan Hello as a guest, or on other “season round-up” episodes of the main Incomparable Mothership show. Vulcan Hello will always be Scott and Jason though, starting off the episode with some jokes, and telling us what they think of the episode before they dive in.
There’s some other meta-level of satisfaction where I’m happy with their discussion of the finale (no spoilers, and if you want to know what I think it just so happens to be closely aligned with what Scott and Jason think on this one.)
I’ll miss Discovery for a variety of reasons. It was an … imperfect … show, but people wouldn’t watch it if they didn’t care about it, and I watched the whole thing beginning to end. CBS All Access to Paramount+ to Paramount+ with Showtime. From the indirect connection to the incredibly uneven Star Trek: Picard to Discovery’s role in spinning up Strange New Worlds it inarguably had an impact for all Star Trek fans.
Vulcan goodbye to Discovery. 🖖
Category: text
This is an Apple fan-fiction post. Even less relevant than when I wrote about why Apple should make AirPorts again. This is a product that I don’t see Apple making, but I’d like to wish it into existence. It struck me that we have a lot of various parts lying around, and a seemingly increasing desire on the part of Apple to be an end-to-end solution where the entire video production process to take place on Apple hardware (they seem a little less motivated on the software front).
What if Apple made a camera?
No, not the QuickTake, smartass.
Apple is one of the biggest camera makers in the world, and everyone sees images and video captured on one of their cameras every day, but those cameras are really part of a whole phone pocket-computer device. What if there was an accessory that was first and foremost a camera.
Last Fall Apple had an event (showed a video on the internet) for “Scary Fast” M3 MacBook Pros, MacBook Airs, and iMacs. They shot the whole event on iPhones and told everyone how they did it. A bunch of first-principles nerds got their computer engineering degrees in a wad over how they used lights because they felt that was cheating.
When Apple talked about the M4 iPad Pro at the Spring iPad Event Apple was a little less forthcoming with the details, but they still shot it with iPhones. More importantly, they showed off Final Cut Camera, an app that could be used on iPhones and iPads to offer some manual video recording controls, but it was really there to enable the iPad Pro to control the focus and pull in streams to Final Cut for iPad version 2.
You, and a few of your closest friends, could put together a multicam production on the fly with the iPad Pro being video village
You don’t have to own all of those iPhones, and it’s supposed to be like authorizing other ad-hoc networked devices, but if they weren’t your iPhones you were reliant on those people to have those iPhones to do your work. What if you were going to shoot several angles of sick skateboard tricks, but then Chet went and got a Samsung Galaxy? Thanks, Chet. Total Chet move.
People might not own all their cameras, and turn to rental houses to provide cameras to them for a shoot, but to my knowledge no one rents iPhones out. It’s possible someone does and I just haven’t heard of it, I’m not all-knowing, but it’s not the price that keeps people from renting them as they rent significantly more expensive cameras than iPhones. I guess because people think of them as cellular devices? How do you set it up and then return it to the rental house every few days?
Buying four iPhones is a thing a person could do too, but again, it seems like overkill. They’re destined to record or stream footage, and the rest of their capabilities are not only a waste, but there’s added complexity for device management.
This is also part of the reason why people have Continuity Camera struggles. Do they keep an iPhone worth over a thousand dollars just to use as a web cam once in awhile? Do they precariously balance that slab of glass on their monitor, or their living room TV with Belkin’s clips?
People turn to dedicated devices for these things if it’s something they do often. They invest in bulky, but robust, Sony mirrorless cameras with interchangeable lenses, or they grab the very popular DJI Osmo Pocket 3 to record their vlogs with something that’s a lot easier to hold for long periods than a glass rectangle. Many other hybrid stills and video cameras are on the market with each camera manufacturer trying to get a slice of the “creator” market.
Keep in mind that the vlogging-centric cameras have screens that either face the subject of the recording, or can swivel to face the subject, for the person recording themselves to monitor what the hell they’re recording.
The iPhone’s FaceTime camera, on the same side as the display, is nowhere near as good as the primary wide-angle lens on the back. That’s why Continuity Camera defaults to use the best camera, and you monitor from your … uh … monitor. With the iPad Pro you can monitor the streams from the iPad.
What if you could monitor from your iPad or iPhone the output of an Apple Camera? The screen could be anywhere you wanted it to be. Many cameras have a way to stream live video feeds to iOS apps made by the camera manufacturer but those apps are janky, and their connections fragile.
Is there a compelling product if you took the camera module from an iPhone, a big battery, some (removable, please) storage, WiFi, and put it all in one package? You could have something more cube-shaped than slab-shaped, and you could mount it anywhere with little concern over breaking something tied to a 2-3 year-long cellular plan.
Apple famously made the iPod Touch for many years, which was a stripped-down iPhone. There was very little reason to own both an iPod Touch and an iPhone because they were so similar.
It wouldn’t be competing with phones, it would be competing with DJI Osmo Pockets, GoPros, Sony and Canon “vlogger” cameras. It wouldn’t run games, or have an app store, and thus no regulation headaches. It wouldn’t cannibalize iPhone sales because it would be an iPhone accessory. You connect it up with Final Cut Camera and you’ve got synergy out the wazoo.
On a darker level, that appeals to the C-suite: If people invest in kitting out a bunch of non-iPhone cameras to go with their iPhone and iPad Pro then they’re firmly in your eco system, instead of relying on friends with iPhones, or relying on a person to over-leverage themselves on buying a stack of unlocked iPhones.
So we’re talking about something cube-ish with multiple mounting points. Ideally it has at least the wide-angle lens if not the complete camera array from the back side of the current generation of iPhone Pro Max. They could give it the thickness needed to have a real zoom lens and only one camera sensor, but I don’t know if they want to put themselves in the position of people asking for that on the increasingly thin phones that couldn’t possibly support it.
It could even be stripped down to just the wide-angle lens and maybe the LiDAR sensor, like the camera package on the M4 iPad Pro. Use that to assist in Cinematic Mode post-defocus. Apple could station those around a space and interpolate multiple LiDar feeds into a more stable mesh for projection shenanigans for Vision Pro.
It’s just a camera though, that’s the important part, and it wouldn’t be called iSight or QuickTake. They’d follow the very bland Apple naming convention of Apple Camera.
There would be some amount of on-device storage, of course, with multiple tiers where no one would want to buy the lowest tier but it would always be on sale on Amazon.
However, I would love to see a Cfast or SD card slot so the device could be entirely contained without external drives hanging off a USB-C port. It could write to the onboard storage, and card/external so there were backups (and a reason for internal storage) or write proxy files to one or the other. The important thing is that it can write out a lot, and it can write it out quickly, because it should support the precious log and ProRes abilities that make the iPhone flexible enough to do this kind of work to begin with.
The camera output would also be streamed to a controlling iPad or iPhone with the Final Cut Camera app, or to a Mac with Continuity Camera.
It seems unlikely Apple would ever design something as ergonomic for holding as the DJI Osmo Pocket 3, something as utilitarian as a Sony ZV-E1, or something with the mounting points of a Sony FX3. At least give it one threaded screw mount on the bottom so not everything is gingerly clipped to it with rubber pads. They could always make it some awkward-to-hold nightmare and rely on SmallRig (or their favorite ugly-accessory maker Belkin) to provide the actual utility.
That would satisfy the needs of any vlogger, influencer, or person-who-wants-to-have-the-nicest-camera-in-the-meeting.
The non-iPhone camera doesn’t have to manage anything other than the camera, the storage, and it’s ad-hoc WiFi. It’s not playing music, browsing the web, getting phone calls, loading QR menu PDFs, getting iMessages, dialing 9-1-1 — it’s just a camera, and it just recycles as many parts as possible.
Why stop there? If I’m imagining something completely improbable so why limit my famously optimistic scope? Let’s talk about something Apple very much wants to be associated with: filmmaking.
Apple shot their Spring iPad Event with iPhones that had Panavision lenses plopped on top of the camera’s own lenses. Stu Mashwitz has all the details.
Apple had to do that because the iPhone has no lens mount system. People have been sticking lenses, and adaptors, over the existing iPhone lenses for a long time. The feature film Tangerine famously stuck an anamorphic adaptor on an iPhone 5S.
What if Apple had their best wide-angle camera sensor and then a bayonet-style lens mount? Even though Apple used Panavision lenses, and Panavision has the PV-Mount (which stands for Panavision), the industry standard in cinema lenses is the PL-Mount (which stands for positive lock).
Naturally, Apple wouldn’t use an existing lens mount, because they’re Apple, and because they’re using a significantly smaller sensor so there’s no need to be married to a very old mounting system. They should still keep a wide mount in case they ever want to increase the sensor size and use up the whole diameter of the mount. They’d have to name it something. A-Mount was already used by Minolta and Sony.
Because of their cinema aspirations they could always go with APL-Mount. Wouldn’t that be fun? I think it’d be fun, and I’m writing this, so shut up.
It wouldn’t matter what it was named, because like any mirrorless camera you can get the lens glass as close or as far away from the sensor as you want to with lens mount adapters that can be as simple as threaded metal tube, or can carry electrical signals over metal contacts around the perimeter of the mount for lens information, or to control the lens itself.
What if Apple had custom auto-focus prime lenses that could be driven by that iPad Pro over in video village? Something that worked in concert with the other controls instead of doing what they did in the Spring Event and manually pull the lens focus.
Anyway, that seems extremely unlikely to happen. Like hell-freezing-over unlikely. The vlogging camera seems marginally more attainable as fan fiction.
Hell, I’d love an electronic viewfinder, but that’s harder to justify than a screen that can be looked at from a few feet away to check framing. That’s definitely more of a photographic concern. I’d just as soon rule it out completely, but maybe Apple can sell an EVF that hooks onto a hot shoe like Leica does. For the real dorks.
Apple is doing just fine selling iPhones, and it’s easy to argue that every time Apple sells another device it’s a “distraction” but if that were true then they’d only sell iPhones. They’ve got to be able to delegate something like this to a group in Apple. Riffle through parts bins and make one of these every 2-3 years.
I’m hard pressed to think of a reason why Apple wouldn’t benefit from selling dedicated cameras though. I already mentioned that they’re accessories that tie into Apple’s far more expensive devices. As an accessory it also helps take the strain off the iPhone needing to do everything all by itself —which is increasingly important as the device gets thinner and camera bumps get weirder.
As a producer of entertainment Apple would also stand to benefit. Imagine a show shot entirely on Apple cameras, without janky lenses on top of iPhone lenses. Apple could array a few cameras and have a stereoscopic rig, or stitch a bunch together for some immersive video —all on Apple hardware. Think of the press releases they could make about that. The adulation. The nitpicking.
Give us a treat from the parts bin, Tim. Just a little, itty-bitty accessory of a camera.
Category: text
I’m still not entirely sure what happened, but I got a message from Glenn Fleishman the other night asking if tvOS 17.5.1 disabled the YouTube screensaver that I wrote about here and mentioned again over at Six Colors.
The short version is that the YouTube app on Apple TV started showing a slideshow when the TV was paused before the Apple TV’s screensaver kicked on at the default five minute mark. The slideshow from YouTube would function as the screensaver. The slideshow consisted of either the thumbnail of the video you were paused on zooming in, over and over, or a random assortment of still frames from landscape, nature, and drone footage if you happened to be outside of a paused video.
To test what Glenn was reporting I went to my living room Apple TV, which is still on tvOS 17.4. No YouTube screensaver. I asked around a few places and people reported that they don’t have the previously reported screensaver behavior now either.
This isn’t something tvOS did (it obviously wouldn’t be in 17.4) and it didn’t appear to be something YouTube released an updated app for. My best guess is that they have a server-side control for the screensaver behavior and they turned it off there. I don’t have any way to confirm that, but it seems like it’s what fits the facts.
YouTube was very proud of where things were going with screensavers, where they were starting to serve ads in a pilot program. Philipp Schindler, Google’s Chief Business Officer, on the 2024 Q1 investor call:
In Q1, we saw strong traction from the introduction of a Pause Ads pilot on connected TVs, a new non-interruptive ad format that appears when users pause their organic content. Initial results show that Pause Ads are driving strong Brand Lift results and are commanding premium pricing from advertisers.
Fun.
So what happened? What made the screensavers go poof all of a sudden? Is this a temporary reprieve as something about the screensavers gets retooled, or is this permanent? Is it because someone from Apple called someone from Google and got this quietly killed only on Apple TVs?
I don’t want too jinx it, but I’ve set all my Apple TV screensaver intervals back to the default 5 minutes, instead of the 2 minute workaround I was using before. Hopefully this will never come up again.
Category: text
I’m not a fan of video generators. That’s not to say that I hate “AI” being used in video, it can be a powerful tool, but I hate these things that are not tools.
Sora, and Google’s recently announced Veo, produce RGB pixel output from the videos they were trained on, which seems to be a lot of stock footage.
They don’t make elements, or assets, bits, or pieces, they bake the whole shebang flattened out into a rectangle. If you don’t like that completed video, you can keep on prompting but there’s no file for you to open and tweak.
It’s no surprise that working with the output of these video generators is like working with stock footage. There’s nothing shameful about using stock footage, that’s why it’s there. You can buy explosions, fire, dust hits, blood squirts, wooden debris falling, arc welding sparks, rain, aerial views of cities, etc.
A person has to decide how to combine those elements, which is where Sora and Veo come in. It decides how to combine the stock footage it was trained on, and flatten it out to a new result.
Sometimes the results are uncanny, but they’re only for that one shot. Sometimes the results are warbling bodies, and boiling fingers. A lot of the time, it’s in slow motion.
Air Head made by Shy Kids was released a month ago. The people at Shy Kids are clever because they know what would be the most difficult thing to do (having a consistent main character with facial features, expression, and lipsync) so they make the head a featureless balloon. However, they can’t even get Sora to give them the same yellow balloon, attached to the same head, and scaled proportionately to the body the same way. That’s why it’s like a found footage documentary.
Instead, continuity is imposed upon this montage of Sora output. Shy Kids had to slice and dice these individual videos to force even a little consistency in there, even though the only thing credited at the end of the video is Sora.
Here’s a video breakdown released by Shy Kids, on their YouTube channel, not on the OpenAI channel where Air Head is:
“How do you maintain a character, and look consistent, even though Sora is very much a slot machine as to what you get back?” — Walter Woodman Director at Shy Kids
You didn’t, Walter. You cut these shots together back to back that aren’t even close.
Shy Kids followed that first short up with Deflated where they combined live action footage with Sora output and explicitly said that “VFX software” was used.
This looks terrible, on a technical level. If you don’t notice the problems, trust me and watch it again. Look at edges. Look at the neck. Look at the black levels (the darkest darks) and the white levels (check out those posters and billboards).
It also still has the problems from the first video, with the balloon, scale, and attachment point changing wildly throughout, but now there are those matting issues where they’re trying to combine their Sora output with their live action. Sora isn’t a compositing tool, so it can’t insert that footage itself.
In a typical production pipeline you could shoot the reporter and the balloon boy walking down the street together, and then paint out the balloon boy’s head (a relatively trivial matter these days, I assure you). Matchmove the actor’s head and neck, render a yellow balloon asset, and composite it over the painted plate. The balloon would look the same in every single shot and it would match the actor’s performance instead of just wobbling.
They close out the video montage with the reporter teasing an upcoming segment with a doctor. “Why are so many people moving in slow motion? And what’s going on with everyone’s hands?” At least they have some humor about it.
Yesterday, a week after releasing Deflated OpenAI’s YouTube channel released a BTS video from Shy Kids.
They don’t include any botched output from Sora, like they did in the BTS of the first short they released on their own YouTube channel. This time around they show how they stabilize, rotoscope, composite, and color correct the Sora output to fit with the live-action reporter. They also replaced the balloon in a few shots, like the magazine covers.
The most effective uses of Sora in the short are the random montage clips inserted in the “Tough Questions” opener, and the montage of people wearing balloons on a runway. It’s no wonder those are more successful because they’re based on the kinds of input you get from stock footage, and they’re being used in place of stock footage.
What about the aerial shots of the balloon deflating? The first shot worked well, but then they cut to other shots and you could see that it wasn’t the same location, and it wasn’t the same balloon. Sure, they didn’t have to rent a helicopter and gear, but people very rarely do that and instead use … stock footage you can buy online and then put your matching balloon asset on top of.
OpenAI and Google are both selling these video generators as technological breakthroughs in filmmaking. The reality is that it’s artifacting stock footage.
Bad clients, and bad producers will tell their editors to put Sora or Veo output in to the initial edit, then they’ll turn to a VFX house and say that the shots are “90% there” and they “just need someone to take it across the finish line.”
How do I know this? Because that happens with stock footage and weird composites and retimes that editors make in Avid when clients want to have something in the edit so they can figure it out. Even if the client agrees to replace it, they can get married to how the stock footage or temp looked, or how it was timed (remember that playback speed is a factor).
That’s why these companies talk about how it empowers directors, and small-scale productions. Until they want to change something just a little bit.
As a VFX artist I’m all for AI/ML to improve work, but that kind of improvement is in better tracking tools. It’s in better keying and extraction tools. It’s in generative fill for paint purposes. It’s in screen inserts for funky monitors and round rect smart phones. It’s in better photogrammetry with gaussian splatting, radiance fields, etc.
Tools are parts of a pipeline where you can adjust individual elements upstream, or downstream without starting something over from scratch because it isn’t just one flattened hallucination. It’s not a piece of stock footage.
Foundry has a tool called CopyCat, where you can train up a model on repetitive tasks, as explained here:
In this example for removing tracking markers from a stack of green paper cards by only painting 5 of the frames to train the model.
Here’s Foundry’s brief explanation of Cattery, a library of models ready to use with CopyCat:
This is how you use machine learning to do filmmaking. These aren’t for making choppy montages.
New breakthroughs in stock footage are being misrepresented by OpenAI, and Google, hungry for investment, and sweaty to establish their product before too many people ask questions. It repulses me as a VFX artist, and it should repulse you too.
Category: text