This story is being highlighted as one of Gamasutra's best of 2014.
Yesterday Samsung formally revealed its Gear VR headset, which was developed in close partnership with the folks at Oculus VR. When it debuts later this year, it will be the first of the new crop of VR headsets sold explicitly to consumers -- and the first opportunity for developers to sell their VR games on a mobile app store.
The headset is powered by Oculus VR's Mobile SDK, a variant version of the company's toolset that plays nice with Samsung's upcoming Galaxy Note 4 smartphone, which slots into the Gear VR headset to drive it. The phone is itself powered by Android 4.4.4 ("Kit Kat"), and Oculus CTO John Carmack told me today during a brief phone interview that his experience building a VR toolset for Android -- his first Android project -- has been frustrating and full of surprises.
I spoke to Carmack (alongside Oculus VR mobile VP Max Cohen and company founder Palmer Luckey) in an effort to better understand Oculus' plans for mobile VR, and we wound up discussing some high-level technical details that mobile-curious VR game makers should find useful when Oculus releases its Mobile SDK later this year.
Thus, I've taken the liberty of transcribing our conversation, editing it for brevity's sake and publishing it here.
Tell me more about the Mobile SDK, and how it differs from the extant Oculus SDK from a developer perspective.
Carmack: Okay, so weave had this project going for about a year. We pushed our first version out to a select few developers only a few months ago, and we’ve had two revs since then. Itas still pretty rough; weare working to get our public SDK done soon, before the device is actually available.
The core Oculus VR functions still exist there, and you can use them for fetching the sensor interface, but thatas basically all you use the core, original Oculus functions for. The mobile side has a simplified interface, basically ‘Enter VR modea with a bunch of initialization parameters like what locked clock levels you want to use, what timewarp swap options you want, and then ‘Leave VR modea which shuts everything down.
[Note that the Oculus Mobile SDK's "asynchronous timewarp" feature essentially keeps the display updating at 60 fps and adjusting what's displayed on-screen based on your head rotation, even if the game itself isn't updating at 60 fps. Gear VR developer E McNeill explains it better in a recent blog post and claims it makes a huge difference for VR development, especially when it comes to eliminating motion-induced nausea.]
90 percent of the developers are using Unity right now, and the Unity integration doesnat change too much from the PC side -- we have people who are building for both platforms, there. But those who want to do more significant stuff or things that are more platform-specific will be doing more native code work, or plugin work, which requires interacting with the not-so-pleasant Android development experience.
I donat think anyone is going to say great things about being a native developer on Android, but
"Brace yourself: Android setup and development really does suck. Itas no fun at all."
at least the plus side here is youare targeting a single device. Itas sort of like a console in that respect: you donat have to worry about the bazillion different Android configurations, broken OS versions, or anything like that. We have one target right now, which is lovely; itas a saving grace for this. But developing in Android, thatas no fun at all.
Most of us went actually through that for our first time as part of this process, and it pained me a lot; for a long time, it was just me working on the main software stuff, and I hadnat done an Android project before I started on this. I still worry that weave done something fundamentally wrong, but we've had real Android developers take a look at everything now, and it doesnat seem like anything is terminally broken.
But weave been rushing to get this functional, and while we have a ton of developers already working on this stuff, itas...Iam not going to try and claim this is the most elegant piece of software Iave ever written. Iave been on a mad dash the last few weeks to try and clean things up as much as possible, but itas not going to look like an Apple-quality SDK when people first get their hands on it.
Cohen: And this is by no means a shift in focus away from PC. One of the great things about the mobile development process is that asynchronous timewarp, for instance, has been implemented on mobile, and thatas going to make its way to the PC SDK as well. So designing within constraints and making a highly-optimized VR experience is something thatas useful, even if you have the most high-end PC in the world. What weave learned from the mobile development process will pay dividends for developers everywhere.
Carmack: And since most of the developers are using Unity right now, everybody that makes a Unity app for mobile should be able to click a button in Unity and get a PC version out of it. Going the other way, which most developers have done so far, definitely involves re-analyzing some of your choices and figuring out what needs to be done to get your game running on the less powerful platform.
But this will be the first place where VR enters a market, with a store and an ecosystem where you can go out and target customers. So I think for actual developers, rather than just enthusiasts, that initial market is going to be very important for them. Much will be learned from that.
PC versions will come out of that almost automatically, and then developers will gravitate to whatever platform best suits their interests. People that are into big high-end extravaganzas obviously arenat going to be doing that on Note 4 hardware, so theyall want to target PC. Anything that makes use of positional tracking also has to be done on PC, because weare not going to have that on mobile for a while -- until we pull some rabbit out of a hat on that.
But I think a lot of developers are going to be thrilled by this, because one of the greatest things about it is that you can load something onto this and walk around with it. Just go somewhere and drop it on peopleas heads, without having to lug your PC VR system around. So it does have a lot of that approachability and kind of impulse appeal of mobile development and deployment.
Also, itas important to note that the Note 4 hardware is way, way better than DK1.
So what did you learn from your first Android development outing, and what should developers know about moving into mobile VR development?
Carmack: Okay, thereas the normal hell of moving to a new platform -- and I gotta say, Android was more hell to move to than most consoles Iave adopted. Just because of the way Google has to position things across a diverse hardware spectrum, and because Google still doesnat really endorse native code development -- theyad still rather everyone worked in Java. And thatas a defensible position, but itas certainly not what you want to be doing on a resource-constrained VR system.
So brace yourself: Android setup and development really does suck. Itas no fun at all.
The other thing that was a real surprise to me was how important the power management was,
"This will be the first place where VR enters a market, with a store and an ecosystem where you can go out and target customers. Much will be learned from that."
not just for battery life, but for performance. Because if you go ahead and fire up all four cores at the top clock rate and draw a bunch of stuff, you overheat within minutes. If you really load everything onto the system that can draw power, you will probably thermally throttle down in under a minute. So you literally just canat use all the capabilities of the system at the same time.
So that became one of the critical things that, honestly, weare still figuring out how to manage in the VR process. One of the great concessions that Samsung made on this, which was huge, is we have this option to set up locked clock rates for VR. And you can choose between this matrix of CPU and GPU values, though you canat just say ‘give me the fastest GPU and the fastest CPU,a thatas not an allowed combination. But you do have the option to say -- and I think many VR games will say -- ‘I am graphics-heavy; I want a top GPU speed, but I can tolerate a very low CPU speed.a
Also, Unity fixed its multi-threaded renderer on Android, which hadnat been working until we kind of prodded them to get it fixed. So we do expect Unity apps to run multithreaded because they are heavyweight, and that does usually mean that if youare doing anything at all in Unity, itas going to be two fairly high-clocked cores, and thatas going to hurt you when it comes to power consumption and thermal issues. So you canat usually choose to be a Unity app and have the graphics turned all the way up, and that has implications on what developers can do here.
And I expect the UE4 integration to be somewhat more efficient -- just because it doesnat have the interpretive scripting and stuff, it has knowledge about stereoscopic rendering beyond just rendering two independent views -- but almost all of our developers right now are currently Unity-focused. I expect there will be some migration to UE4 as that becomes available. But still, any big game engine thatas designed for PC will likely have CPU usage thatas way higher than what you want on mobile. Another aspect of that is, on PC or consoles, you optimize until your game runs the way you want and then youare done. In mobile, you have the choice of CPU clock settings and how much power you draw, so you have the option of saying ‘well, it works until I run it on the highest clock rate,a and then you have to decide: continue optimizing to try and get your game to run at a lower clock rate and lower power settings, or be okay with making a game that might be awesome and great and fun, but it overheats after fifteen minutes.
How can developers work around that?
Carmack: Our plan right now is, if you do run things hard and the hardware eventually hits a thermal limit, your game can be notified and we can pop into our global UI. We havenat decided exactly what the messaging is, because of course Samsung hates the idea of terms like ‘throttlinga or ‘thermala, but something about ‘the system is now resource-constrained,a and the player has the choice of saying ‘well, I’'m going to end my session here,a or they can put it down to the low-power settings if your app is capable of adapting to them.
The power issue is going to be a big battle, and itas going to be interesting to see because we donat know yet how many people will like smaller nuggets of entertainment experiences instead of longer-form things. Weare hoping developers make a lot of short, bite-sized nuggets of entertainment because they work well on mobile -- they donat tax the thermal and power so much, and they also have another benefit: since we donat have positional tracking on mobile, we know weare going to cause problems for more sensitive people. So thereas a class of people that might feel comfortable in VR for fifteen minutes, in our experience, but if you sat them down and asked them to play for more than an hour it might be more of an issue.
Thereas a whole new crowd of game developers that are doing interesting PC titles with relatively modest budgets and development timeframes, that have focused really strictly on the creative aspects. And itas great theyave been able to do that, but if they want to play in the mobile VR space they need to adjust how they work. Maybe tap some old-timer to teach them the ropes about actually optimizing when CPU and GPU matter in a way that, in some cases, they really havenat on the PC for a while.
Cohen: Can you imagine John Carmack with like, a long white beard and a classroom full of young developers in 15-20 years?
Carmack: I have to say, I think I'm making the transition from wizkid to graybeard pretty well!