Interview with Joshua Wise

Smithers Apex: Your talk focuses on the dataflow within imaging systems, what are the current challenges and limitations you are working to overcome?

Joshua Wise:  Broadly, the biggest challenge that I see facing me in the automotive imaging ecosystem is the transition.  Design cycles, component interdependencies, and device lifetimes: these are all elements that have a strong influence from the mobile imaging world, and each of these will have to be reconsidered as we migrate to an automotive-first environment.

The mobile ecosystem, for the most part, was comprised of three interacting entities: the sensor vendor, the SoC vendor, and the integrator.  The automotive ecosystem has many more moving parts; a serdes is just one more component, but isn’t the only thing that changes.  As more components enter the datapath, we start to see cascading effects — new HDR formats bumping up against protocol limitations in a serdes, system bandwidth availability, and new components that must transmit and pass along system safety metadata. 

 Smithers Apex: As automotive sensors become more sophisticated what role do you expect deep learning to play in improving vehicle performance and driving experience?

Joshua Wise: Deep learning has the potential to transform almost every part of a vehicle’s behavior, from improving sensor performance, through controlling “piloted driving” experiences, all the way to the very core functionality of the vehicle.  Every single day, more new innovation comes in the field of deep learning, and NVIDIA is proud to have the GPU at the heart of that work.   

I’m particularly excited about the way that deep learning has the potential to take on more and more workloads that we previously delegated to fixed algorithms.  These “end-to-end” models have shown incredible performance on applications like speech recognition and synthesis — and our team has recently demonstrated an end-to-end deep learning model for a self-driving car, which runs on a single DRIVE PX ECU.

 Smithers Apex: Where do you see the biggest near term opportunities in the automotive sensor market?

Joshua Wise: The operating environment for automotive sensors is very different from any of the usual environments that we are traditionally used to.  Automotive sensors have a level of quality that they must meet, and below that level, safety-critical algorithms begin to fail, and the user’s experience is dramatically degraded.  As near as I can tell, the challenge of maintaining image quality in extremely challenging operating environments (perhaps low light at high operating temperatures — think “desert driving at night”) presents a great opportunity; new noise-management solutions will greatly extend the “long tail” of conditions that automotive sensors can be used in.

Smithers Apex:  Your talk will include recommendations on compatibility – what are the key standards issues need to be addressed as components get more complex and diverse?

Joshua Wise: There are two that are top on my list right now.  The first that comes to mind is safety and compliance: an important issue in the automotive environment is the ability to self-diagnose issues.  In short, the system must “know when it doesn’t know”.  As more components enter the ecosystem, there is more opportunity for data to be damaged in transit — and, similarly, more components result in more health data that must be aggregated and transmitted.

The second on my list is the need for a standard in transmitting data with a high dynamic range.  There are as many implementations of sending pixels with greater than 12 bits of data as there are vendors right now — perhaps even more!  We’ve been working with vendors to come up with solutions that work with both modern and legacy components, but we see an opportunity to unify and standardize here.

There are, of course, a handful of other issues, too.  For instance, as more cameras attach to the system, more configuration data needs to be carried, which leaves a performance hole for I2C.  I'll talk some about all of these, and I’ll also talk some about procedural recommendations for ensuring that new features are well-supported throughout the entire data chain.

Smithers Apex: What are you most looking forward to about the conference?

Joshua Wise: I’m excited to get the opportunity to build relationships with partners in the image processing community, from the image sensor side and the side of system integrators (and everywhere in between!); IS is a conference with a rich lineage, and I look forward to the opportunity to participate this year.