Katsumi Kuwayama, Camera Development Expert

CMOS Sensor Module found in a complicated system — System designers are getting more and more needs to embed cameras to their systems, and such cameras are improving performance along with a variety of functions and performances. Historically, significant high skilled engineering resources were required to develop electronic systems with high performance cameras. Due to the increased need for embedded cameras, there is now a shortage of the high skilled engineers needed.

In this whitepaper, we will examine a new approach to support significantly reduced engineering resources than previously required to embed high performance cameras into systems

Historical design cycle used

I have led a team to develop Image Signal Processors (ISP) while supporting some of the major smartphone suppliers worldwide, where I found my customers went through the whole following design process by themselves:

Analyze the latest CMOS image sensors
Define camera hardware requirements as well as software ones
Manage ISP firmware development with their ISP supplier
Manage CMOS camera module (CCM) development with their CCM supplier
Set up proof of concept within final product architecture
Set up a platform to develop system software working with the new camera system
Develop prototypes and production line

Engineers needed to follow this process from the very beginning and if the performance of the selected image sensor and/or functions are complicated, the process duration gets much longer.

Tools enabling today’s design paradigm shift

THine proposes a new approach to solve the shortage of the engineering resources for camera development by introducing several kits needed in camera system development lifecycle.

Based on the long time experiences supporting customers worldwide, THine believes the design cycle of camera system development can be categorized as follows.

Stage 1. Show the concept of the camera system works with “Proof of Concept”

Stage 2. Create image quality and camera functions

Stage 3. Embed the camera into entire system

Requirements for each of these stages can be described as follows.

In the first stage, Proof of Concept should be developed in time with minimal engineering resources for company management to provide design approval. Sometimes management expects to see a working concept within a few weeks instead of months.

In the next stage, image performance and camera functions should achieve the specification and criteria to implement the concept demonstrated by Proof of Concept. It needs deep experiences of technology and engineering in the camera/image world. Sometimes you are asked to make “hotter red” where you need not only to interpret what ‘hotter’ means and how to make it happen on your PC screen.

In the final stage, the developed camera system with well-tuned image quality should be embedded into the entire system, which is usually based on Linux based SoC system. It needs another deep knowledge in engineering of embedded systems.

THine can support engineers to go forward along with these three steps by providing three tools.

THSCU101, a 13M pixel PDAF UVC camera, supports to develop Proof of Concept for variety types of camera systems. It supports not only precise images of 13M pixel or 4K2K but also 1080p with higher SNR images with the same camera module.

Once the Proof of Concept got the approval of management, then THine offers Camera Development Kit (CDK) to implement the camera functions and image quality including, color reproduction, auto white balance under several lighting condition, lens and led shading compensation, a variety type of auto exposure controls required by applications and JPEG compression, MIPI CSI-2 and parallel video interfaces, resizing, cropping and zooming, framerate control, noise reduction, edge enhancement control, and so on.

Finally, THSCM101, Linux camera for i.MX 8M EVK supports to embed the camera into Linux based embedded system easily by utilizing Linux camera driver for THP7312, Camera Processor. (note: THSCM101 is the first product in Linux camera series and it is dedicated for NXP i.MX 8M EVK. Products supporting other Linux based processors are planned)

In following sections, we will see some highlighted features of each kit.

Stage 1: THSCU101 for Proof of concept

You can easily develop Proof of Concept with THSCU101 UVC camera. It reduces the time and engineering resources to develop the camera needed for your system.

There are two type of the camera functions supported by THSCU101 UVC camera, one is UVC compatible function and the other is extended optional function. Please refer to following table.

Stage 2: CDK for Image Creation

There will be several paths for the final product development.

Case 1. Embedding THSCU101 as is into the customer’s system.

Case 2. Embedding THSCU101 hardware as is into the customer’s system and tune some image quality using THine’s Camera Development Kit, CDK.

Case 3. Customizing THSCU101 hardware for the customer’s system and tune some image quality using THine’s Camera Development Kit, CDK.

CDK is a GUI based easy to use camera development environment consisting of following 3 parts.

1) EVB - Camera development hardware including frame grabber and THP7312, THine’s Camera Processor.

2) THine Tuning Tool - GUI based PC application to tune SDKs for THP7312.

GUI tools example for AE related features

3) SDK - THP7312’s firmware development environment which include all the components need to create camera system.

All the algorithms and functions are provided as pre constructed binary library in the SDK. You can customize the image quality and camera functions via GUI based THine Tuning Tool and APIs directly control the SDK.

THine has variety of sample code of image sensor drivers for many image sensors for wide application areas. THine also supports many optional functions suit for AR/VR, drones, medical, identification, surveillance, education, automotive and others.

Stage 3: Integration of camera system

Unless the final system is standalone camera, after validating the image quality and camera functions with CDK, there is only one remaining work to complete: integrate the camera into the system.

THine believes that a part of the most time consuming portion of camera system development is this integration of the camera with variety of SoCs in the systems.

THSCM101 is the first product in Linux camera series and it is dedicated for NXP i.MX 8M EVK.