TYZX Patent Summary

TYZX has been awarded patents for technology that is critical for low-latency stereo vision, object- and person-tracking, and guidance for robots and other unmanned systems. The table below summarizes the company’s U.S. patents.

U.S. Patent 6,188,777 - Method and apparatus for personnel detection and tracking

Priority Right  Aug. 1, 1997; Awarded February 13, 2001

Summary
This patent covers the use of 3D data for object detection and tracking, in combination with the use of appearance cues (color, face features, height, etc.) and segmentation methods. The patent also covers classifying individual people based on characteristics of range, color, and pattern.

Relevance
Real-time 3D data is very useful for segmenting and tracking people. One of stereo vision’s advantages over other sources of 3D data is that it provides an intensity image that minimizes false positives and makes tracking more accurate.
3D data can be augmented with 2D data to improve the richness and accuracy of person-tracking applications. Classical 2D methods can continue to play a role in person-tracking applications – for example, in classifying an object as a person by detecting facial features on the person’s head.

Back to top

U.S. Patent 6,215,898 - Data Processing System and Method

Priority Right April 15, 1997; Awarded  April 10, 2001

Summary
This is a broad patent with 40 claims addressing parallel methods of computing stereo range. The patent addresses both functional parallelism (reading pixels in parallel with computing correlation window transforms in parallel) and low-level parallelism (i.e., generating and scoring multiple disparities in parallel). It also covers the computation of sub-pixel disparity. While TYZX uses the Census transform for its stereo vision products, the patent covers parallel processing of dense stereo regardless of the transform algorithm used.

Relevance
Many aspects of this patent are critical for using stereo vision for real-time data generation. Some of the key points are:

• Parallelism for speed
  Stereo correlation is a computationally expensive search process. Parallel processing reduces system latency and increases frame rate.
• Parallelism for efficiency
  By exploiting low-level parallelism and making use of partial results, a stereo vision solution can be implemented efficiently in hardware with a reduced gate count and economical power requirements.Parallelism for accuracy
  A stereo vision system can achieve more accurate results through concurrent left-to-right and right-to-left consistency checks, searching a larger number of disparities, and computing sub-pixel disparities.

Back to top

U.S. Patent 6,445,810 - Method and Apparatus for Personnel Detection and Tracking

Priority Right Dec. 1, 2000; Awarded September 3, 2002

Summary
This patent covers executing an application in accordance with the presence of an individual identified through an identification system which includes stereo vision-derived 3D data, tracking data, and an ID system which uses 2D image characteristics of the tracked individual.

Relevance
Interactivity includes the notion of taking action. In this case, taking action can mean running programs that alter a person’s appearance while leaving the rest of the scene unmodified (e.g., blurring the individual’s face), or that target messages or merchandizing to a particular individual.

Back to top

U.S. Patent 6,456,737 - Data Processing System and Method

Priority Right April 17, 1997; Awarded  September 24, 2002

Summary
Related to Patent 6,215,898, this patent’s claims address latency challenges when processing stereo images by beginning to generate stereo disparity results before the complete left and right images are received.

Relevance
Processing latency is a critical performance factor in real-time control systems, such as gesture control systems in interactive games or computer-controlling braking systems in cars and unmanned systems. Increasing latency degrades system performance. Games become unresponsive, and vehicles require longer stopping distances. By beginning image processing as soon as pixels become available, this critical processing delay is minimized.

Back to top

U.S. Patent 6,661,918 - Background Estimation and Segmentation Based on Range and Color

Priority Right December 4, 1998; Awarded December 9, 2003

Summary
Covers modeling a scene using 3D data – from any source not just stereo vision – and intensity data (B/W and color images), and then comparing new frames of data to the model to detect changes in the scene.

Relevance
3D data is valuable for detecting the size and location of objects in a scene. Tracking people and objects, however, imposes additional requirements, such as analyzing and following motion.
Background models are a very popular mechanism for detecting changes in a scene. The simplest form of background-model analysis compares pixels from two sequential images. Groups of neighboring pixels whose values have changed beyond some threshold are a strong indicator of motion in the scene. In some situations, however, background models may lead to a “false positive” detection of motion. For example, lighting changes can cause the entire image to appear to be in motion. Shadows might also be perceived as motion, leading to a false detection or “false positive”.

With a 3D range-based background model, instead of detecting a change in a pixel’s intensity value, we are detecting a change in a pixel’s 3D value – such as its distance from the camera. A change in lighting will alter the appearance of a tree, but its 3D value will remain the same, avoiding a false positive detection of motion. Similarly, a shadow cast on a wall will change the appearance of the wall, but not its 3D location; again no false positive.
The best solution is to combine approaches, i.e., detect both a pixel’s intensity and its 3D value (such as distance). This patent’s methods do just that and make scene modeling highly robust.

Back to top

U.S. Patent 7,003,134 - Three Dimensional Object Pose Estimation Which Employs Dense Depth Information

Priority Right March 8, 1999; Awarded February 21, 2006

Summary
This patent covers several methods for estimating the pose of an articulated or non-articulated figure from dense 3D data from any source (not just from stereo vision).

Relevance
One of the most interesting uses of real-time, low-latency dense 3D data is determining the pose or positional relationship of an object to a data source. For example, you can use real-time 3D data to determine the pose of articulated objects like people. Video games are an excellent example of applications that need to determine a player’s pose.

Back to top

U.S. Patent 7,158,656 - Three Dimensional Object Pose Estimation Which Employs Dense Depth Information

Priority Right August 1, 2005; Awarded January 2, 2007

Summary
Related to patent 7,003,134, this patent describes additional methods for estimating the pose of an articulated or non-articulated figure from dense range estimates.

U.S. Patent 7,317,830 - Background Estimation and Segmentation Based on Range and Color

Priority Right November 7, 2003; Awarded January 8, 2008

Summary
Related to patent 6,661,918, this patent describes additional methods of creating and updating background models using range and color information.

Back to top

U.S. Patent 7,567,702 - Data Processing System and Method

Priority Right April 15, 1997; Awarded  July 28, 2009

Summary
This patent includes claims for providing additional data integrity checks during parallel stereo correlation, as well as claims pertaining to the use of low-latency stereo correlation results.

Relevance
Low-latency, real-time stereo vision processing accomplished via parallel processing enables computationally complex applications such as robot guidance, scene compositing based on distance, and Z-keying (a form of green screen that uses distance rather than color to distinguish figures and objects in the foreground from a background image that can be replaced; movie studios can use Z-keying to make scenes shot in a studio appear to be scenes shot on location).

Back to top

U.S. Patent 7,664,315 - Integrated Image Processor

Priority Right November 3, 2004; Awarded February 16, 2010

Summary
Patent 7,664,315’s claims address the integration of additional functional units with stereo correlation to further improve performance through greater parallelism and efficiency via close integration. These units perform operations frequently required when working with stereo vision such as image rectification, background modeling, and projections of the 3D data.

Relevance
Paraphrasing Amdahl’s law, “you are only as fast as your slowest operation.” While the previous patents address making stereo correlation fast, additional system performance gains can be made by performing other required operations in parallel. This increased parallelism provides two additional benefits. First, tighter integration makes implementations more efficient. Second, more efficient data representations (such as 3D data projections) can reduce processing time in subsequent operations, such as tracking.