A vari-focal or multi-focal plane head-mounted display (HMD) technology is one of our most recent display projects to address a fundamental problem inherent to the existing stereoscopic display technologies. In a typical stereoscopic display, the eyes of a user are forced to accommodate to a fixed focal distance while changing their convergence angles to view 3D content rendered at different depths. Existing stereoscopic displays are therefore incapable of correctly rendering focus cues and force the decoupling of the accommodation and convergence cues which are naturally coupled in viewing a real-world scene. Many psychophysical studies suggest that this fundamental problem likely contributes to the perceived depth compression phenomena and visual fatigue when using stereoscopic displays. We recently explored an elegant method to design an HMD with addressable focal planes. The focal plane of the display is dynamically controlled, matching the depth cue of virtual objects with the focal distance of the display.

We further investigated a time-multiplexed multi-focal plane approach to achieve the capability of rendering pseudo-correct focus cues for a volume of 3D objects. The virtual objects therefore can be rendered with correct or near-correct focus cues from optical infinity to the near point of the eye, analogous to the naturally coupled convergence cues. A pilot study suggested improved depth perception and correct accommodation response. Continuing research is being carried out to develop prototypes which will be used to perform studies to understand a variety of visual artifacts found in stereoscopic displays.

Polarized Head Mounted Projection Display  (p-HMPD)

In existing optical see-through head-mounted displays (HMD), it is a common challenge that the displayed image lacks brightness and contrast compared to the direct view of a real-world scene. Consequently, such displays are usually used under dimmed indoor lighting conditions, excluding the feasibility of applying such information displays outdoor or in scenarios where well-lit environments such as in an operation room are demanded. The lack of image brightness is further aggravated in the design of a see-through head-mounted projection display (HMPD) due to the multi-pass beam splitting scheme and low retro-reflection efficiency. The existing designs of HMPDs typically leads to an overall light efficiency below 10%. We propose a simple but elegant design of a polarized head-mounted projection display (p-HMPD). By managing the polarization property in the optical system, the proposed p-HMPD design can greatly improve the image brightness (by about four times), contrast, and color vividness of a conventional HMPD system.  This makes it possible to use the p-HMPD system in a well-lit indoor environment. The first prototype showed on the left was a VGA-resolution design with 55-degrees of diagonal field of view. We recently completed the a prototype based on a pair of XGA-resolution FLCOS microdisplays. The prototype was shown on the left as well.

Click here to learn more

Hybrid-SCAPE

With the ever-increasing scope and volume of data available for exploration and the increasing complexity of tasks, many of which require a team of experts working together, there is great need for 3D visualization systems that are capable of displaying a complex dataset in ways that facilitate information discovery and collaborative activities. While 3D displays are key enabling technologies to a fully-integrated, 3D interactive system, visualization methods and interaction techniques are essential components that have a critical impact on the effectiveness of the system for complex tasks. Visualization methods support users to correlate and interpret complex datasets by exploring alternative perspectives, scales, resolution, and temporal dynamics. Interaction techniques allow users to manipulate digital objects, perform system control, and add or modify digital information. However, most of the existing 3D visualization methods are incapable of adaptively visualizing data at appropriate levels of complexity according to a user’s task needs. The interaction techniques based on conventional interface devices such as joysticks and 3D mice lack intuitiveness and ease of use. Moreover, few existing visualization systems support and mediate fluid collaborative activities.

To respond to these challenges, under a recent NSF CAREER award (Years 2007-2012), we have been developing a heterogeneous display environment that contains a large-scale immersive environment, a 2D/3D hybrid display in which a 3D display area is surrounded by a high resolution 2D tabletop display, and various other 2D and 3D hand-held displays. The system is and will be integrated with an array of 2D and 3D user interaction techniques such as multi-touch sensing, hand gesture recognition, and other advanced user interfaces. From the display perspective, this platform assists users in navigating and controlling their levels of immersion into the digital realm. From the interaction perspective, the platform supports intuitive user interaction with both the physical and digital worlds through physical manipulation and gesture-based interaction metaphors. Furthermore, this platform provides a unique display environment for studying visualization and interaction techniques that are potentially capable of facilitating users’ ability to correlate and understand complex datasets as well as to support collaborative tasks. For instance, we recently evaluated the effect of three multi-scale interfaces in large scale information visualization. The comparison study is helpful to understand how information window arrangements and methods to position information relative to a viewer affect the user’s ability to gather information and interpret spatial relationships.

This project is based on another earlier project: SCAPE

Design of a hybrid-WIM workbench display

Multi-Touch User Interface

We recently are exploring tangible Magic Lens interfaces in 3D augmented and virtual environments, to augment the bi-modal (micro- and macro-scales, exocentric and egocentric perspectives) visualization concept in SCAPE.

Click here to learn more......

Fovea-Contingent Display

Fovea-contingent display (FCD)  is another technology that we have been pursuing. Unlike conventional display technologies in which the finite number of pixels is spread evenly across the entire visual field, this technology adopts an active foveation approach which mimics the human visual system. In this approach, a user’s instantaneous attention is dynamically tracked and is engaged with high-resolution information in a narrow visual field which is embedded into a wide, low-resolution peripheral visual field to provide adequate context awareness.

FCD technology can potentially find numerous applications in 3D visualization, human computer interaction, and vision research. For instance, this technology can be used to develop techniques enabling physically-challenged individuals to interact with and access digital information

Click here to learn more......

Fovea-Contingent Imaging System

In the process of creating a mixed- or augmented reality application, it is imperative to capture and interpret the physical world, especially as the world becomes more dynamic or complex. Image acquisition systems are an essential component in the loop of capturing a real-world scene and augmenting it with synthetic information. The capability of acquiring high-resolution, wide field of view and high dynamic range images at video rate is highly desired. Such capabilities are also critical for many vision-based applications such as surveillance and robot navigation. The field of view, resolution, and depth of field of a conventional digital camera, however, are limited by the choice of imaging sensors and optics. We have been developing various novel imaging systems to achieve the aforementioned capabilities. One example of such an imaging system is a foveated imaging system, which mimics the human visual system. A wide visual field is captured with a dynamically embedded, high-resolution fovea region: the peripheral sensor captures the context for target detection and tracking; the foveated sensor, with a resolution many magnitudes greater than the peripheral sensor, captures the fine details for target recognition and detail examination. This acquisition method not only serves as an input for the fovea-contingent display research discussed earlier, but can also find a wide range of applications in surveillance, robot navigation, and microscopic imaging.

Click here to learn more......

Modeling and Simulation of Eye Illumination in Eye-tracked Head-Mounted Display

This project aims at the optimization of illumination schemes for the development of a robust eye tracking system. Based on the Arizona Eye Model, we have been modeling a complete eye illumination and imaging system which includes the modeling of the optics of the human eye, IR illumination methods, and an eye image capture system. Using this simulation, we are able to study how to optimize illumination schemes to achieve good quality of IR-illuminated eye images for the development of robust eye tracking methods. In particular, we are able to study how the quality of eye images would change as eye rotates.

• Eye model in LightTools

  • Arizona eye

  • Arizona eye with structure

• Simulated eye features in LightTools

  • Simulation Setup

  • Single glint simulation

  • Two glints simulation

  • Four glints simulation

  • Optimized illumination (Simulation, Real image)

Click here to learn more......

Robust Eye Tracking Methods

This project aims at the development of robust and accurate 2D/3D eye tracking methods. We are particularly interested in eye tracking solutions that are suitable for the integration with head-mounted displays. We have been developing feature-based eye tracking  methods, using multiple near IR LEDs to  improve the robustness of eye illumination, the accuracy of pupil/cornea tracking, the range of eye movements tracking, as well as the simplification of tracking algorithm for speed concerns. We are also developing 3D tracking methods to obtain point of gaze information. This project has been funded by a three-year NSF HCI grant.

This project aims at the development of robust and accurate 2D/3D eye tracking methods. We are particularly interested in eye tracking solutions that are suitable for the integration with head-mounted displays. We have been developing feature-based eye tracking  methods, using multiple near IR- LEDs to  improve the robustness of eye illumination, the accuracy of pupil/cornea tracking, the range of eye movements tracking, as well as the simplification of tracking algorithm for speed concerns. We are also developing 3D tracking methods to obtain point of gaze information. More details are reported in Prasanna's MS thesis.

Robust algorithm for tracking with artifacts

Recovered eyelid (1, 2)

Eyelash (1, 2)

Extended tracking range (Left, up-left, up, right)

Click here to learn more......

Integral Imaging

Integral imaging is a new method to obtain a three-dimensional (3D) imaging/display by capturing a set of two-dimensional (2D) elemental images with different perspectives. Compared to other 3D imaging and display technologies, InI has some inherent advantages, such as full parallax, continuous viewing points, operating with incoherent light, etc.

Click here to learn more......

Volumetric display and

Light field display

Currently we are in the process of analyzing existing volumetric display systems.  After analyzing systems, a classification system will be developed that enables the categorization of the characteristics of the systems.  By creating a metric system and weighing system that will rank each of the characterizations, we will distinguish priorities.  Ultimately, the goal is to determine if a new display system that renders 3D images in space is a superior technique in comparison to other 3D imaging options. 

The light field displays are an interesting new development in the history of 3D display , a merger of the multiview and volumetric display systems.  These systems possess a display volume for emission, which is similar to the volumetric displays, but a rendering scheme that is geared towards a multiview display.  However, these are not mutiview displays in the sense that the design is not established for specific directional viewing for small windowed regions.  Light field systems produce angular data that have a smaller separation then between human eyes, thus reproducing varying images for each eye.  This 3D reproduction is achieved within a region known as the viewing volume.  The integration of this angular information by the eye is the underlying factor that enables the reproduction of the 3D image.  Our research is to determine if it is feasible to create a new 3D light field display.

SCAPE

SCAPE stands for Stereoscopic Collaboration in Augmented Projective Environments.  SCAPE is a 3D collaborative infrastructure that is capable of blending the traditionally separate paradigms of immersive virtual environment and augmented reality technology into one single cohesive system.

Based on the visualization capabilities of head-mounted projection displays (HMPDs), SCAPE presents two simultaneous perspectives of a 3D synthetic data set: an egocentric view of a virtual world appears on an immersive wall display, while a corresponding exocentric augmented view appears on a stationary workbench display. The workbench gives us an overview of the dataset at low detail, whereas the room display provides us with a first-person view of the data set at high detail, but with a relatively narrow field of interest. Therefore, in SCAPE, a synthetic 3D dataset is concurrently visualized at dual scales and dual perspective, opposed to single perspective, single scale visualization capability in most existing 3D system. 

Different from the CAVE-like spatially immersive display environment, SCAPE supports multiple styles of collaboration among a small group of co-located users, including symmetrical mode where each user can control his or her viewpoint independently, privileged lead mode where one of the users lead the team by selecting a region of interest and the rest of users can freely control their viewpoints within the designated region, and slaved mode where one of the users completely takes over the control of the navigation in the  environment. The slaved mode of collaboration is similar to that available in CAVE-like projection displays. SCAPE also offers a set of unique tangible user interfaces to facilitate interaction with 3D data sets and cooperation with other users. A detailed discussion on SCAPE's conceptualization and interface framework can be found in our 2004 Presence Paper.

The SCAPE construction shown on the left was completed in July 2004 at the University of Arizona. The design was upgraded from the 1st-generation system Hong and her students built at the University of Illinois at Urbana-Champaign (Please refer to the IEEE VR 2003, and CGA 2004 for detailed implementation), while she also built a temporary prototype at the University of Hawaii in 2003.

• SCAPE concept simulation (also as the cover page for the April Issue of the 2004 Presence Journal)

• SCAPE construction at UA (Exterior, Interior)

Click here to learn more...

AR Display Calibration

One of the major challenges in augmented environment is achieving accurate dynamic registration of the real and virtual views. We have been investigating methods to calibrate HMPD-type of display system to achieve accurate registration in augmented environments. The calibration methods can also be adopted for other type of HMD display calibration.

Click here to learn more...

Head-Mounted Projection Displays (HMPD)

Pioneered by Fisher, head-mounted projective displays (HMPD) have been proposed as an alternative to conventional eyepiece-based head-mounted displays (HMD). Its novel concept and properties address part of the problems in state of the art visualization devices and makes it suitable for wearable systems and multiple user collaborative applications. Collaborated with Dr. Jannick Rolland at the University of Central Florida, we investigated the design and prototyping of the first-generation compact displays for augmented reality applications.

Click here to learn more...

Omni-Directional Vision

Panoramic vision systems are useful in many applications like special effects, immersive virtual environment to name a few. In this project, we study the design of such panoramic systems for single- or multi-view panoramic video capture and their applications in remote reality and distance collaboration.

Click here to learn more......