Today the US Patent & Trademark Office published a patent application from Apple that generally relates to computer systems for virtual/augmented reality, including but not limited to electronic devices for interacting with augmented and virtual reality environments.
The development of computer systems for virtual/augmented reality has increased significantly in recent years. Example virtual/augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices, such as touch-sensitive surfaces, for computer systems and other electronic computing devices are used to interact with virtual/augmented reality environments.
Example touch-sensitive surfaces include touchpads, touch-sensitive remote controls, and touch-screen displays. Such surfaces are used to manipulate user interfaces and objects therein on a display. Example user interface objects include digital images, video, text, icons, and control elements such as buttons and other graphics.
However, methods and interfaces for interacting with environments that include at least some virtual elements (e.g., augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, using a sequence of inputs to select one or more user interface objects (e.g., one or more virtual elements in the virtual/augmented reality environment) and perform one or more actions on the selected user interface objects is tedious, creates a significant cognitive burden on a user, and detracts from the experience with the virtual/augmented reality environment.
In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-operated devices.
Apple's invention covers computer systems with improved methods and interfaces for interacting with augmented and virtual reality environments. Such methods and interfaces optionally complement or replace conventional methods for interacting with augmented and virtual reality environments.
Such methods and interfaces reduce the number, extent, and/or nature of the inputs from a user and produce a more efficient human-machine interface. For battery-operated devices, such methods and interfaces conserve power and increase the time between battery charges.
In some embodiments, the computer system includes a desktop computer. In some embodiments, the computer system is portable (e.g., a notebook computer, tablet computer, or handheld device).
In some embodiments, the computer system includes a personal electronic device such as a headset as noted in patent FIG. 5A2 below wherein the user is viewing an augmented reality environment and able to interact with the environment with a touch sensitive remote control, a wand, a touch sensitive surface found on the headset and so forth. The image below illustrates the user touching an iPhone as a navigation tool.
The user is able to add virtual people, cars, trees, bushes and so forth and dissect the building floor by floor if so desired. You could see architects loving this when putting together a proposal and wants investors to be able to see a total environment, building by building and so forth.
Apple has shown during their annual iPhone or iPad events that next-gen interactive games could be enjoyed using AR. Apple also provides developers with Swift Shot for gaming as presented in the video below, and a lot of the patent and its figures delve into such gaming and interactive features.
Apple's patent figures below also show that Apple has patented Swift Shot and the principles behind it.
Apple also notes that in some embodiments, the user interacts with the GUI in part through stylus and/or finger contacts and gestures on the touch-sensitive surface.
In the future Apple envision users with a mixed reality headset being able to use apps for image editing, drawing, presenting, word processing, spreadsheet making, telephoning, video conferencing, e-mailing, instant messaging, workout support, digital photographing, digital videoing, web browsing, digital music playing, note taking, and/or digital video playing. This was a part of a 2018 patent covering a user working on a work project while being driven to work in an autonomous vehicle.
Under the heading "User Interfaces and Associated Processes," Apple touches on "a projector, a heads-up display" which was covered in more depth in an invention that was published in August 2018 in relation to an AR Windshield.
Little tidbits found in the patent pointed to a headset being locked and unlocked with just "verbal input for activation" and that Augmented Reality logic could be integrated into future televisions as well as the Apple TV box.
Apple's 292-page AR patent application was originally filed for back in Q3 2018. Although it's a very long patent filing, it boiled down to only 18 patent claims focused primarily on a "method" and a "computer system" that could generate an augmented reality environment.