Consume the things you’d do on your phone/laptop, but on the smart glasses display.

Use your computer (with multi-monitors), watch movies, join a video call, scroll social media, read a book, etc. on your smart glasses.

Contextual Search Engine

Search engines are required when one is thinking and identifies a knowledge gap that limits further progress in reasoning. The user then switches context from what they are doing to a browser, opens a search engine, forms a  query, and inputs that into a search engine in order to fill that knowledge gap. Search engines of today automate the step that goes from query to answer.

A contextual search engine automates the identification of knowledge gaps, context switch, query formation, and search. It understands the context of a meeting, lecture, conversation, podcast, work session etc. and runs constant search on public and private data for relevant info, overlaying that on the user’s vision as soon as it’s needed.

A contextual search engine constantly listens to a user’s context, loads relevant information, and overlays it on their vision.

• Someone said a word you don’t know? Instantly see a definition of that word.
• Mention of a politician you’ve never heard of? See their screenshot and summarised bio.
• Someone comes from a country you’ve never heard of? Instantly see it on a world map, along with description of the demographics, language, etc.
• Telling someone about a paper you read last night? The system automatically loads the paper summary and sends a reference to your conversation partner with a simple command.
• Can’t remember what your colleague said they are using for cloud services? The system searchers your past conversations and pulls up the answer as soon as you realise you need it.

Ray Kurzweil on Lex Fridman: “It would actually be quite useful if somebody would listen to your conversation, and say, oh that’s so and so actress, tell you what you’re talking about…

that can just about be done today where we listen to  your conversation, listen to what you’re saying, listen to what you’re missing and give you that information.”

Live Captions

Smart glasses can provide live captions for the deaf and hard of hearing to help them better understand and communicate.

Smart glasses can read text out loud, identify and speak objects, or find your things in a room. These tools can be combined to restore many visual skills to the blind.

Time what you do with simple annotation voice commands that start a timer for whatever event you have queued. For example, the system can keep you on track when cooking a meal, doing the laundry, following your morning routine, etc. This can help turn unimportant necessity actions into algorithms to improve time efficiency.

This can essentially be thought of as a wearable web browser.

Read while you walk, run, do errands, commute, etc.

Not only read, but have direct input to the computer, so you could read a static source (e.g. book) or just as easily be actively searching the web.

It’s for use at times where it’s not realistic to get a lot of human output, past where one would use their phone, i.e. a time when you can read, listen, observe, but not actively create and write

Wearable web browser would really be the best, always any time google right in front of you.

Speech to text could read out loud what you select – see Thirst for Google glass –

https://mashable.com/archive/thirst-google-glass-app

Make it easy to save content, bookmarks content, send to your computer, share with a friends

Ask a natural language question and receive an answer overlaid on your vision or spoken out loud.

Search for images based on a text query. Learn what something looks like that you don’t know.

Ability to video call people while you are in a scenic, calm location, hands-free.

Basically just this but done well in a comfortable, long lasting pair of glasses:

http://vcai.mpi-inf.mpg.de/projects/EgoChat/

When in the store, whenever I look at a product, I want to be able to see the price of that product at many other stores. This will tell me if the product is outrageously overpriced and I should buy it somewhere else, or perhaps it’s a good deal and I should get 2 while it’s on sale.

The search could be automated and the glasses could read the price from other stores, find the average, compute the “bargain score” of every item, and overlay that score on the grocery items one sees.

Shopping Item Info

Multi-media Notes

Take notes that combine text, speech, audio, and video, anytime, anywhere.

https://zackfreedman.com/2013/03/08/the-five-lowest-hanging-fruit-of-google-glass/

Simple voice UI to instantly check your calendar and create an event on the go. Use your voice to specify a title and time – never again forget to enter information in a calendar, do it mid-conversation.

Could be mid-conversation, during a run, when you remember you need to do something tomorrow and don’t want to change tasks, etc.

Read recipes aloud.

Display the recipe algorithm steps overload on your vision.

Voice commands to move through the recipe.

IOT connection to your cookingware and AR display to understand the cooking instruments.

Sample UI:

https://www.linkedin.com/posts/laurencason_ar-mr-xr-activity-6876256868193316866-4PBA

AR overlay which defines words you don’t know when you read or hear them.

The system would have a vocabulary model of all the words that you know (which you’ve used or been exposed to many time) and a model of what words are more rare, to score words based on how likely it is that you don’t know them.

Then, words you don’t know can be automatically defined when one is exposed to them, with no need to request a definition from the computer.

An early application of wearable BCI could watch for ERPs that indicate novelty/surprise (e.g. P300) which could signal that a word heard was not understood.

A map overlaid on your vision with head up directions. Voice command or mobile phone could enter data, it’s then displayed on the glasses.

Smart watch good for with heart overlay

I personally get annoyed when biking and having to constantly stop to check my phone. Or worse, when running and have to stop the run, take off gloves, pull out phone from pocket, etc.

Further this would help you stay in the moment, as the navigation could disappear and only reappear if you go off course.

When inside a building, one wants to be able to ask where a specific location within that building is, and be guided on how to get there.

For example, being in a grocery store, one might ask where the peanut butter is, and the glasses could tell you where to go.

Better, put in a shopping list before you start, the system defines a route, and the route is overlaid on your glasses, and when you get to the product you want to buy, it is highlighted and pointed at in AR.

Or in a mall, find the store you need.

Or find the nearest washroom

Visual search takes a picture of anything that you’re looking at and tells you what that thing is, and tells you information about that thing. See Google Lens.

See your exact location, looking down from above, in a satellite view.

Extend you experience and awareness of the 3D environment to an expanded area and scope.

Help improve your spatial awareness.

Appreciate the beauty and the size of the place where you are.

Could explore drone/camera, topography, weather, map, etc. different views to help you understand your environment.

Smart glasses are the ideal location for biosensing because they allow for subsequent body and brain sensing, capturing almost all of the same body signals that a wrist worn sensor can capture.

Wearable sensing of the body has already proven itself extremely important in terms of physical health and performance. Wearable brain sensing is now doing the same thing for mental health.

An AR speaking coach that listens to your speech and helps answer the questions:

– what is my speaking rate?
– how often do I use filler words?
– what percentage of the conversation do I spend speaking?
– what is the breadth of my vocabulary?
– how understandable am I?
– how engaging is my speaking?

Use speaker diarization, facial recognition, and ASR to recognize when you are in a conversation and figure out your words-per-minute WPM speaking rate. This could also detect filler words, sentiment, etc. to serve as an automated speaking coach.

Detailed map-based view of your location as a time-density plot – understand where you spend the most time, when you spend your time there, daily, weekly, monthly, yearly patterns, etc.

Fitness/sports assistant providing you with live fitness and sport related information.

All physically demanding sports could features live overlay of fitness physiological metrics:

Sports involving movement (running, biking, etc.) could feature live HUD navigation with satellite overview, HUD overlay distance tracking, speed tracking on HUD

See: https://us.engoeyewear.com/products/engo-2

Biosignals overlay on your vision immediately available at any time that you request it.

Alerts if your heart rate is abnormal – for example goes very high when you aren’t exercising.

This is like “Exercise+fitness mode” but you would access the biosignals at times you aren’t exercising.

During a conversation, when someone says a word I don’t know, I want to be able to define it immediatley.

This may be a highly abstract word, requiring a semantic definition, maybe it’s a thing that is better shown (image), or maybe it’s somewhere in between. The wearable should decide and display the correct thing.

The system should know what I know (know my vocabulary) and thus should be able to guess with high cetainty if I will need a word defined.

Two people who speak different langauges can each wear a pair of smart glasses and use live language translation to have a full conversation. If both parties have smart glasses, then each party can speak their own language naturally, and see everything said overlaid on their vision in their native tongue, with no need to pause to wait for translation.

This is a teleprompter for presentations, etc. for when you want to read exactly what’s on the screen

Wearable Presenter

Live in a human-machine feedback loop where the computer co-processor provides you with insights into the body language being displayed around you.

This functionality is useful to anyone who wishes to improve their emotional intelligence by better understanding the affective state of others around oneself. Combining body language, facial expression, tone of voice, language sentiment, facial expressions, and more, one can generate an affective summary of others in the room. Then, the system can feed the user specific useful information, such as acute stress responses, confidence, etc.

This use case is especially useful for autistic people who may have a hard time understanding affective communication, as it makes quantitative what is otherwise qualitative information.

[1]: Picard –

https://mitpress.mit.edu/books/affective-computing

AR Dating

Recognize faces and tell the user the name of the person they’re looking.

Never forget names.

When I am stuck on a problem, or trying to figure out how to do something, or operating in an area that I don’t know a lot about, the best thing to do is to “go fishing” and ask my network for help + guidance

If I had constantly on audio, constantly embedded transcripts/conversations, constant facial recognition, I could put together a database that creates profiles of the expertise, knowledge, skills of everyone I know. I could then make queries to this knowledge/skills human netwrok representation and ask things like:

This could tie into LinkedIn, personal websites, resumes etc. as more information about each person to deduced their skill area.

Pull up a profile of someone I know or someone that I’m just meeting and see an overview of everything I know about them and everything they choose to publically share.

Like a generative linked in, but also with affective information, personal/relationship information, historical information,

Could tell me what we ussually talk about.

A timeline of every time I’ve seen them.

How they react to me (affective information).

Their personality profile as extracted form online, offline, and personal interactions, etc.

Eventually would be able to predict what they would like, dislike, how they would respond to things, etc.

Set a human timer – any time you think of something you want to discuss with a specific person, make a short note of it and set it as a human timer. Then, next time you see that person, the note will popup under “Human timers” – a reminder of the things you wanted to discuss.

Show a list of all the conversation you’ve ever had.

Conversations can be sorted by time of course, but they can also be sorted by semantic information (who you were with, what you were talking about, where you were, what the weather was like, the rough time of day)

Opening a conversation will show the raw transcript, and a list of topics.

Live stream to Twitch or Youtube.

See the live chat overlaid on your glasses.

Stream multimodal sensor streams to others.

Lots of people want this, it seems.

Input a phone number into your device hands free and add a new contact using voice command on smart glasses.

Listen to (transcribe) everything that’s being said, pull up relevant info from my own personal knowledge database (eventually personal knowledge graph) that is closely related to what is being discussed.

Closely related to Contextual Search Engine.

After a conversation, I want to be able to review a summary of that conversation, read it out.

I should be able to quickly/easily search through past convos (on phone or glasses), pick one, and get the summary/highlight reel/most important points.

The user need only filter the conversations down to an acceptably size list, and can then allow the user to pick the correct conversation from a list of conversation using the thumbnail and meta data to identify the correct conversation.

When a past convo has been selected, the system automatically displays and/or reads out loud a summary of the conversation, with a user setting defining the length (longer is more detailed, short is more abstract) of the summary.

This could be a memory upgrade of the most educational and important time we spend – communicating with other people.

People try to take meeting minutes. The best of the best often aren’t in meetings per se, but many, many microinteractions in a day (when one is working inside of a company). Imagine a wearable that could create notes, subconsciously, of every one of those conversations. Whenever the user wants to remember a conversation, they can pull it up on their glasses with a natural language memory

Example: “we were at Acme’s headquarter and I was talking with Matt Smith about 2 weeks ago. “

The user need only filter the conversations down to an acceptably size list, and can then use a smart glasses ui to pick the correct converstaion from a list of conversation using the thumbnail and meta data to identify the correct conversation.

When a past convo has been selected, the system automatically displays and/or reads out loud a summary of the conversation, with a user setting defining the length (longer is more detailed, short is more abstract) of the summary.

This could be a memory upgrade of the most educational and important time we spend – communicating with other people.

The metadata by which to remember conversations:

A running cache which you can fill will all of yours ideas related to the thing that you are currently thinking about.

One would keep making a new cache all of the time, when you are done thinking through an idea (e.g. end of a walk) you should close the previous cache with a voice command.

Next time you want to remember some thing, you should be able to view a quick highlight reel of automatically generated summary of whichever previous cache you want to pull up

A personal knowledge management system is used to record information in a way that leads to better ideas, memory, knowledge connection/cohesion, etc. Current system are largely laptop based, with smart phones serving as an input interface while on the go. By bringing a PKMS onto your face with smart glasses, one can manage all of their knowledge, because smart glasses can see what you see and hear what you hear.

See vimwiki, Roam, Obsidian, Notion, etc., and imagine being able to capture and connect information from your glasses, anywhere, anytime.

Voice commands to add items to a “TODO list”.

Simple voice command to pull up list, auto scroll, use voice command to scroll, or scroll on glasses touchpad

delete items with command

upload to do to computer/ PKMS

What if the TODO list always sense what you were doing, where you were, the urgency/priority of your TODOs, etc, and combined all of that info to figure out, contextually, when the best time to give you your TODO list is?

Some times, things enter your head, and you need to just deal with them later. They’re too important to leave to change of memory, and too important to just be another note in a list. But you don’t want to or can’t spend the time to think about them right now. So, there should be a way to throw off items into a TODO list with a rough deadline, and when you have some free time soon, pull them up to deal with those thing.

An example use case is to “throw off” all of the things you really need to do asap but can’t do today, and then load this note while you plan tomorrow’s schedule, and you’ll know exactly what you have to fit in.

Vehicle / Car

Remote Work in Vehicle

Using your smart glasses as a monitor while working in your car. All cars will be autonomous very shortly, and then we’ll have a lot of time to do things in the vehicle. There’s not enough space for a laptop. Screen/monitor smart glasses will allow triple monitor setups in the vehicle.

Driving Aids

See information that helps you when you’re driving, including:

– Speedometer/tachometer
– live maps/navigation
– live driving metadata (waze data)
– current gear + shift indicator
– radar detector (for visual indication instead of a loud ringing)
– dashboard info (gyroscope, engine/oil temp, boost pressure, lap times, etc)
– Live weather
– Dangerous driver detection visual overlay
– Obstacles (parked cards, pedestrians, etc.) visual overlay
– Infotainment/environment

Travel + Tourism while in Vehicle

See cards overlaid on buildings indicating the shops, reviews, menu, etc.

See cards overlaid on landmarks with information and history

See notes from your social network that are tagged to various locations, landmarks, etc.

Parking Memory / AR Car Locator

Remember where you parked your car with GPS and give visual aid in the parking lot to find your car (make car glow, give arrows to get to car).

This could also assist you in identifying your ride when hailing a cab, getting in an Uber, or summoning an autonomous vehicle, visually overlay the vehicle, make it glow in the user’s vision.

Gaming in Vehicle

Playing games with other passengers in an autonomous vehicle

Where are my keys?

Where did I park my car?

Where is my wallet?

Did I eat breakfast this morning?

What did I eat for dinner last night?

What restaurant did we go to last time we met?

Smart glasses can answer these questions by running object, scene, person, etc. recognition on one’s POV at all times. Then, when a question about the past is requested, a semantic search/QA NLP system can be used to return the answer.

Egocentric audio, video, biosensing, all come together to form a stream of information that can capture the unique view of the user.

Simple use case is just taking a picture when a voice command is given.

Voice commands to take a picture, start/stop audio recording, start/stop video recording, show a viewfinder, review a video that was taken, etc.

This is also a content generation device for artists, youtubers, industry making training videos, etc.

There is also the possibility to use generative models to create a new type of human-machine music generation experience. Put on a backing track, start playing guitar with smart glasses on. The glasses will listen to the music, generate mulitple possible guitar music to accompany it, convert it to tab, and overlay on your glasses.

This would create a human-computer-instrument system where sometime the human would improvise, and sometime the human could interface/work with the computer on what to play, by choosing the best generated tab that the smart glasses are presenting, live, for the current piece.

A doctor working in emergency wards who has needs to view patient information live as she interacts with them.

There are many professions that could improve by having instant access to information that is industry-specific.

Professionals can use smart glasses as a mobile display to pull up that information anywhere they need, without having to change context, hands-free.

Voice command check lists – see checklists overlaid on your vision, gradually check things off and scroll down through the list until everything is done.

This is the “MultimediaContentGenerationWearable” but with the specific purpose of generating training content for trainees in your company

Someone with tons of experience films themselves doing something, explains what they are doing, etc. Then trainees watch that video and learn how to do it hands on

Imagine a specific commercial hardware unit, like a furnace. Everytime it’s worked on, imagine the GPS location was used to track which unit the tech was working on. Immediately upon walking up to the unit, information about the unit is overlaid on their vision, with the ability to go deep into the maintenance log to understand more information about that specific unit. They could use voice commands to record any novel information about the system, notes about what they changed, what they noticed, etc.

All the data the field worker records could directly update tables, values, and notes that comprise the digital twin of that hardware/machinery

The single biggest use case for smart glasses in industry today.

Have a remote call with an expert where they can see your POV video and audio. The remote expert can walk you through a number of steps/actions, and can draw on your AR overlay to highlight parts, etc.

http://files.vuzix.com/Content/pdfs/VuzixTelemaintenanceWhitePaper-042016.pdfMWSItem