
The Big List of Smart Glasses Use Cases
Author: Cayden Pierce
Finally, this is a work in progress. If you find errors, and want to submit a use case to be included, please reach out.
General
Wearable Display
Consume the things you’d do on your phone/laptop, but on the smart glasses display.
Use your computer (with multi-monitors), watch movies, join a video call, scroll social media, read a book, etc. on your smart glasses.
Contextual Search Engine
Search engines are required when one is thinking and identifies a knowledge gap that limits further progress in reasoning. The user then switches context from what they are doing to a browser, opens a search engine, forms a query, and inputs that into a search engine in order to fill that knowledge gap. Search engines of today automate the step that goes from query to answer.
A contextual search engine automates the identification of knowledge gaps, context switch, query formation, and search. It understands the context of a meeting, lecture, conversation, podcast, work session etc. and runs constant search on public and private data for relevant info, overlaying that on the user’s vision as soon as it’s needed.
A contextual search engine constantly listens to a user’s context, loads relevant information, and overlays it on their vision.
- Someone said a word you don’t know? Instantly see a definition of that word.
- Mention of a politician you’ve never heard of? See their screenshot and summarised bio.
- Someone comes from a country you’ve never heard of? Instantly see it on a world map, along with description of the demographics, language, etc.
- Telling someone about a paper you read last night? The system automatically loads the paper summary and sends a reference to your conversation partner with a simple command.
- Can’t remember what your colleague said they are using for cloud services? The system searchers your past conversations and pulls up the answer as soon as you realise you need it.
Ray Kurzweil on Lex Fridman: “It would actually be quite useful if somebody would listen to your conversation, and say, oh that’s so and so actress, tell you what you’re talking about…
that can just about be done today where we listen to your conversation, listen to what you’re saying, listen to what you’re missing and give you that information.”
Assistive
Live Captions
Smart glasses can provide live captions for the deaf and hard of hearing to help them better understand and communicate.
See:
- https://www.xander.tech/
- https://xrai.glass/
- https://www.transcribeglass.com/
Blind Vision
Smart glasses can read text out loud, identify and speak objects, or find your things in a room. These tools can be combined to restore many visual skills to the blind.
Tools
LifeTimer
Time what you do with simple annotation voice commands that start a timer for whatever event you have queued. For example, the system can keep you on track when cooking a meal, doing the laundry, following your morning routine, etc. This can help turn unimportant necessity actions into algorithms to improve time efficiency.
Wearable Web Browser And Reader
This can essentially be thought of as a wearable web browser.
Read while you walk, run, do errands, commute, etc.
Not only read, but have direct input to the computer, so you could read a static source (e.g. book) or just as easily be actively searching the web.
It’s for use at times where it’s not realistic to get a lot of human output, past where one would use their phone, i.e. a time when you can read, listen, observe, but not actively create and write
Wearable web browser would really be the best, always any time google right in front of you.
Speech to text could read out loud what you select – see Thirst for Google glass –
https://mashable.com/archive/thirst-google-glass-app
Make it easy to save content, bookmarks content, send to your computer, share with a friends
Natural language information query
Ask a natural language question and receive an answer overlaid on your vision or spoken out loud.
Image Search
Search for images based on a text query. Learn what something looks like that you don’t know.
HUD Videoconferencing
Ability to video call people while you are in a scenic, calm location, hands-free.
Basically just this but done well in a comfortable, long lasting pair of glasses:
http://vcai.mpi-inf.mpg.de/projects/EgoChat/
Shopping Price Checker
When in the store, whenever I look at a product, I want to be able to see the price of that product at many other stores. This will tell me if the product is outrageously overpriced and I should buy it somewhere else, or perhaps it’s a good deal and I should get 2 while it’s on sale.
The search could be automated and the glasses could read the price from other stores, find the average, compute the “bargain score” of every item, and overlay that score on the grocery items one sees.
Shopping Item Info
Multi-media Notes
Take notes that combine text, speech, audio, and video, anytime, anywhere.
https://zackfreedman.com/2013/03/08/the-five-lowest-hanging-fruit-of-google-glass/
Add Calendar Entry with Voice
Simple voice UI to instantly check your calendar and create an event on the go. Use your voice to specify a title and time – never again forget to enter information in a calendar, do it mid-conversation.
Could be mid-conversation, during a run, when you remember you need to do something tomorrow and don’t want to change tasks, etc.
HUD Cooking
Read recipes aloud.
Display the recipe algorithm steps overload on your vision.
Voice commands to move through the recipe.
IOT connection to your cookingware and AR display to understand the cooking instruments.
Sample UI:
https://www.linkedin.com/posts/laurencason_ar-mr-xr-activity-6876256868193316866-4PBA
Define Words While Reading and Explain What You Read
AR overlay which defines words you don’t know when you read or hear them.
The system would have a vocabulary model of all the words that you know (which you’ve used or been exposed to many time) and a model of what words are more rare, to score words based on how likely it is that you don’t know them.
Then, words you don’t know can be automatically defined when one is exposed to them, with no need to request a definition from the computer.
An early application of wearable BCI could watch for ERPs that indicate novelty/surprise (e.g. P300) which could signal that a word heard was not understood.
Science
AR Data Visualization
Navigation
HUD Navigation
A map overlaid on your vision with head up directions. Voice command or mobile phone could enter data, it’s then displayed on the glasses.
Smart watch good for with heart overlay
I personally get annoyed when biking and having to constantly stop to check my phone. Or worse, when running and have to stop the run, take off gloves, pull out phone from pocket, etc.
Further this would help you stay in the moment, as the navigation could disappear and only reappear if you go off course.
HUD Indoor Navigation
When inside a building, one wants to be able to ask where a specific location within that building is, and be guided on how to get there.
For example, being in a grocery store, one might ask where the peanut butter is, and the glasses could tell you where to go.
Better, put in a shopping list before you start, the system defines a route, and the route is overlaid on your glasses, and when you get to the product you want to buy, it is highlighted and pointed at in AR.
Or in a mall, find the store you need.
Or find the nearest washroom
Sensing
Visual Search – POV Define What You See
Visual search takes a picture of anything that you’re looking at and tells you what that thing is, and tells you information about that thing. See Google Lens.
Satellite Overhead View of Current Location
See your exact location, looking down from above, in a satellite view.
Extend you experience and awareness of the 3D environment to an expanded area and scope.
Help improve your spatial awareness.
Appreciate the beauty and the size of the place where you are.
Could explore drone/camera, topography, weather, map, etc. different views to help you understand your environment.
Brain And Body Sensing Smart Glasses
Smart glasses are the ideal location for biosensing because they allow for subsequent body and brain sensing, capturing almost all of the same body signals that a wrist worn sensor can capture.
Wearable sensing of the body has already proven itself extremely important in terms of physical health and performance. Wearable brain sensing is now doing the same thing for mental health.
Self-sensing
Speaking CoachWhat is my speaking rate?
An AR speaking coach that listens to your speech and helps answer the questions:
– what is my speaking rate?
– how often do I use filler words?
– what percentage of the conversation do I spend speaking?
– what is the breadth of my vocabulary?
– how understandable am I?
– how engaging is my speaking?
Use speaker diarization, facial recognition, and ASR to recognize when you are in a conversation and figure out your words-per-minute WPM speaking rate. This could also detect filler words, sentiment, etc. to serve as an automated speaking coach.
Where do I spend the most time?
Detailed map-based view of your location as a time-density plot – understand where you spend the most time, when you spend your time there, daily, weekly, monthly, yearly patterns, etc.
Exercise + Fitness
Fitness/sports assistant providing you with live fitness and sport related information.
All physically demanding sports could features live overlay of fitness physiological metrics:
Sports involving movement (running, biking, etc.) could feature live HUD navigation with satellite overview, HUD overlay distance tracking, speed tracking on HUD
Heart Overlay
Biosignals overlay on your vision immediately available at any time that you request it.
Alerts if your heart rate is abnormal – for example goes very high when you aren’t exercising.
This is like “Exercise+fitness mode” but you would access the biosignals at times you aren’t exercising.
Communication / Conversational
Define Words Live While They’re Being Said
During a conversation, when someone says a word I don’t know, I want to be able to define it immediatley.
This may be a highly abstract word, requiring a semantic definition, maybe it’s a thing that is better shown (image), or maybe it’s somewhere in between. The wearable should decide and display the correct thing.
The system should know what I know (know my vocabulary) and thus should be able to guess with high cetainty if I will need a word defined.
Language Translation
Two people who speak different langauges can each wear a pair of smart glasses and use live language translation to have a full conversation. If both parties have smart glasses, then each party can speak their own language naturally, and see everything said overlaid on their vision in their native tongue, with no need to pause to wait for translation.
Live Fact Checker
Presentation
Wearable Teleprompter
This is a teleprompter for presentations, etc. for when you want to read exactly what’s on the screen
Wearable Presenter
Social
Affective Computing EQ Smart Glasses
Live in a human-machine feedback loop where the computer co-processor provides you with insights into the body language being displayed around you.
This functionality is useful to anyone who wishes to improve their emotional intelligence by better understanding the affective state of others around oneself. Combining body language, facial expression, tone of voice, language sentiment, facial expressions, and more, one can generate an affective summary of others in the room. Then, the system can feed the user specific useful information, such as acute stress responses, confidence, etc.
This use case is especially useful for autistic people who may have a hard time understanding affective communication, as it makes quantitative what is otherwise qualitative information.
[1]: Picard –
https://mitpress.mit.edu/books/affective-computing
AR Dating
Smart glasses could enhance the dating experience by giving singles more information and tools to help them meet someone special.
Glasses could provide a HUD with:
Wearable Face Recognizer
Recognize faces and tell the user the name of the person they’re looking.
Never forget names.
Auto-networking – Who do I know that can help?
When I am stuck on a problem, or trying to figure out how to do something, or operating in an area that I don’t know a lot about, the best thing to do is to “go fishing” and ask my network for help + guidance
If I had constantly on audio, constantly embedded transcripts/conversations, constant facial recognition, I could put together a database that creates profiles of the expertise, knowledge, skills of everyone I know. I could then make queries to this knowledge/skills human netwrok representation and ask things like:
This could tie into LinkedIn, personal websites, resumes etc. as more information about each person to deduced their skill area.
What is this Person Like?
Pull up a profile of someone I know or someone that I’m just meeting and see an overview of everything I know about them and everything they choose to publically share.
Like a generative linked in, but also with affective information, personal/relationship information, historical information,
Could tell me what we ussually talk about.
A timeline of every time I’ve seen them.
How they react to me (affective information).
Their personality profile as extracted form online, offline, and personal interactions, etc.
Eventually would be able to predict what they would like, dislike, how they would respond to things, etc.
Human Timers
Set a human timer – any time you think of something you want to discuss with a specific person, make a short note of it and set it as a human timer. Then, next time you see that person, the note will popup under “Human timers” – a reminder of the things you wanted to discuss.
Wearable Conversation Bank
Show a list of all the conversation you’ve ever had.
Conversations can be sorted by time of course, but they can also be sorted by semantic information (who you were with, what you were talking about, where you were, what the weather was like, the rough time of day)
Opening a conversation will show the raw transcript, and a list of topics.
POV Streamer Lifelog Multimodal Wearable
Live stream to Twitch or Youtube.
See the live chat overlaid on your glasses.
Stream multimodal sensor streams to others.
Lots of people want this, it seems.
Semantic Memory + Knowledge
Contact + Phone Number Memory
Input a phone number into your device hands free and add a new contact using voice command on smart glasses.
Remembrance Agent
Listen to (transcribe) everything that’s being said, pull up relevant info from my own personal knowledge database (eventually personal knowledge graph) that is closely related to what is being discussed.
Closely related to Contextual Search Engine.
Wearable Conversation Summarizer
After a conversation, I want to be able to review a summary of that conversation, read it out.
I should be able to quickly/easily search through past convos (on phone or glasses), pick one, and get the summary/highlight reel/most important points.
The user need only filter the conversations down to an acceptably size list, and can then allow the user to pick the correct conversation from a list of conversation using the thumbnail and meta data to identify the correct conversation.
When a past convo has been selected, the system automatically displays and/or reads out loud a summary of the conversation, with a user setting defining the length (longer is more detailed, short is more abstract) of the summary.
This could be a memory upgrade of the most educational and important time we spend – communicating with other people.
Meeting Minutes Wearable
People try to take meeting minutes. The best of the best often aren’t in meetings per se, but many, many microinteractions in a day (when one is working inside of a company). Imagine a wearable that could create notes, subconsciously, of every one of those conversations. Whenever the user wants to remember a conversation, they can pull it up on their glasses with a natural language memory
Example: “we were at Acme’s headquarter and I was talking with Matt Smith about 2 weeks ago. “
The user need only filter the conversations down to an acceptably size list, and can then use a smart glasses ui to pick the correct converstaion from a list of conversation using the thumbnail and meta data to identify the correct conversation.
When a past convo has been selected, the system automatically displays and/or reads out loud a summary of the conversation, with a user setting defining the length (longer is more detailed, short is more abstract) of the summary.
This could be a memory upgrade of the most educational and important time we spend – communicating with other people.
The metadata by which to remember conversations:
MXT Cache
A running cache which you can fill will all of yours ideas related to the thing that you are currently thinking about.
One would keep making a new cache all of the time, when you are done thinking through an idea (e.g. end of a walk) you should close the previous cache with a voice command.
Next time you want to remember some thing, you should be able to view a quick highlight reel of automatically generated summary of whichever previous cache you want to pull up
PKMS Smart Glasses
A personal knowledge management system is used to record information in a way that leads to better ideas, memory, knowledge connection/cohesion, etc. Current system are largely laptop based, with smart phones serving as an input interface while on the go. By bringing a PKMS onto your face with smart glasses, one can manage all of their knowledge, because smart glasses can see what you see and hear what you hear.
See vimwiki, Roam, Obsidian, Notion, etc., and imagine being able to capture and connect information from your glasses, anywhere, anytime.
HUD TODO List
Voice commands to add items to a “TODO list”.
Simple voice command to pull up list, auto scroll, use voice command to scroll, or scroll on glasses touchpad
delete items with command
upload to do to computer/ PKMS
What if the TODO list always sense what you were doing, where you were, the urgency/priority of your TODOs, etc, and combined all of that info to figure out, contextually, when the best time to give you your TODO list is?
Some times, things enter your head, and you need to just deal with them later. They’re too important to leave to change of memory, and too important to just be another note in a list. But you don’t want to or can’t spend the time to think about them right now. So, there should be a way to throw off items into a TODO list with a rough deadline, and when you have some free time soon, pull them up to deal with those thing.
An example use case is to “throw off” all of the things you really need to do asap but can’t do today, and then load this note while you plan tomorrow’s schedule, and you’ll know exactly what you have to fit in.
Vehicle / Car
Remote Work in Vehicle
Using your smart glasses as a monitor while working in your car. All cars will be autonomous very shortly, and then we’ll have a lot of time to do things in the vehicle. There’s not enough space for a laptop. Screen/monitor smart glasses will allow triple monitor setups in the vehicle.
Driving Aids
See information that helps you when you’re driving, including:
– Speedometer/tachometer
– live maps/navigation
– live driving metadata (waze data)
– current gear + shift indicator
– radar detector (for visual indication instead of a loud ringing)
– dashboard info (gyroscope, engine/oil temp, boost pressure, lap times, etc)
– Live weather
– Dangerous driver detection visual overlay
– Obstacles (parked cards, pedestrians, etc.) visual overlay
– Infotainment/environment
Travel + Tourism while in Vehicle
See cards overlaid on buildings indicating the shops, reviews, menu, etc.
See cards overlaid on landmarks with information and history
See notes from your social network that are tagged to various locations, landmarks, etc.
Parking Memory / AR Car Locator
Remember where you parked your car with GPS and give visual aid in the parking lot to find your car (make car glow, give arrows to get to car).
This could also assist you in identifying your ride when hailing a cab, getting in an Uber, or summoning an autonomous vehicle, visually overlay the vehicle, make it glow in the user’s vision.
Gaming in Vehicle
Playing games with other passengers in an autonomous vehicle
Episodic Memory
Where is my x? Did I do x?
Where are my keys?
Where did I park my car?
Where is my wallet?
Did I eat breakfast this morning?
What did I eat for dinner last night?
What restaurant did we go to last time we met?
Smart glasses can answer these questions by running object, scene, person, etc. recognition on one’s POV at all times. Then, when a question about the past is requested, a semantic search/QA NLP system can be used to return the answer.
Art
Multimedia Content Generation Wearable
Egocentric audio, video, biosensing, all come together to form a stream of information that can capture the unique view of the user.
Simple use case is just taking a picture when a voice command is given.
Voice commands to take a picture, start/stop audio recording, start/stop video recording, show a viewfinder, review a video that was taken, etc.
This is also a content generation device for artists, youtubers, industry making training videos, etc.
Musical Notes and Chords Tab Smart Glasses Overlay
There is also the possibility to use generative models to create a new type of human-machine music generation experience. Put on a backing track, start playing guitar with smart glasses on. The glasses will listen to the music, generate mulitple possible guitar music to accompany it, convert it to tab, and overlay on your glasses.
This would create a human-computer-instrument system where sometime the human would improvise, and sometime the human could interface/work with the computer on what to play, by choosing the best generated tab that the smart glasses are presenting, live, for the current piece.
Industry
Wearable Quick Reader Information Stream
A doctor working in emergency wards who has needs to view patient information live as she interacts with them.
There are many professions that could improve by having instant access to information that is industry-specific.
Professionals can use smart glasses as a mobile display to pull up that information anywhere they need, without having to change context, hands-free.
Picking
Checklists
Voice command check lists – see checklists overlaid on your vision, gradually check things off and scroll down through the list until everything is done.
Training Video Creation and Viewing
This is the “MultimediaContentGenerationWearable” but with the specific purpose of generating training content for trainees in your company
Someone with tons of experience films themselves doing something, explains what they are doing, etc. Then trainees watch that video and learn how to do it hands on
Hardware Digital Twins
Imagine a specific commercial hardware unit, like a furnace. Everytime it’s worked on, imagine the GPS location was used to track which unit the tech was working on. Immediately upon walking up to the unit, information about the unit is overlaid on their vision, with the ability to go deep into the maintenance log to understand more information about that specific unit. They could use voice commands to record any novel information about the system, notes about what they changed, what they noticed, etc.
All the data the field worker records could directly update tables, values, and notes that comprise the digital twin of that hardware/machinery
Remote Expert + Telemaintenance + Remote Support
The single biggest use case for smart glasses in industry today.
Have a remote call with an expert where they can see your POV video and audio. The remote expert can walk you through a number of steps/actions, and can draw on your AR overlay to highlight parts, etc.
http://files.vuzix.com/Content/pdfs/VuzixTelemaintenanceWhitePaper-042016.pdfMWSItem
Acknowledgements
Most of these idea did not originate from the author, and many can’t be attributed to any one person. These use cases are an amalgamation from a plethora of conversations, reading, imagining, and community input, and simply edited, combined, and (sometimes) refined by the author.
I’d like to recognise a growing list of people who have made some idea contribution to this list, directly or indirectly. Thanks to (not ordered):
Terry Huhtala, Jeremy Stairs, Alex Israelov, Adrian Papineau, LMichelle Salvant, Aaditya Vaze, Derek Lam, Kyle Simmons, Kyle Mathewson, JD Chibuk, Stephen Wolfram, Tom Fleay, Steve Mann, Jimi Tjong, Bradley Rhodes, Thad Starner, Pat Pataranutaporn, Valdemar Danry, Nataliya Kos’myna, Patrick Chwalek, Robert Scoble, David Rose, Ray Kurzweil, and many, many other pioneers of smart glasses use cases.