Show HN: Eyesite – Experimental website combining computer vision and web design

(blog.andykhau.com)

135 points by akchro 5 days ago

26 comments

View on Hacker News

I wanted Apple Vision Pros, but I don’t have $3,500 in my back pocket. So I made Apple Vision Pros at home.

This was just a fun little project I made. Currently, the website doesn't work on screens less than 1200x728 (Sorry mobile users!) It also might struggle on lower end devices.

For best results, have a webcam pointing right at you. I tested my website with a MacBook camera.

Any comments, questions, or suggestions are greatly appreciated!

blog: https://blog.andykhau.com/blog/eyesite

check it out: https://eyesite.andykhau.com/

github: https://github.com/akchro/eyesite

weeb 4 days ago

Nice to see people getting interested in eye gaze. There are two things that you might like to look at that can help the UX.

1 - Calibration. Looking at static dots is BORING. The best idea I've seen is Tobii's gaming calibration where you look at dots to make them wobble and pop. This makes the whole process feel like a game, even when you've done it a hundred times before. I would love to see more ideas in this space to give a much more natural-feeling calibration process - even better if you can improve the calibration over time with a feedback loop, when users interact with an element.

2 - Gaze feedback. You are absolutely right that seeing a small, inaccurate and jumpy dot does more harm than good. Again, Tobii have led the way with their 'ghost overlay' for streamers.

For an example, see the following video. After calibration the ghost overlay is used to give approximate feedback. This is enough that some naive users are able to make small adjustments to a constant calibration error, or at least give feedback that the gaze is wrong, not that the UI is not responding.

https://youtu.be/mgQY4dL-09E?feature=shared&t=36

Reply View 2 replies
  • naveen_k 4 days ago

    I have been experimenting with using 5 phases of movements with each phase covering different areas of the screen while being actively moving. The last phase makes the dot move in a Lissajous-like motion which is more fluid like you are suggesting.

    The challenge is recording and syncing the motion at a higher frequency and being able to save without much drift and the performance of these landmark/gaze models is often slow.

    One more option to speed it up is not to do the eye tracking at record time, just record a crop video of the face and the screen first at 60Hz and then run the model on each frame and update the metadata of the dataset.

    Reply View 0 replies
  • akchro 4 days ago

    Thank you for the feedback!

    1 - I experimented with some calibration involving staring at a point, but I found it troublesome as blinking would make lead to some inaccurate calibration data (webgazer doesn't have blink detection). It was also a little more fatiguing since the user would have to really focus on staring the entire time. I found that it was less mentally fatiguing if the user could control their own calibration give themselves room to blink or just rest their eyes for a second.

    2 - Ghost overlays is a really good idea. I'll see what I can do to implement that feature.

    I really appreciate you taking your time to write this!

    Reply View 0 replies
jll29 4 days ago

The WebGaze software used in this page was introduced in by Papoutsaki and co-workers by a team from Brown university and Georgia Tech in 2016:

- Papoutsaki, A. et al. (2016). "WebGazer: Scalable Webcam Eye Tracking Using User Interactions." Presented at the International Joint Conference on Artificial Intelligence (IJCAI). https://www.ijcai.org/Proceedings/16/Papers/540.pdf

While we are at it, you may also find the following research publications relevant to this discussion:

- "Improving User Perceived Page Load Times Using Gaze" (USENIX NSDI 2017) https://www.usenix.org/conference/nsdi17/technical-sessions/...

- "No clicks, no problem: using cursor movements to understand and improve search" (Huang, White, Dumais from Microsoft Research, ACM SIG CHI '11) https://dl.acm.org/doi/abs/10.1145/1978942.1979125

- Virtual gazing in video surveillance (SMVC '10) https://dl.acm.org/doi/abs/10.1145/1878083.1878089

Reply View 0 replies
naveen_k 4 days ago

Ha! The timing is impeccable. This is a great demo. I've been experimenting with using gaze and eye tracking for cursor prediction as a side project. I like the idea of pressing 'space' for each dot. I just had the 9-dot movement going from one point to another. I'm using Mediapipe's face landmarks model (I wasn't aware of WebGazer). I'll continue to work on it, but it's great to see a similar thought process.

Reply View 0 replies
cwmoore 5 days ago

I may be the result of some evolutionary bottleneck, but wherever there is a camera lens, I assume eye tracking (and sentiment prediction) is at least possible, and at most globally always on.

Reply View 8 replies
  • jeffhuys 4 days ago

    Next time you see a digital ad display, look for the little black circle on top of it. Yes, they do. Feel free to put a sticker over it.

    Reply View 7 replies
    • noduerme 4 days ago

      Just to play devil's advocate here... I'm about as extreme a privacy absolutist as you can find. But given that we're all on camera all the time in public spaces, like it or not, I don't consider tracking on digital billboards to be inherently evil. It could be used for evil (like - an authoritarian government tracking who looks at a billboard for an opposition political leader, for instance). But it could just be innocent data gathering. If you ran a plumbing business and paid $10k for a billboard, wouldn't you want to know if it was worth it or not? It's not as if decades of focus groups and hand-wavey feelings about what is or isn't effective advertising didn't already steer us into a society entirely dominated by big loud ads everywhere.

      People who make products and sell services need to advertise. They in turn pay taxes. The many layers of parasitism in the advertising world historically relied on conning these people and taking their money in exchange for an unprovable proposition, namely that if you run this ad we tell you to run, right here, your sales will go up - but we'll never be able to actually tell you for sure by how much, or whether it was a good deal for you. From that perspective, more and better viewership data helps undermine the advertising bullshit machine and close the gap between people who run businesses and the people they're trying to sell their services to.

      Reply View 6 replies
      • jll29 4 days ago

        Remember this incident?

        Italians deploy fearsome SPY MANNEQUINS to win Fashion Wars (2012) https://www.theregister.com/2012/11/22/bionic_mannequins_are...

        According to the media at the time, the mannquins quickly disappeared again after the scandal broke. If you are more cynical, you might question that narrative.

        Reply View 1 reply
      • jeffhuys 4 days ago

        We're just going deeper and deeper into attention-grabbing displays; we now even have ANIMATED screens next to the ROADS in the Netherlands. Flashing, showing texts like "WATCH OUT! We have a new product!".

        I have epilepsy. It's managed, I mean, I'm allowed to drive again. But what about people who don't have it managed (for some no meds will work)? They're just f'd when they walk down the street and Nike NEEDED to switch the screen every 0.2s?

        If I can in _any_ way inhibit their ability to grow, I will.

        If I need your product, I'll find it. I'm against any and all ads - I know it's unrealistic, but I'll do everything in my power to lower the amount of ads I see or to be a nuisance to them.

        Really don't care about morality in this case.

        Reply View 2 replies
      • catlifeonmars 4 days ago

        > People who make products and sell services need to advertise.

        Do they though?

        No one is entitled to better data. It should be on the advertisers to figure out more privacy preserving ways of getting feedback.

        Ubiquitous technical surveillance is, in economics parlance, a negative externality.

        Reply View 0 replies
venk12 3 days ago

Impressive work - I have experimented with Tobii trackers - they are pretty accurate to work with. But accomplishing this with a single camera is definitely something. Would love to follow your work further. Keep going :)

Reply View 0 replies
holografix 5 days ago

Love this, any experimental HCI project managers inspires me to think differently about computers and tech

Reply View 0 replies
mdrzn 4 days ago

Very cool demo! Once this becomes good enough (now it's very wobbly and requires huge UI) I'd love to be able to read articles and navigate just using the eyes. Feels very natural.

Reply View 1 reply
  • akchro 4 days ago

    I'm glad it feels natural! I wanted it to feel similar to Apple vision pros where the eye tracking just works. Hopefully in the future, I can just curl up in a blanket and let my eyes do all the scrolling when I'm reading.

    Reply View 0 replies
renierbotha 4 days ago

This is very cool!! Have you considered making a WebGL game that uses eye tracking for things like aiming? Could be very cool and one of the very few accessible games

Reply View 1 reply
  • akchro 4 days ago

    Thanks for your interest! An eye tracking game is an interesting concept. My concern is that there is already a lot of overhead used for eye tracking that an entire game on top of that would be too laggy to be playable. Also, the eye tracking software right now doesn't seem precise enough for things like aiming. I could totally see a pure standalone game that utilizes eye tracking.

    Reply View 0 replies
estsauver 4 days ago

Hey, I just wanted to say this is one of the coolest demo's I've seen all year. This is really, really fun.

Reply View 1 reply
  • akchro 4 days ago

    Thanks! It means a lot that people also find my stuff cool

    Reply View 0 replies
BugsJustFindMe 4 days ago

> Screen Too Small This application requires a minimum screen size to function properly.

Why?

Reply View 2 replies
  • akchro 2 days ago

    All the components needed to be big enough for eye tracking "selection" to consistently work. It's frustrating when you are trying to look at a button to press, but the eye tracking thinks you are looking elsewhere. If the screen were any small, the components wouldn't fit. I believe the dimensions should work on most computer screen sizes and just bar out mobile devices. This was also intended since the eye tracking does not work on mobile.

    Reply View 1 reply
    • BugsJustFindMe 2 days ago

      > the eye tracking does not work on mobile.

      Isn't that a function of platform and not screen size?

      > All the components needed to be big enough for eye tracking "selection" to consistently work.

      Isn't this also a function more of angles and not pixels? A phone close to my face has the same apparent size as a computer screen farther away.

      Reply View 0 replies