Shoulder connects to nothing — no logins, no API keys, no “is my bank supported.” It works with any site or app because it watches the screen, the same way you would. Here’s what’s actually happening between recording and chat.

Projects, recordings, extractions

Shoulder organizes everything around three things:

  • Project — a goal you’re tracking, like “remember how I pay the mortgage” or “collect what I spend on the renovation.”
  • Recording — one screen-capture session inside a project.
  • Extraction — what Shoulder pulled out of that recording, measured against the project’s goal.

Why the goal matters

Shoulder doesn’t transcribe everything on screen — it extracts against the goal you stated when you created the project. A vague goal gets vague extractions. The more specific the goal (“track the confirmation number and due date for every bill I pay,” not just “bills”), the more useful the answers later.

Tip: You can edit a project’s goal at any time from the project settings — it only affects extractions going forward, not ones already saved.

What happens after you stop recording

  1. The recording uploads to the Shoulder relay for extraction — this is the only step that touches the cloud.
  2. A cloud model (Gemini Flash) reads the recording against your project’s goal and pulls out the relevant facts, links, and numbers.
  3. The extraction is written to your project’s on-device knowledge base, and the raw video is discarded from the relay.

Chatting with a project

Once a recording has been extracted, you can ask the project thread about it in plain language. Chat runs against your on-device knowledge base for that project — answers reference the extraction (and, through it, the recording) they came from, so you can always trace an answer back to where it was seen.

Current build note: chat responses are generated through the Shoulder relay rather than fully on-device — see Privacy & Your Data for exactly what that means for your data today.