# Arya: full documentation
# For judges
Arya is a grocery app that learns how you shop from your receipts and helps shoppers help each other. It puts the whole grocery chore in one place: shared lists, a guided route through the store, a clear view of your spend and usage, and out-of-stock warnings from other shoppers. Every step that needs AI uses a Meta model. The money, timing and list logic is plain code.
This is an Indie track entry. I built it alone in the hackathon window (Oct 27 to Nov 4, 2026). The problem statement is **Build for Your World**: a tool for everyday life that uses Meta's models to read receipts, labels, voice and shelves.
## Start here
- **Demo video (3 min):** [youtu.be/PeWF66ss3oI](https://youtu.be/PeWF66ss3oI). The live demo and the guided path work without it.
- **Try it, no sign-up:** https://arya.teamnightshift.org, then tap "Try it now". It opens a made-up household in a sandbox that exists only for your visit. Use a phone if you can. On a laptop, make the window about 390 px wide. Camera steps need a phone. The 8-minute guided path is [`docs/quest/JUDGE-PLAYBOOK.md`](/docs/quest/judge-playbook), with a [sample receipt photo](/repo/docs/images/sample-receipt.jpg), a [sample shelf photo](/repo/docs/images/sample-shelf.jpg), and a walkthrough video of the same path, with voice and captions (3 min 48 s): [youtu.be/cbJflgCXV5U](https://youtu.be/cbJflgCXV5U).
- **Everything else:** the [README](https://github.com/vibhortayal/arya/blob/main/README.md) (what it is, why I built it this way, measured results, and known limits) and the [docs site](https://vibhortayal.github.io/arya/).
## How do I try it? (Demo)
- You can play the same loop as the video: Today, Ask ("We finished the Greek yogurt"), Start shopping, and Lists under More. [`README.md`](https://github.com/vibhortayal/arya/blob/main/README.md#try-it-in-one-minute)
- These steps need a phone camera or mic: voice add, photo of a note, Find it for me, Map my store, and receipt snap. Everything else works on a laptop.
- Nearly every feature works live in the demo. The staple-judge verdicts were computed beforehand. Only the data (the household, stores and other shoppers) is simulated. Nothing you add is kept after your visit. What the demo household contains: [`assistant/docs/DEMO.md`](https://github.com/vibhortayal/arya/blob/main/assistant/docs/DEMO.md).
- **Run it yourself** with your own keys (Meta for the AI, Google Places to add stores on your own account): see [Run it yourself](/docs/run-it-yourself). It has a prompt that lets [an AI agent set it up for you](/docs/run-it-yourself#let-your-ai-agent-set-it-up). The [setup in the README](https://github.com/vibhortayal/arya/blob/main/README.md#run-it-yourself) also works.
- **Want an account?** Sign-up is invite only while I harden the cost guard rails. Tap "Join the waitlist" on the [landing page](https://arya.teamnightshift.org) and contact me ([support@teamnightshift.org](mailto:support@teamnightshift.org) or [LinkedIn](https://www.linkedin.com/in/vibhortayal)); judges can be approved.
## Does it matter to real people? (Impact)
**What is the real-world potential?** Everyone who shops for a household does the same weekly work. They keep the list in their heads, try not to rebuy what is already in the cupboard, and wonder where the money went. A receipt has all of that, and nobody reads it. Arya reads it, checks every line against the printed total, and turns it into a list with reasons ([how it works](https://github.com/vibhortayal/arya/blob/main/README.md#what-is-arya)). On my own 114 receipts (1,226 lines), every one passed the arithmetic check. About 4 in 5 of the staple judge's suggestions matched my own purchase rhythm (precision 0.79 against a purchase-cadence proxy; [measured results](https://github.com/vibhortayal/arya/blob/main/README.md#measured-results)).
**Who benefits, and how much does it matter?** People who feed a household or a group, and share a list ([who it is for](/docs/principles)). Shopping is a weekly chore and a household cost, not a rare event. I have not measured time or money saved. I tested on one household, so I do not claim a number ([Known limits](https://github.com/vibhortayal/arya/blob/main/README.md#what-are-the-known-limits)).
**Could it become something people use?** It already runs: it is live, installable and free. It has a demo that needs no account, shared lists for groups, a Lists API, and an MCP connector so an agent can edit your lists ([agent connector](/docs/guide/agent-connector)). Store maps already merge across shoppers, so each trip helps the next. Shared prices are planned and not switched on. It runs as one server today. The plan to scale it is in the [roadmap](/docs/roadmap). I built the base. The more people use Arya, the better it gets for them.
**Why this is built for our world:** it is a shopping problem I had myself ([why Arya exists](/docs/principles)), and it is free to use. What existed before Oct 27, and a dated table of what I made in the window, are in [`docs/WHAT-WE-BUILT.md`](/docs/what-we-built).
## Which Meta models does it use, and for what? (Meta AI tooling)
Arya uses six Meta products plus the Meta Model API. Each has one job. Every call goes through one module, [`assistant/api/meta_client.py`](https://github.com/vibhortayal/arya/blob/main/assistant/api/meta_client.py). The full table, with where each one is in the code, is in [`docs/quest/META-PRODUCTS.md`](/docs/quest/meta-products).
- Spark reads receipt photos first. Wonder re-reads a receipt only if the arithmetic checks flag it. Code: [`decode/service.py`](https://github.com/vibhortayal/arya/blob/main/assistant/api/decode/service.py), [`checks.py`](https://github.com/vibhortayal/arya/blob/main/assistant/api/decode/checks.py).
- Muse Spark Open (open weights, hosted on Baseten) is the second seat on the staple judge. It sees item names only. [`open_client.py`](https://github.com/vibhortayal/arya/blob/main/assistant/api/open_client.py)
- SAM 3.1 draws boxes around shelf items. It pixelates the people it detects before any frame is kept. If that step fails, the frame is thrown away. Detection is not perfect, and I have not scored it. [`shelfscan.py`](https://github.com/vibhortayal/arya/blob/main/assistant/api/shelfscan.py)
- Muse can use Arya. Any assistant that supports MCP can connect to Arya's MCP address. Muse is the reference client. I tested it end to end on my own Muse on 2026-11-02. It read my list, added items and filed receipts through six tools. Setup and details: [README, "Muse can use Arya"](https://github.com/vibhortayal/arya/blob/main/README.md#why-i-built-it-this-way), [agent connector](/docs/guide/agent-connector), code in [`mcp.py`](https://github.com/vibhortayal/arya/blob/main/assistant/api/mcp.py).
- Available but not used: Muse Glimmer (open weights, self-hosted) and the Spark Contributor tier (cheaper, with different data terms). The reasons are in [`docs/MODELS.md`](/docs/architecture/models).
- Built with Muse Code. It wrote the first version of the engine. The Meta Muse agent on my VM (Muse Spark, which used Muse Wonder) coded in parallel with Claude. The 253 commits named "Spark" are its work. Details: [README, "How this was built"](https://github.com/vibhortayal/arya/blob/main/README.md#how-this-was-built).
### Which model does which job?
| Meta product | Id | Job | Measured |
|---|---|---|---|
| Muse Spark 1.3 (Standard) | `muse-spark-1.3` | Reads receipt photos first; staple judge, Curator, Ask Arya, list and date photos, shelf labels, crop checks | 92 of 98 lines on the Wonder comparison set; staple precision 0.79. [Details](https://github.com/vibhortayal/arya/blob/main/README.md#measured-results) |
| Muse Wonder 1.0 | `muse-wonder-1.0` | Re-reads a receipt the checks flag, or when Spark fails | 97 of 98 lines after hand adjudication, read alone. [Details](https://github.com/vibhortayal/arya/blob/main/README.md#measured-results) |
| SAM 3.1 | via Meta Model API | Finds shelves and signs, boxes items for Find it for me, pixelates the people it detects | Used in the guided store scan |
| Muse Voice Transcribe | `muse-voice-transcribe-1.0` | Dictation for the list mic and Ask Arya | n/a |
| Muse Image 1.0 | `muse-image-1.0` | The 23 grab-celebration characters (generated once) and share looks (live, per tap) | n/a |
| Muse Spark Open | `meta-models/muse-spark-open-1026` (hosted on Baseten) | Second seat of the staple judge; dictionary labels | 32 of 32 labels agree with hosted Spark; 3 of 4 staple verdicts |
Spark and Wonder run in Meta's Standard mode. The live reader is `auto` (the code default, `ARYA_PHOTO_READER=auto`). Spark reads first, and Wonder re-reads a flagged receipt. I have not yet measured the pair on photos. Every server call goes through one client with a daily spend limit. The history of the reader is in [`assistant/docs/MODEL-POLICY.md`](/docs/architecture/model-policy).
The eval numbers come from one small set (9 public receipts, 98 lines). Scores change from run to run. So quote the gap between the models, not one number alone. The runs, the scorer and the hand adjudication are in [`docs/evals/`](https://github.com/vibhortayal/arya/tree/main/docs/evals/).
## How is it built, and how well? (Technical depth)
- The model proposes and the code checks. Arithmetic reconciliation, returns, cadence and prices are plain code, so a rerun gives the same answer. There is a cost ledger and there are daily spend caps. The full list of claims, and where to check each one, is in the [judge playbook](/docs/quest/judge-playbook#how-is-it-built-technical-depth).
- Why I made these choices (the idea, a PWA, the stack, hosting, the coding model and each Meta model): [README, "Why I built it this way"](https://github.com/vibhortayal/arya/blob/main/README.md#why-i-built-it-this-way).
- Tests and decisions: about 1,900 Python tests and 550 web tests (as of Nov 3), and a dated [decision log](https://github.com/vibhortayal/arya/blob/main/docs/DECISIONS.md).
- Read more: design decisions, what I could finish in the window and what is pending, and where it stops scaling, in [`docs/TECHNICAL-DEPTH.md`](/docs/architecture/design-decisions).
## What does it not do yet?
- **Data.** I tested on one household's 114 receipts and a public set of 9 receipts. The measured numbers come from small samples.
- **Markets.** I built and tested with US receipts, dollars and stores first. I did not test other markets.
- **Devices.** I tested on one real iPhone. I did not test the Quest-start and live-walk location checks indoors on a real phone. A store with no stored address point cannot be checked.
- **Store maps.** No store comes pre-mapped. A store's map comes only from shoppers' guided scans.
- **Scale.** Arya runs as one server process on one VM. A few dozen people at once is about the limit ([details](https://github.com/vibhortayal/arya/blob/main/README.md#what-are-the-known-limits)). Scaling it out is a plan for after judging. It is not built.
- **Demo allowance.** Each demo visit can spend about $2 on live Meta calls. The demo also has a daily cap. When either runs out, the app says so. Lists and everything that needs no live read keep working. **Use a sample receipt** still works.
- **Accounts.** Sign-up is invite only. It is for a few beta testers and hackathon judges ([how to get in](#how-do-i-try-it-demo)). The demo needs no account.
- **Licence.** Arya's own code is MIT licensed ([`LICENSE`](https://github.com/vibhortayal/arya/blob/main/LICENSE)). The Meta models and third-party code keep their own terms ([`NOTICE.md`](https://github.com/vibhortayal/arya/blob/main/NOTICE.md)).
---
# Judge playbook
An eight-minute path through the live demo. Each step says what to do, what you see, and which Meta model is involved. To watch the same path first, with voice and captions, see the [judge walkthrough video](https://youtu.be/cbJflgCXV5U) (3 min 48 s).
**Live demo:** https://arya.teamnightshift.org/ (no account). Use a phone, or a browser window about 390 px wide. Allow the camera and microphone for the steps that use them. The demo household is fictional. Your changes live in a sandbox for your visit only ([what is real, and what is not](https://github.com/vibhortayal/arya/blob/main/README.md#what-is-real-and-what-is-not)).
**What you need.**
- For step 6, a grocery receipt photo. Without one, use [the sample receipt photo](/repo/docs/images/sample-receipt.jpg) (a real receipt of mine), tap **Use a sample receipt**, or **Skip**.
- For step 4, if you are not in a store, open [the sample shelf photo](/repo/docs/images/sample-shelf.jpg) (a Trader Joe's soup and broth shelf, a frame from my own video) on a second screen or a computer. Point the phone camera at it. Turn that screen to full brightness and fill the camera view with the image, to avoid glare. Any other shelf photo works too.
---
## The walk, eight minutes
### 1. Front door
Tap **Try it now**. A short sheet ("Hi, I'm Arya") offers **Show me every screen · 2 min** or **Skip for now**. You land on **Today**, with a bar at the top that says the household is a demo. Today shows your next trip, store news from the simulated town (marked Simulated), what is running low (each with a short reason, such as how often you buy it and when you last did), and your lists.
The navigation tab at the bottom is **Today, Spending, Quest, Ask, More**. **More** holds Lists, Receipts, Stores, Help and Settings.
### 2. Lists
Tap **More**, then **Lists**. The demo has a Costcozy list and a Trader Jill's list (the town's stores have fictional names). The Trader Jill's list holds Bananas, Apples, Organic Milk, Tomato Soup and Hummus. You will look for the tomato soup with the shelf camera in step 4. Hummus is the item Demo Town reports out of stock. The other three are on the sample receipt for step 6. Due staples show under **Suggested for** the store where they are usually bought. Tap one to see why. A **Muse Spark 1.3** staple judge decides which items may be suggested (about 4 in 5 of its suggestions matched my own purchase rhythm: precision 0.79 against a purchase-cadence proxy, [details](https://github.com/vibhortayal/arya/blob/main/README.md#measured-results)).
### 3. Add items by voice and photo
In a list, use the add row. Type an item, or tap the mic (**Muse Voice Transcribe**) and say "two gallons of milk, eggs, and blueberries". The camera (**Muse Spark** vision) reads a photo of a handwritten note. Adds apply at once, with Undo. On a laptop with no camera, type the items instead. The photo read is shown on a phone.
### 4. Start a quest
To reach Tomato soup, start from the Trader Jill's list: **More**, **Lists**, open it, tap **Start shopping**, then on the Quest tab tap **Start at Trader Jill's**. (On Today, **Start shopping** on the next trip also works, if that trip is Trader Jill's.) The quest opens at that list's store.
- An unmapped store plays as a labelled typical layout ("A guess, not a pin").
- Tap **Got it**: +10 points (the Golden item is worth double). The celebration characters were painted with **Muse Image**. There is no clock and no way to lose.
- When you reach **Tomato soup**, tap **Can't find it**, then **Find it for me**. Aim the camera at the shelf (or at the sample shelf photo). **SAM 3.1** boxes candidates and **Muse Spark** checks each crop. Look for the highlight on a soup carton. If it lands elsewhere, try again closer. From the same place, **The shelf is empty** tells other Arya shoppers it is gone.
- **Map my store** (on the Quest tab) opens the guided scan. Pan along the shelves. SAM 3.1 pixelates the people it detects, and Muse Spark reads sections, aisle signs and prices.
### 5. Ask Arya
Open **Ask**. Type "What am I running low on?", then "We finished the Greek yogurt." The answer comes from the household data. The stated fact becomes an action that the server validates before it applies.
### 6. Finish with a receipt
When the list is done, tap **Head to checkout**, then **Add your receipt**. **Snap the receipt** with a photo, or tap **Use a sample receipt**. Behind the screen, **Muse Spark** reads each line, in Meta's Standard mode, and code checks that the lines add up to the printed total. If Spark fails, or the checks flag the read (the lines do not add up, or too many lines are unsure), **Muse Wonder 1.0** reads the receipt again. The screen names the model only when the checks flag a read: "Read by Muse Spark · checking", then "Checked by Muse Wonder" or "Read by Muse Wonder". A clean Spark read shows no model name. The demo reads the receipt but keeps nothing. Use a sample receipt if you would rather not send one.
**With the sample receipt.** Download [the sample receipt photo](/repo/docs/images/sample-receipt.jpg) (Trader Joe's, Milpitas, Nov 3, 2026; it is my own receipt). It matches the demo's **Trader Jill's** list: bananas, apples and organic milk are on the receipt. Tomato soup is the one you find with the shelf camera, and hummus is the item Demo Town reports out of stock, so neither is on the receipt. On a laptop, pick the file after tapping **Snap the receipt**. On a phone, save it to Photos first. A read takes about 30 seconds, or up to about 90 when Muse Wonder re-reads it. Expect 3 lines (organic milk $4.49, 2 apples $2.38, 2 bananas $0.58) and a total of $7.45 with no tax. Tap **Looks right** to accept the read. The filed receipt, under **More**, **Receipts**, shows **Total checks out**. The sample is built by code, so no model name shows. The card number, auth codes and cashier name are blurred. The receipt is dated before your trip. A real account refuses that. The demo accepts it and says so on screen ("Demo only: this receipt is dated ..., before this trip began"). How the readers score on a public set: [`docs/evals/`](https://github.com/vibhortayal/arya/tree/main/docs/evals/).
---
## What if something fails?
| Symptom | What to do |
| --- | --- |
| "demo rate limit" | Wait a minute and retry. |
| "You've used this demo visit's live reads" | The visit's share (about $2) is spent. Lists, Today, Spending and the rest keep working, and **Use a sample receipt** still works. Only the live reads (receipt, photo, voice, Ask) pause. |
| "The demo's live reads are paused for today" | The demo's shared daily budget is used up. The rest is the same as above. The real app has its own daily budget, so judges cannot use it up. |
| Receipt spinner over 90 s | A receipt that Muse Wonder re-reads can take up to about 90 seconds. Wait, or Skip. |
| Voice gives nothing | Check microphone permission, or type. |
| Page looks stale | Reload twice, or clear site data. |
## What do I do when I am done?
Tap **Finish demo** (top right) to leave feedback, replay the tour or see the code.
---
## Does it matter to real people? (Impact)
The four rubric questions are answered on [the judges page, Impact](/docs/judges#does-it-matter-to-real-people-impact).
## How is it built? (Technical depth)
| Claim | Where to verify |
| --- | --- |
| One module calls the Meta API | [`meta_client.py`](https://github.com/vibhortayal/arya/blob/main/assistant/api/meta_client.py) |
| Receipts verified by arithmetic | [`test_receipt_math.py`](https://github.com/vibhortayal/arya/blob/main/assistant/tests/test_receipt_math.py) |
| Demo writes never reach disk | `state.set_overlay` in [`state.py`](https://github.com/vibhortayal/arya/blob/main/assistant/api/state.py); sandbox tests |
| People are pixelated before a frame is kept | `shelfscan.scrub_people` in [`shelfscan.py`](https://github.com/vibhortayal/arya/blob/main/assistant/api/shelfscan.py) (fails closed) |
| Model proposes, code checks: arithmetic reconciliation, returns, cadence and prices are deterministic | [`ARCHITECTURE.md`](/docs/architecture/overview) |
| Cheapest model that holds the score, with a ledger, a rate gate and daily spend caps | [`MODEL-POLICY.md`](/docs/architecture/model-policy) |
| Measured results, small samples, stated in Known limits | [`docs/evals/`](https://github.com/vibhortayal/arya/tree/main/docs/evals/), [README, Known limits](https://github.com/vibhortayal/arya/blob/main/README.md#what-are-the-known-limits) |
| Open API and docs | https://arya.teamnightshift.org/developers |
| Outside services: the Google Places API finds stores for the store picker; Supabase, Resend and Baseten | [README, Built with](https://github.com/vibhortayal/arya/blob/main/README.md#built-with) |
| Tests | `cd assistant && python -m unittest discover -s tests`; known failures in [`STATUS.md`](https://github.com/vibhortayal/arya/blob/main/STATUS.md). CI is manual-only (`.github/workflows/tests.yml`) |
| How the build was run (roles, hard rules, review and test gates) | [`AGENTS.md`](https://github.com/vibhortayal/arya/blob/main/AGENTS.md) |
| Build diary, one log per work session | [`assistant/loops/`](https://github.com/vibhortayal/arya/tree/main/assistant/loops/); how the early commits arrived fast: [README, "How this was built"](https://github.com/vibhortayal/arya/blob/main/README.md#how-this-was-built) |
Read more: design decisions, what I could finish in the window and what is pending, and where it stops scaling, in [`docs/TECHNICAL-DEPTH.md`](/docs/architecture/design-decisions).
## What is not in the demo?
- Imports and batch jobs are switched off in the demo (bulk receipt imports, PDF and text decode, the Curator and staple-judge batch runs).
- Friends, the Store and Global boards, and Shop-Off and Spotter reports show only your own account's data (plus labelled samples), in the real app too. Shelf-empty reports are shared between shoppers in the real app (three shoppers must report the same item at a store, and only shoppers with that item on their list see it). A demo report reaches no one, because the demo writes to its own sandbox.
- Open weights are used in one place: Muse Spark Open (hosted on Baseten) is the second seat of the staple judge. It sees item names only, never personal data. Glimmer is not used.
Everything else it does not do yet, for the real app too: [the judges page, "What does it not do yet?"](/docs/judges#what-does-it-not-do-yet).
---
# What is Arya
Arya is a grocery app that learns how you shop from your receipts and helps shoppers help each other. It puts the whole grocery chore in one place: shared lists, a guided route through the store, a clear view of your spend and usage, and out-of-stock warnings from other shoppers.
**Groceries are an integral part of our lives.** Hunters and gatherers did it to survive. Think about what it takes today.
You have many lists in your notes. Your partner adds items over text while they are shopping at a different store. So **both of you bought yogurt, and neither bought the toilet paper that ran out.** Oh, and **the store shelf for the milk your kids need is empty.** The receipts end up in the trash, so **you never see how the money was spent,** or that the same item was cheaper at another store. **That cheese you bought a few weeks ago went bad** before anyone used it.
> *You hold all of it in your head, and it's still never quite right.*
I built Arya for this hackathon to manage groceries in one place: shared lists, guided store routes, spend and usage insights, and community powered alerts. The idea is simple. When shoppers share what they learn, where items are located, what's cheaper, what's out of stock, every trip gets a little easier.
---
**Arya is not just a chatbot on a grocery list.** Using Muse Wonder, I created Arya's learning engine to understand my household usage from my Costco receipts. Arya showed me patterns in my shopping. Arya also drafts my next list based on what we use.
**I add items** by typing, speaking, or taking a photo of a handwritten list. And Arya makes the chore more fun with **Quest mode**, which turns shopping into a game.
**Before heading out for my quest,** Arya warns me if an item from my list is out of stock. **In the store,** it guides me aisle by aisle. If I need help spotting an item, I point the camera at a shelf: **SAM** finds the item, **Spark** points me to it, and I move forward with my quest. If an item is out of stock, I tap **Not here**, and the next shopper knows.
**At checkout** I snap the receipt. **Spark** reads it, and if needed, **Muse Wonder** takes a second look. And the learning engine improves with every trip.
**Back home,** Arya tells me if something from my previous trips is about to go bad. And when I use up an item, I just tell Arya we're out, and it's on the list automagically.
---
**Arya is soft launched.** The demo is open to everyone, but accounts are invite-only for beta testers. I also integrated the **Meta Muse** AI agent over **MCP** to talk to Arya. A [developer portal](https://arya.teamnightshift.org/developers) is live for API integrations.
During the hackathon, I built the first step. Arya gets wiser with every shared list, every receipt, every "Not here" tap.
> **Humans helping humans.**
>
> **This is Arya.**
*Built for the Meta Global AI Developer Hackathon 2026 (Team Nightshift, Indie track).*
*Today and Lists in the demo household.*
## What is real, and what is not?
| Real | Simulated |
|---|---|
| The live app. Receipts read by Meta models and checked in code. The Lists API and the Muse connector. | The demo household and Demo Town: invented stores and receipts, labelled as simulated. The demo reads a receipt live but keeps nothing. |
Size and other limits: [Known limits](https://github.com/vibhortayal/arya/blob/main/README.md#what-are-the-known-limits).
---
# Built in the hackathon window
The hackathon window opened on Oct 27, 2026. This page says what existed
before it, what I made in it and when, taken from git. Times are US Eastern.
What git cannot show is listed under "What git cannot confirm".
## What existed before the window
- **The idea search.** By my own account it took about two days. The commits for
it are in a separate research repo ([`vibhortayal/meta-hackathon-research`](https://github.com/vibhortayal/meta-hackathon-research)). They start
on Oct 26 at 12:37. The 28 commits that day, up to 16:51, hold candidate ideas
and a plan for spikes, with no spike results. This is before Oct 27, and I
say so plainly.
- **My 114 Costco receipts** (back to 2023) are my own purchase history, imported
as data. They are not in this repo.
- **The server.** The Oracle VM already existed for my other hobby projects.
Setting up Arya on it began on Oct 27.
- **Nothing in the Arya repo is older than Oct 27, 15:13 ET.** Its first commit
already holds the design documents (`SPEC.md` and `IMPLEMENTATION.md`, about
900 lines), which I wrote before committing them.
## What I made, and when
| When (ET) | What | Evidence |
|---|---|---|
| Oct 27, morning | Grocery Navigator brainstorm notes, stamped 11:37 and 11:46 | research repo, `IDEAS-owner-brainstorm.md` |
| Oct 27, 13:50 | Spike 1: receipt decode on 7 real Costco PDFs with Muse Spark, arithmetic matched 7 of 7, $0.20 | research repo, `89ff3ab` |
| Oct 27, 14:26 | Spike 2: photo receipts with Spark, 32 of 32 lines on 4 receipts (5 photos) | research repo, `218f332`, `6ca6d7f` |
| Oct 27, 14:29 | Spike B: the Costco pull test hit a bot wall, so I chose receipt uploads | research repo, `a2ef346` |
| Oct 27, 15:03 to 15:06 | My Costco receipts staged: 63, then 114 | research repo, `25cba97`, `f1cbdd0` |
| Oct 27, 15:13 | **First Arya commit** | `ca9d3ee` |
| Oct 27, 15:31 to 15:38 | Decode, item dictionary, ledger and patterns, and the first phone app land in four commits | `e134814` to `b210863` |
| Oct 27, 15:43 to 15:52 | Public HTTPS, then the DuckDNS address | `e326e348`, `c2430d6d` |
| Oct 27, 16:02 to 16:03 | Photo decode path and Gate 0 verdict, PASS (9 public photos, hand labels) | `cb907e3d`, `e7ebb235`; research repo `39d6433` |
| Oct 27, 21:00 | Muse Wonder commissioned to author the store-frame localizer | `881b115` |
| Oct 27, 21:12 | Model policy amendment: Wonder reads photos through Nov 4 (superseded Nov 2: Spark first, Wonder on flagged receipts, through Nov 9) | `c01c007a` |
| Oct 27, 22:19 | Wonder's first localizer spike vendored as `api/localizer.py` (removed Oct 29, `51cab53`) | `47b2471` |
| Oct 27, 22:59 | Photo decode switched to Wonder | `7f32f2f` |
| Oct 27, 23:02 | Wonder-vs-Spark eval committed in the research repo | `b39a961` |
| Oct 28 | Accounts, the judge front door; Supabase, Resend and Baseten set up | |
| Oct 30 | teamnightshift.org registered at Cloudflare; live site moved to arya.teamnightshift.org with invite-only sign-in | |
| Oct 31 to Nov 2 | Hardening, the navigation tab, lists, the Kroger aisle lookup, docs, "Run it yourself" | |
The Wonder eval is the one the model policy cites. On the 9 Gate 0 photos (98
hand-labeled lines), Wonder scored 97/98 and Spark 92/98. The Wonder figure is after hand adjudication; Spark's is from the automatic scorer, which gives Wonder 86 of 98. Price errors were 0 for Wonder and
4 for Spark, for $4.47 of model spend. Why Spark shows 92 here and 90 at Gate 0 is
explained once, under the model map in [For judges](/docs/judges#which-model-does-which-job).
## What git cannot confirm
- **Who wrote the first engine commits.** Muse Code, running Muse Wonder, wrote
the first version of the engine with me. Git lists those commits under the author name "Spark", at 15:31 to
15:38 on Oct 27. Git records no model or tool, so the author name alone does
not show it. That is my account of it.
- **The other "Spark" commits.** 253 of the 1,333 commits are named "Spark".
They come from the Meta Muse agent on my VM. It ran Muse Spark and used
Muse Wonder through the agent, and it coded in parallel with Claude. Git records the
author name only, so the name alone does not show it. That is my account of it.
- **When the Muse Wonder spike sessions ran.** No start or end time is recorded
for them, and the spike folders the docs mention are in neither repo. I give
no date for them.
## What exists today
The 3-minute video cannot show everything. This list is the standout parts
of Arya that exist today, each with why it matters and where the detail lives.
Everything here is built and in the code. Ideas that are not built are on the
[After the hackathon](/docs/roadmap) page. For the product in full, see
[`assistant/docs/SPEC.md`](https://github.com/vibhortayal/arya/blob/main/assistant/docs/SPEC.md) and
[`assistant/docs/USER-JOURNEYS.md`](/docs/guide/user-journeys).
## Receipts you can trust
- **A model reads, code checks.** Muse Spark reads a receipt photo first (Wonder is the backup
reader). Plain code then checks every line
against the printed total before anything posts. *Why:* a wrong price never
goes in silently. [`assistant/docs/MODEL-POLICY.md`](/docs/architecture/model-policy), [`docs/evals/`](https://github.com/vibhortayal/arya/tree/main/docs/evals/)
- **Returns net out** against purchases.
- **"To check"** marks lines Arya is unsure about. **Fix this line** edits the
name, quantity and price, with Undo. The printed total never changes.
The original photo is kept. *Why:* you stay in control of your record.
- **Background reading with progress:** several receipts at once, and you can
leave while Arya reads them.
## A list that learns
- **Patterns from your own receipts:** how often you buy each item and what you
pay at each store. From that, the list shows what is **running low** and what to
**use soon**, each with a reason (the "why" sheet).
- **A staple judge with two seats:** Muse Spark decides what counts as a staple.
Muse Spark Open (open weights, hosted on Baseten) judges again from item names
only. Disagreements are kept for review. *Why:* names advise, purchase facts
decide. [`assistant/docs/ARCHITECTURE.md`](/docs/architecture/overview)
- **The Curator** proposes product merges and splits with reasons; nothing
changes until you apply it.
- **Your own lists, and every item sits on one:** you make your own lists, for example, one for each store, such as Trader Jill's or Costcozy. When you add an item, it goes to the list you say,
or to your only list, or Arya asks "Which list?".
- **Works offline (signed-in accounts):** lists and other screens show as last saved, and list edits
queue until you are back online. Receipts, Ask, photos, voice and sharing need a connection. Signed-in accounts keep a cached copy. The demo saves offline list changes for that visit only.
## Prices from your own receipts
- Cheaper store, price trend, similar items per ounce, which store to buy each
item at, and the cost of food waste, on a Spending screen. *Why:* nothing
is fetched from store sites, flyers or the web, so every figure is yours.
[`assistant/docs/ARCHITECTURE.md`](/docs/architecture/overview)
- Community prices and live "how busy is the store" are not live; the opt-in
records a choice only (`SPEC.md`, FR-11.7).
## Ask Arya
- A chat over a snapshot of your household's data that can also act on
statements such as "we finished the yogurt".
- Answers in the language you write; Spanish and Hindi were the ones verified
([`assistant/docs/MULTILINGUAL-ASK.md`](https://github.com/vibhortayal/arya/blob/main/assistant/docs/MULTILINGUAL-ASK.md)).
- Voice dictation and a photo of a handwritten list.
## Quest, the optional game
- A hunt through a model of your store with **no clock and no way to lose**.
Start shopping on a list opens it at that list's store and ends on your
receipt. [`docs/quest/GAME-DESIGN.md`](/docs/quest/game-design)
- **Find it for me:** point the camera at a shelf; SAM 3.1 boxes candidates and
Muse Spark checks each one.
- **Map my store:** a guided camera scan. SAM 3.1 pixelates the people it
detects before anything is kept. If that fails, the frame is discarded. Detection is not perfect and is not scored. A store counts as mapped only where it has a usable,
fresh photo, so the status is earned. [`docs/quest/MAP-MY-STORE.md`](/docs/quest/map-my-store)
- **Build a store** from three questions, practice in it, and save it.
## Groups and everyday use
- **Groups** share lists between the people you live with, joined by a
single-use code or an invite link. Receipts stay with whoever filed them.
- **Daily reminders** for what is running low. Live on the hosted app. You opt in, and on iPhone the app must be added to the Home Screen first.
- **Add to Home Screen**, so it opens like an app.
- A navigation tab (the bottom buttons) you arrange yourself.
## Transparency about AI and cost
- **Keeping Arya running** (Settings, About Arya) shows your own AI use by kind
(receipts and photos, Ask, store maps, lists and voice), this month and since
you joined, and the whole app's AI cost and who pays. *Why:* Arya is free, and
the paid AI reading is shown openly.
- **How Arya uses AI** says where AI is used and when to double-check, and that
Arya is not a food-safety check.
- **Arya's powers** lists what Arya does for you now and what turns on as you
add receipts.
- **Guards:** a daily budget for the real app and a separate one for the demo,
a per-visit cap on the demo, and a heads-up and pause for unusually heavy use.
[`assistant/docs/OPS.md`](https://github.com/vibhortayal/arya/blob/main/assistant/docs/OPS.md)
## Open, and bring your own tools
- **Lists API and developer portal** at `/developers` (quickstart in curl, Python
and JavaScript, interactive reference). [`docs/API.md`](/docs/api/lists-api)
- **A connector for your own AI agent (MCP):** Make my code, then ask the Muse
agent to connect. [`assistant/docs/AGENT-CONNECTOR.md`](/docs/guide/agent-connector)
## Built to run
- One image, also run locally with Docker in a few steps
([`docs/RUN-IT-YOURSELF.md`](/docs/run-it-yourself)).
- A `/health` check that shows the running commit, nightly backups, and an
owner-only Ops screen with a daily report, a pause switch for paid reads,
spend ceilings and a cost ledger with one row per model call.
- Tests and measured results: [`STATUS.md`](https://github.com/vibhortayal/arya/blob/main/STATUS.md), [`docs/evals/`](https://github.com/vibhortayal/arya/tree/main/docs/evals/).
## In the code but switched off
- **Use your own Meta key (unfinished, not used).** A Settings row, a Help topic,
an Ask answer and a sign-up door. An account with its own key would never fall
back to Arya's keys. I decided not to use it, so it is switched off
and not part of the demo.
[After the hackathon](/docs/roadmap)
- **Alexa skill (built, switched off, not tested on a real device).** An Alexa
skill that adds items to your Arya list by voice. The code is merged (#515,
#516) behind the `ARYA_ALEXA` flag, which is off. I paused it until after the
hackathon, and it has not been tried on a real Echo. It is voice add only.
Amazon has retired its Alexa list API, so Arya cannot sync with Alexa's
own lists.
---
# Why Arya exists, and what it stands for
Arya is my answer to "Build for Your World". It puts Meta's models to work in a world I know firsthand: the weekly grocery shop.
## The problem
Keeping track of groceries is a small job that never ends. You remember the list, then forget it at the store. You buy what you already have. You cannot say where the money went. I had this problem, so I built Arya for myself first. How the app solves it, step by step, is in [the README, What is Arya](https://github.com/vibhortayal/arya/blob/main/README.md#what-is-arya).
## Who it is for
Shoppers. The people who keep a household or a group fed, and who remember what is running out. If you shop for a household, or share a list with housemates, family or a group, Arya is for you.
## Principles
1. **For shoppers, and built up with them.** The store maps come from shoppers' own scans, and lists are shared by the households and groups who use them. Shoppers helping each other is the point.
2. **Free to use.** It costs money to keep running (the AI reading). You can see what you use in Settings, About Arya, Keeping Arya running. Nothing in it is for sale, there are no ads, and I do not sell your data.
3. **The model proposes, code checks.** A model reads the receipt. Plain code checks every line before anything posts. You can trust the numbers because the arithmetic is checked, not guessed.
4. **Yours by default.** Your receipts and lists are yours. A group sees its members' lists and receipts. If you leave, your receipts go with you. Arya uses no prices from store sites, flyers or the web.
5. **Limits stated.** Built and tested with US receipts, dollars and stores first. Other markets are not tested. What is not live is marked as such. The full list is under [Known limits](https://github.com/vibhortayal/arya/blob/main/README.md#what-are-the-known-limits).
## Where to read more
[The README](https://github.com/vibhortayal/arya/blob/main/README.md) for the app, [who else is out there](https://github.com/vibhortayal/arya/blob/main/docs/quest/BUSINESS.md), [the roadmap](/docs/roadmap) for what is next, and [the decision log](https://github.com/vibhortayal/arya/blob/main/docs/DECISIONS.md) for the choices behind it.
---
# Arya user journeys
Seven journeys, from the first receipt to a shared household. Each one
has a person, a goal, the screens in order, and sample data. The numbers
come from the demo household (`python -m harness.make_demo_household
data`), so you can open the live demo and see the same thing.
Store names: these journeys use real store names from my own shopping. The hosted demo (Demo Town) uses invented ones, such as Costcozy and Trader Jill's.
The list comes first. The Quest hunt is optional.
## How to read this page
**Status tags**
| Tag | Meaning |
|---|---|
| Shipped | On `main` and in the demo. |
| Shipped, flag | On `main`, but a setting keeps it off until I turn it on. |
| Needs a model | Works only when `MODEL_API_KEY` is set. The demo uses the key on the VM. |
| Planned | Not built. Listed so nobody thinks it is there. |
**The demo household.** The fictional Sharma family. 39 receipts from
May 2 to Oct 24, 2026: Trader Joe's 18, Costco 12, Safeway 9. Total
spend is $1,115.42. That is $1,090.42 of groceries, plus a $25.00 gift
card (bought for someone else). One chicken was returned for $12.30.
The demo slides all dates to the current week. The screenshots were
taken on Oct 29, 2026, so a date such as "Oct 19" in a picture is a
seed date moved forward. Money amounts never move.
**Where prices come from.** Every price in this page comes from the
household's own receipts. Arya does not scrape stores, read flyers or search the web for
prices, and it calls no store API for prices. (Google Places finds stores for the
store picker, and Kroger's API gives aisle hints; neither supplies prices.)
**Screenshots** are real 390x844 captures of the demo, in
`assistant/docs/img/`.
---
## Journey 1. First receipt, and the list starts to write itself
**Person.** Priya Sharma. She shops every Saturday. She writes lists in
her head and forgets milk.
**Goal.** Put in last week's receipts and see what Arya learns.
**Status.** Shipped. Reading the photo or PDF needs a model.
| Step | What Priya does | What Arya shows |
|---|---|---|
| 1 | Opens the app, taps **Try it now** or creates an account. | Landing: "Your grocery run, smarter." Three real rows (Whole Milk "Running low"). |
| 2 | Lists, **Receipts** (the group at the bottom), **Add receipts**. She picks the Costco PDF from Oct 17. More than one file is fine. | One progress line per file, then one summary: filed, already filed, could not read. |
| 3 | Waits a few seconds. | Muse Spark reads the lines. Code adds them up. The receipt shows a check mark and its total only if the lines match the printed total. |
| 4 | Files the same PDF again by mistake. | "Already filed." Arya compares a fingerprint (store, date, lines, total) and does not count the trip twice. |
| 5 | Wants one old receipt. In **Receipts**, types the store in the search box, opens **Date** and picks a range, and picks **Store A to Z** or **Highest total** in **Sort**. | A line shows how many receipts match and their total. **Clear** removes the filters. |
| 6 | Spots a misread line on a filed receipt. Opens it, taps the line, and changes the name, quantity or price in **Fix this line**. | The line shows an "edited" tag. A new name also fixes the item on her other receipts. The printed total never changes. If the lines stop adding up, the receipt says so and no longer counts as checked. **Undo** puts the old values back. In a group, any member can fix a line, but only the person who filed the receipt can delete it. |
| 7 | Opens **Today**. | **Running low** lists items that are due. |
**Sample data: the Costco receipt, Oct 17**
| Line | Price |
|---|---|
| Pasture Eggs 12 ct | $6.47 |
| Jasmine Rice 10 lb | $13.49 |
| Paper Towels 6 ct | $9.99 |
| AA Batteries 8 ct | $8.99 |
| Kirkland Organic Milk 0.5 gal | $4.14 |
| **Total** | **$43.08** (the lines add up, so it posts) |
**What she learns on Today.** After 26 Saturdays the pattern for Whole
Milk is clear: "Every ~7 days; last bought 10 days ago", window Oct 24
to Oct 27. Sourdough Loaf has the same line. Each row has an **Add**
button.
**The rules behind it**
- A pattern needs 3 kept buys before Arya calls an item "running low".
- Returns count as not bought. The returned chicken (Oct 5, $12.30)
never comes back as a suggestion.
- Bad lines go to a review queue. Nothing posts in silence.
- The gift card is tracked under "Bought for others". It is never
suggested.
---
## Journey 2. Plan a Costco and Trader Joe's trip
**Person.** Priya, on Friday evening.
**Goal.** One list per store, with the usual things already there.
**Status.** Shipped. Typing and tapping are code. Voice and photo adds
need a model.
_Update 2026-11-02: "Everything else" is no longer an automatic list (it exists only if you make one), and Lists now sits under More. The steps below and their screenshot show the earlier layout and are due a re-walk._
| Step | What Priya does | What Arya shows |
|---|---|---|
| 1 | Taps **Lists**. | Cards: **Costco** (2 items), **Trader Joe's** (3 items), **Everything else** (1 item). |
| 2 | Opens **Trader Joe's**. | Avocados 4 ct, Pasta 1 lb x2, Hummus. Under them, suggestions from her patterns for this store. |
| 3 | Types "mlik" in the add row and taps **Add**. | The add applies at once, with Undo. Arya links the typed name to the tracked item, so "mlik" becomes Whole Milk 1 gal. |
| 4 | Taps Whole Milk to see why. | The **why sheet** (below). |
| 5 | Swipes Basil to **Move**. | Basil goes to the Trader Joe's list. She can use **Select** to move many items. |
| 6 | Taps the pencil by the list name. | The name edits in place. |
**Sample data: lists after the setup**
| List | Items |
|---|---|
| Costco (store: Costco) | Paper Towels 6 ct, Jasmine Rice 10 lb |
| Trader Joe's (store: Trader Joe's) | Avocados 4 ct, Pasta 1 lb (x2), Hummus |
| Everything else | Basil |
**The why sheet for Whole Milk** shows Arya's reasons in plain words:
"You buy it every ~7 days. 25 times since May." "Last bought 10 days
ago. That is 3 days past your usual." Price lines follow in Journey 4.
**Also on this screen**
- Typing "milk" finds "Whole Milk". Typing a non-grocery item, for
example "light bulbs", keeps it under Other errands.
- Items you add are not changed without your tap. Riskier changes
(out of stock, never again, snooze) ask first.
- Planned: none for this journey.
---
## Journey 3. In the store with Quest
**Person.** Priya, Saturday at Trader Joe's.
**Goal.** Get everything on the list without walking back.
**Status.** Shipped. A quest needs no model. "Find it for me" needs one.
Express, the plain-checklist mode, no longer has an entry point. Its
screen stays only for runs started before it was removed.
| Step | What Priya does | What Arya shows |
|---|---|---|
| 1 | Opens the Trader Joe's list. Taps **Start shopping · 3 items**. | A first-time how-to: Find it, Grab it, Snap the receipt. A list with no items sends her to Add items instead. |
| 2 | Taps **Got it**. | "Trader Joe's · Quest", a points counter, "0 of 3". The store's layout is labelled as a typical layout until shoppers map it. A card says "Pasta 1 lb x2, Pasta Shelf". |
| 3 | Taps **Grab**. | +10 points. She can put an item back. |
| 4 | An item is not on the shelf. She taps **Not here?** and confirms. | The report is shared. Once three different shoppers have reported the same item at a store, other shoppers who have it on their list and use that store see a "3 shoppers say not here" alert on Today. |
| 5 | Taps **Find it for me** and aims the camera at a shelf. | SAM 3.1 boxes candidates, Muse Spark checks each one, and the match lights up for one tap. |
| 6 | Taps **I'm done shopping**, then photographs the receipt. | +50 points. The receipt is filed with the same checks as Journey 1. |
**Rules.** No clock. No lives. No way to lose. Quest is optional. The
list works without it.
**Sample data.** "Trader Joe's · Quest", 3 items on the card: Avocados 4
ct, Pasta 1 lb x2, Hummus.
---
## Journey 4. Compare prices from my own receipts
**Person.** Arjun Sharma, who pays the bills and wants to spend less.
**Goal.** Find out where milk is cheaper, and what has gone up.
**Status.** Shipped. Arya uses only the household's own receipts.
Community prices, **Deals** and **Arya Shoppers** are Planned. Today the
opt-in choice is saved and nothing else is built. There are no price
charts. Trends show as words and percent.
| Step | What Arya shows | Where |
|---|---|---|
| 1 | "Whole Milk 1 gal: $4.17 at Safeway. You usually pay $4.56 at Trader Joe's." | Today, **Store alerts** |
| 2 | "Up 10% since May. $4.31 then, $4.76 now, at Trader Joe's." | Whole Milk why sheet |
| 3 | "Cheaper at Safeway: $4.40. You paid $4.76 at Trader Joe's. Safeway on Oct 19." | Whole Milk why sheet |
| 4 | "Similar items you bought. This one: 3.4 cents/oz. Kirkland Organic Milk 0.5 gal, $4.14, Costco, 6.5 cents/oz." | Whole Milk why sheet |
| 5 | The **(i)** button on each line gives the small print, for example how many buys the number comes from. | Why sheet |
**Sample data: milk price by store (own receipts)**
| Store | Buys | Median | Last |
|---|---|---|---|
| Trader Joe's | 16 | $4.54 | $4.76 |
| Safeway | 9 | $4.17 | $4.40 |
Eggs are different. Arya has 12 Costco buys (median $6.03, last $6.47,
up 17% since May) and one Trader Joe's buy at $6.96. It shows no "cheaper
at" line, because there is no better store in the data.
**Spending, "Your prices over time"** (the Spending page, called Analysis before the labels audit)
| Card | What it says |
|---|---|
| Your basket | "Your basket costs 1.5% more than in June. Same 14 items you buy often. About $4.04 more a month." Movers: Sourdough Loaf +6.6%, Pasture Eggs +5.9%, Greek Yogurt +5.3%. |
| Split smarter | "Save about $2.61 a month. Buy Whole Milk 1 gal and Bananas at Safeway." Milk $1.48, bananas $1.13. |
| Spend | Oct 2026 total $165.39. By store (all time, not just October): Trader Joe's $453.55, Safeway $396.21, Costco $265.66. Top item: Chicken Breast, $152.21. |
**Arya's powers.** Settings, About Arya, has a page that lists what Arya already
does and what turns on next, with one line per power. One "Did you
know?" line shows on Today each day.
**Limits.** Arya says "your prices" and never "the store's price". With
no receipt for an item at a store, it makes no claim for that store.
Spark never finds or guesses a price.
---
## Journey 5. A household with Groups
**People.** Priya and her partner Arjun. Each has their own account on
their own phone.
**Goal.** One shared set of lists and receipts. A clean way out.
**Status.** Shipped, on by default in `docker-compose.yml`
(`ARYA_GROUPS=1`, PR #128). Set `ARYA_GROUPS=0` in `deploy/.env` to turn
it off. The demo household cannot join a group, so this journey needs
two real accounts. It needs a deploy before the live site has it.
Live sync uses Postgres LISTEN/NOTIFY when `DATABASE_URL` is set.
It has not been tried against the Supabase session pooler. If the
pooler refuses it, set `ARYA_LIVE_PG=0`.
| Step | Who | What happens |
|---|---|---|
| 1 | Priya | Settings, **Start a group**, names it "Sharma home". Her lists become the group's lists. She is the owner. |
| 2 | Priya | Taps **Invite someone**. Arya shows a 6-character code, for example `K7M2QX`. It lasts 48 hours. Any member can make one. A group holds up to 6 people. |
| 3 | Arjun | Opens **Join a group**, types the code. Arya shows a preview first: who invited, how many people, how many lists. |
| 4 | Arjun | Joins. His lists merge into the group's. Lists with the same name become one list. Both people now see Costco and Trader Joe's. |
| 5 | Arjun | Adds Hummus to Trader Joe's in the aisle. |
| 6 | Priya | Her open list reloads in about a second. A tab skips the echo of its own change. |
| 7 | Both | Receipts from both accounts show in **Receipts**, tagged with the person who filed each one. |
**Leaving**
| Step | What happens |
|---|---|
| 1 | Arjun taps **Leave group**. The sheet says: "You keep a copy of the lists and your own receipts." |
| 2 | He taps **Leave group** again. |
| Data | After he leaves |
|---|---|
| Lists | Duplicated. Arjun has a copy. The group keeps its own. |
| Receipts | Belong to the account that filed them. Each goes with its owner. |
| Running low, patterns, spend | Recomputed for both sides from the receipts each side now has. |
| Personal items (profile, badges, Quest, push, currency) | Never shared, so nothing changes. |
If the owner leaves, the member who joined first becomes owner. When the
last member leaves, the group is gone.
**Not planned.** Roles, per-list privacy, kids' accounts.
---
## Journey 6. Expiry dates and Use soon
**Person.** Priya, unpacking the Saturday shop.
**Goal.** Use the spinach before it turns.
**Status.** Shipped. Reading a date from a photo needs a model. In the
demo, a canned sample read stands in for the photo.
| Step | What Priya does | What Arya shows |
|---|---|---|
| 1 | Taps **+ Snap a date** on Today, then photographs "USE BY OCT 31" on the spinach. | Spark reads the date. Arya shows what it read: "Baby Spinach 16 oz, use by Oct 31". |
| 2 | Taps **Looks right, track it**. | Nothing is saved until she confirms. The photo is never kept. |
| 3 | Looks at Today three days later. | **Use soon** shows Baby Spinach 16 oz with the days left. It shows items up to 7 days ahead. A past date stays for 7 days, then fades. |
| 4 | Taps the row in Use soon. | She can snooze it (it waits 2 days), fix the date or stop tracking. |
| 5 | Buys spinach again. | A new receipt means a new package, so the old date drops. |
**Still good?** Arya asks by itself when an item "usually turns about
now". The demo shows **Still good? · 3 to check**: Baby Spinach 16 oz,
Chicken Breast and 1 more. Priya taps **Still good** or **It expired**.
This part is code, not a model.
**Rule for a past date.** A past date is neutral ("your call"), not a
red alarm.
---
## Journey 7. Ask in the aisle, a reminder, and the Home Screen
**Person.** Priya, in the week.
**Goal.** Get answers without opening a spreadsheet, and get a nudge
before milk runs out.
### 7a. Ask Arya
**Status.** Shipped. **Needs a model.** Without `MODEL_API_KEY` the
Ask tab says it could not answer.
In the Ask tab Priya can type, or tap **Talk** to speak. Arya answers in text only:
| She says | Arya does |
|---|---|
| "We finished the Greek yogurt." | Applies the change and the list updates. |
| "What do we spend on coffee?" | Answers from the ledger (Coffee Beans 12 oz, $104.80 over 26 weeks in the demo). It does not invent a number. |
| "Where is milk cheaper?" | Reads the price facts that code computed from her receipts (see Journey 4). The model only reads them. |
| "What is Arya?" | Answers about Arya itself. |
| A photo (Photo chip under the Ask field) | Opens a chooser over the existing readers (list, receipt, fridge, expiry date), or "Ask about this photo" once. The photo is not stored. |
**During a hunt** a bar "Ask Arya..." sits at the bottom. It opens a
sheet with three quick questions: "Where's the {item}?", "Is there a
cheaper one?", "I can't find it". The sheet knows the store, the item
on screen and what is left, so "this" and "it" work.
### 7b. Push reminder
**Status.** Shipped, live on the hosted app (flag `ARYA_PUSH_REMINDERS=1`,
VAPID keys, daily cron at 13:00 UTC; enabled Nov 2). Setup is in
`assistant/docs/PUSH-REMINDERS.md`. Demo accounts never see it.
1. Settings, Preferences, **Remind me**. Priya turns it on and picks the notice
window (1 to 7 days, default 2).
2. At 9:00 the daily job runs. It sends at most one push per account
per day. No model call. $0.
3. Example push, one item: **"Whole Milk 1 gal is probably running
out"** with "Add it to this week's list? Tap to open your list."
4. Example push, several: **"3 things are about to run out"** with
"Whole Milk 1 gal, Sourdough Loaf, Pasture Eggs 12 ct. Tap to see
them."
5. A cycle is announced once. A new receipt starts a new cycle.
On iPhone, reminders work only after Add to Home Screen (iOS 16.4 or
newer).
### 7c. Add Arya to the Home Screen
**Status.** Shipped (PR #133). Hidden in the demo and once Arya runs
from the Home Screen.
| Phone | What happens |
|---|---|
| Android and desktop Chrome | A card at the top of Today: **Add Arya to your Home Screen. Full screen, with reminders.** The **Install** button runs the browser's own prompt. |
| iPhone | The card says **Show how**. It opens 4 steps, including "leave Open as Web App on". Off means a plain Safari tab, and then push cannot work. |
The card shows from the second visit, only when the list has an item.
If Priya taps the cross, it stays away for 30 days. Settings always has
an **Add to Home Screen** row while installing is possible.
---
## Status summary
| Capability | Status |
|---|---|
| Receipts (photo, PDF, many files), duplicate guard, returns netted | Shipped |
| Receipts search, Date menu, Sort menu | Shipped |
| Running low, why sheet, patterns | Shipped |
| Named lists, smart add, swipe to Move, Select | Shipped |
| Start shopping, Quest | Shipped |
| Prices from own receipts (store compare, trend, similar items, Your prices) | Shipped |
| Groups with live sync and Leave | Shipped, on by default |
| Expiry dates, Use soon, Still good? | Shipped |
| Ask Arya (list, aisle) | Shipped, needs a model |
| Push reminders | Shipped, live (flag `ARYA_PUSH_REMINDERS` on since Nov 2); you opt in, and on iPhone add the app to the Home Screen first |
| Add to Home Screen | Shipped |
| Supabase sign-in (email code, Google) | Shipped, live (flag `ARYA_SUPABASE_AUTH` on). New sign-up is closed by an Ops flag, so only existing and approved accounts get in |
| Community prices, Deals, Arya Shoppers | Planned (opt-in choice saved, nothing built) |
| React app (`web/`) at `/` | Shipped (`/next/...` answers 302 to `/...`) |
---
# Arya Quest: doc pack
Quest is the optional game layer of Arya, a grocery app that learns how you
shop from your receipts and helps shoppers help each other (see the root `README.md`). Team Nightshift,
Indie track, submission Nov 4, 2026, 9:00 PM ET.
Quest turns your grocery list into a treasure hunt in your real store. Pick the store. Arya sorts your list by where you usually buy
each thing. Walk the hunt through real photos of the aisles and grab each
item (points only go up, no clock). With **Find it for me**, you point
the camera at the shelf and the item lights up. Snap the receipt at checkout. It banks your
points and teaches Arya your household, so next week's list writes
itself.
An unmapped store still plays, as a labelled typical layout. A guided
camera scan maps it for every shopper after you.
## Loop
**Plan → Find → Grab → Receipt.**
1. **Plan.** The list arrives in any form (photo, paste, typing, voice,
Muse chat). Pick the store. The list is sorted for it from your
receipts.
2. **Find.** Shoppers' photos where a store section is mapped (people
pixelated by SAM 3.1), and a labelled typical layout elsewhere. Find it for me: SAM 3.1 boxes candidates,
and Muse Spark verifies them.
3. **Grab.** +10 per grab, milestones, a full-haul bonus. Shortage
reports and sightings feed a live store map.
4. **Receipt.** Muse Spark reads it (Spark first; see the model map in
`JUDGES.md`). Code checks the arithmetic. The receipt then
feeds patterns, spend and returns.
## Documents
| File | What it holds |
| --- | --- |
| [GAME-DESIGN.md](/docs/quest/game-design) | Rules, loop, rewards, Spotter, community |
| [UX.md](https://github.com/vibhortayal/arya/blob/main/docs/archive/quest/UX.md) | Sessions, flow, store view, intake. Oct 27 rules, archived on 2026-11-04; the wireframes are the design of record |
| [wireframes.html](https://github.com/vibhortayal/arya/blob/main/docs/quest/wireframes.html) | Clickable design of record |
| [game-flow-prototype.html](https://github.com/vibhortayal/arya/blob/main/docs/quest/game-flow-prototype.html) | The game flow, tap through on one phone |
| [ARCHITECTURE.md](https://github.com/vibhortayal/arya/blob/main/docs/quest/ARCHITECTURE.md) | Data planes and pipelines |
| [META-PRODUCTS.md](/docs/quest/meta-products) | Each Meta product and its job |
| [JUDGE-PLAYBOOK.md](/docs/quest/judge-playbook) | An 8-minute path through the live demo |
| [PITCH-SCRIPT.md](https://github.com/vibhortayal/arya/blob/main/docs/quest/PITCH-SCRIPT.md) | The spoken script. The early game-first [PITCH.md](https://github.com/vibhortayal/arya/blob/main/docs/archive/quest/PITCH.md) was archived on 2026-11-04. |
| [BUSINESS.md](https://github.com/vibhortayal/arya/blob/main/docs/quest/BUSINESS.md) | Who it is for and who else is out there |
| [BUILD-PLAN.md](https://github.com/vibhortayal/arya/blob/main/docs/archive/quest/BUILD-PLAN.md), [SUBMISSION.md](https://github.com/vibhortayal/arya/blob/main/docs/archive/quest/SUBMISSION.md) | Road to Nov 4 and the package checklist. Archived on 2026-11-04. |
| [DISCUSSION.md](https://github.com/vibhortayal/arya/blob/main/docs/quest/DISCUSSION.md) | My steering, in order |
Current state: `STATUS.md` at the repo root. Every document in the
repository, grouped by reader: [`docs/README.md`](https://github.com/vibhortayal/arya/blob/main/docs/README.md).
---
# Game design
## Core loop
**Plan → find → grab → snap the receipt.** The grocery list is the quest
log. The real store is the dungeon. The items on their real shelves are
the treasures. Mobile first. The keyboard is for the demo only.
## My rules (2026-10-28)
- No lives, no game over, no clock, no way to lose. Points only go up.
- Dopamine during the hunt and after it, never stress or pressure.
- Upgrade the existing hunt; no separate game.
- The game must keep feeding four things: community shortage reports,
store layout and shelf maps, consumption and spend from receipts and
list completion (plus receipts for returns), and fun with kids or a
partner.
## Plan a hunt
1. **Pick the store.** Each store says what you usually buy there,
counted from your receipts ("my milk is from TJ, eggs from Costco").
2. **Move items into the quest.** The Arya list, sorted for that
store. Items usually bought here, and new items, start ticked. Items
usually bought elsewhere stay on the list. Usuals not on the list
are suggested. ✕ removes an item from the list (Undo). Typed extras
join the list. Planning is list management, not a second list.
3. **Check my list** appears only when Arya has a question.
4. A quest starts. Express, the plain-checklist mode, has no entry point
any more; see Modes below.
## The hunt (Quest)
- One store view, picked by the store: real photos of a mapped store
(people pixelated by SAM), shelf cards anywhere else.
- The next-item card suggests a stop; any stop can be grabbed in any
order. A fresh shopper photo shows the item boxed when one exists.
- **Find it for me:** the camera scans for that one item. SAM 3.1
boxes candidates and Muse Spark checks each crop. The item lights up
and one tap grabs it. A miss never says the item is not there.
- Tools: Scan shelf, Not here?, How to
play (?), sound.
- Put back any grab; nothing finishes by itself.
- I'm done shopping works any time. Leftovers are saved for the next trip or dropped. Nothing counts as failure.
- How to play shows once, on the first Quest hunt on a phone.
## Rewards
| Event | Reward |
| --- | --- |
| Grab | +10, the chip bumps, a fun fact |
| Combo count | Combo: each grab of the suggested stop adds 1. It never resets and put back never lowers it. |
| Boss | +40, only when the boss item was grabbed; a boss is an item you reported missing here on 2 earlier trips. |
| Halfway, full haul | milestone; full haul +25 and confetti |
| Receipt snapped | +50; the trip's points bank |
| Level line crossed | level-up moment (L2 at 100, L3 at 300, L4 at 600, +100 more each level) |
Sound and buzz are opt-in, off by default. Express stays quiet.
Offline grabs queue on the phone and sync when the signal returns.
## Modes
Quest was designed with a second mode, Express: the same list as a
plain checklist in walking order, with no points. Express no longer has
an entry point (see `STATUS.md`); its screen stays only for runs that
started before. The list works without Quest, and shopping from the
list needs no game.
Quest is a mood, not an age. With a child holding the phone, the kid
hunts and grabs while the adult handles quantities and checkout.
## Store drift: Spotter
Every hunt is a shelf survey. A grab proves a product is there; "Not
here?" (Empty shelf / Something else here / Can't tell, optional photo)
says it is not. A report edits nothing. It flags the stop and opens a
window. A later grab at the stop means a stockout. A second player
finding it elsewhere means a move, applied as a shelf-graph edit.
Find it for me sightings add the same signal.
Points on truth, never on reports: +2 for a true stockout, +10 for a
verified move. Eagle Eye badge (1 / 10 / 50 verified spots), Store
Guardian per store. Status only, never cash.
## Community
Every metric is receipt-verified: the receipt is the referee.
- **Boards:** Fastest Hands, Clean Sweep, Top Contributor, Big Haul,
World Mapper, Globetrotter.
- **Badges:** Egg Baron, Banana Republic, Dairy Royalty, Midnight
Shopper, Speed Demon, Return Regret, First Mapper, Perfect Week.
- **Races:** Shop-Off (first verified receipt wins), Split the List
(not built), Clue Hunt (rollout).
- **Friends:** handle-based; no chat, no DMs, no lists shared. Run
notifications are designed (Discussion 24), not built.
- Handles, never real names. Household boards are private. Spend boards
are opt-in.
## Contribution
Store videos earn permanent first-mapper credit. Receipts stay
private; their reward is a smarter list and a hunt that plans itself.
## Never
- No random spawns or sponsored placements in routes.
- No purchases, external links or chat in the hunt.
- No pictures of people kept: frames are kept only after SAM pixelates
the people it detects.
---
# Map my store ("Teach Arya your store"): design
Me, 2026-10-28: replace "take a photo or record a video" with a guided
mapper. Map only the aisles you use, over many visits. Many partial maps
merge into one store map. A "not found" or "map is outdated" asks for one
specific capture. The engine keeps the best images.
Wireframe (v2): `docs/quest/map-my-store-wireframe.html`. Code: `api/storemap.py`,
`assistant/web/src/screens/Mapper.tsx`.
## Coverage model
A **segment** is one side of one aisle, or one wall or freezer run. A store
map is its list of segments. Shoppers add segments ("Add an aisle"). No store
starts seeded: the old Trader Joe's walkthrough seed was removed (2026-10-29),
and a saved section that only had a walkthrough baseline is dropped on load.
Status of a segment, from its best capture:
- **unmapped**: no capture, no baseline.
- **mapped**: best capture is 60 days old or less and no open flag.
- **aging**: best capture older than 60 days.
- **stale**: a "not found" or "outdated" flag is newer than the best capture.
Coverage is mapped segments over known segments. The denominator grows as
people add aisles, so the bar means "of what is known so far".
## Capture prompts
`next_prompt()` returns one capture: kind, title, why, numbered steps, time.
Order of need: stale, then unmapped, then aging. The shopper's **focus**
("Mine" aisles, saved per account) is served first. Two or more unmapped
segments in one zone get one **walk** prompt (about 20 s, the phone sends its
6 sharpest of 12 sampled frames as photos; no video upload). Otherwise a
**photo** prompt (about 10 s). A stale prompt names the missing item.
## How contributions merge
One shared file per store (platform plane, like the stores directory).
Segments merge on `zone_key`: "Aisle 4", "aisle #4 - Pasta" are one zone;
other labels merge on normalized words, minus a trailing "shelf". Each
segment keeps its 3 best captures. Contributors are stored as a hash, never
the username. Demo visitors write to their sandbox overlay, never the shared
map.
## Framing (me, 2026-10-28)
The shopper teaches Arya so Arya can help them: "Arya knows 64% of
Trader Joe's", and every capture says what it unlocks for you (a real
shelf photo in your next hunt). Helping other shoppers is the bonus.
## Capture: no retakes, ever
I said: "I click pictures, then app says not good... this will cause
friction." So the shopper opens a live camera and the app does the work:
- **Photo:** the phone grades every frame (sharpness, light), shows one
live hint ("Hold still…", "A bit dark. Face the shelf light."), and
snaps the best frame after 1.5 s steady, or the best seen by 6 s.
- **Walk:** the sharpest frame of every 2 s is kept; no video upload.
- The server never refuses a shot. A weak one is kept as a **partial**
(`weak: true`). The segment stays "aging", the next better capture
replaces it, and the engine asks again on a later visit.
- If the people scrub fails, nothing is kept and the app says "No need
to redo it." It never asks for a retake.
- Without a live camera (old browser, permission denied) the system
camera is the fallback, under the same rules.
## Splitting a long aisle
The walk's length decides: about 8 s of walking per section, 1 to 6
sections. The app calls `POST /api/storemap/split` and spreads the
walk's frames over the sections in time order (front / middle / back,
or first half / second half). Parts keep the zone key, so sightings and
"not found" reports still find them. A split is idempotent, so a second
shopper walking the same aisle fills the same sections. Old whole-aisle
captures are dropped on split; the walk that caused it fills the parts.
## When Arya asks
- **After a hunt:** one skippable card on the score screen ("Got 20
seconds? Teach Arya Aisle 4. +10"). `GET /api/storemap/next?items=`
puts an aisle where something on today's list was seen first. "Not
now" mutes the store for 7 days (on the phone).
- **Can't find it:** a photo attached to the "Not here" report is
ingested straight into the flagged section (no extra screen). Without
one, the toast offers "Open camera". An item with no known aisle asks
"Which aisle are you in?" as one tap.
## Rewards (no pressure)
+10 for a section that becomes mapped, +3 for a refresh or
confirmation. Titles: Scout 1, Pathfinder 5, Cartographer 15, Store
whisperer 40 sections. Each capture shows the store going from a% to
b% and the sections to the next 25% step. Credit is per account
(`map_credit.json`); nothing is ever taken away, and there are no
streaks or timers.
## How the engine chooses frames
All deterministic, $0, except two model calls per upload in the route:
SAM 3.1 people scrub (fails closed) and "grocery shelf" boxes for fill
(summed over bays, measured on the scrubbed JPEG).
1. Grade: sharpness (Laplacian variance), exposure (mean grey level),
shelf fill. A weak frame is kept as a partial, never refused.
2. Score = 0.5 sharpness + 0.3 exposure + 0.2 fill (fill is neutral if the
check failed).
3. Value = score x freshness; freshness halves every 30 days.
4. Same view as a kept capture (dhash distance 6 or less) is a
**confirmation**: it refreshes the old capture's date, and swaps the image
only if clearly sharper (+0.05). That clears a stale flag.
5. Keep the top 3 by value. The best one is shown for that segment.
## Refresh loop
`POST /api/storemap/outdated` or a "Not here" report marks a segment stale.
For an item, the segment is where the item was last seen (`note_sighting`
from Find it for me). If unknown, the item is parked as unplaced and the
shopper is asked which aisle they are in. The report response carries
`map_task`; the app offers "Take the photo".
## Privacy
Only scrubbed frames are kept. No opt-in screens yet (deferred by me
for the hackathon). Raw frames never touch disk.
## Not built yet
Item-level locations from captures (Muse Spark labels); automatic
`resolved` transitions for reports; stale flags expiring when no one
confirms; mapping silently from Find it for me frames during a hunt.
---
# The agent connector (MCP)
Arya does not compete with the list apps and notes you already use.
It collects from them. The collection point is your own assistant.
Any MCP-capable agent connects to Arya once, with your account token.
From then on, anything you capture anywhere (a note, a reminder,
a chat message, an email receipt) can become part of your Arya list.
Your assistant can then answer from Arya's model instead of a stale
note.
Arya's list is the shared list: the one place the pattern engine can
compute on. Other apps are doors in and mirrors out, never a second
source of truth.
## Connect
I tested this on my own Muse (2026-11-02), with no terminal and no settings
file. I asked the Muse agent to connect to Arya. It asked for the code, opened
a secure connector-credentials vault for me to paste it into, and the
connection worked. The code never goes in the chat.
1. In Arya, open More → Settings → About Arya → Use Arya in Muse and tap "Make my
code". Copy it. It is shown once and starts with `arya_`. A new code
replaces the old one. (The token works with any MCP client, not just Muse.)
2. In your AI app, ask it to connect to Arya at
`https://arya.teamnightshift.org/api/mcp`. When it asks for your code,
paste it into the app's secure credentials step, never into the chat.
Apps that take a server list instead use transport streamable HTTP
(stateless; one POST per message) and the header
`Authorization: Bearer `.
3. Ask it "What is on my Arya list?" Then try "Add oat milk to my Arya
list" and "What am I running low on in Arya?" The new item also shows on
the Lists screen in Arya.
4. If it says it cannot connect, make a new code (the old one stops working)
and paste it into the vault again. Tap "Turn off this code" in Arya to
switch the connection off.
The token acts as your account. It is accepted on the MCP door and on
the app's routes under `/api`, but never on `/auth` (it cannot mint tokens
or delete the account). A short list of routes is closed to tokens and
answers 403: calls that spend model credits in bulk (chat, photo and
shelf reads, avatars, the judge), groups, friends, push, feedback,
settings changes, price sharing, account export and receipt delete
through the app (`ARYA_TOKEN_GUARDS`). Revoke it from
the same screen at any time; minting a new one replaces the old.
### Backup: the Muse Code settings file
Only if the agent flow above is not available. Muse Code reads remote MCP
servers from `~/.config/muse/settings.json` (per the Meta docs at
dev.meta.ai/docs/muse-code/extending; not yet run end to end). Put the code in
an environment variable so it is not typed into a file or chat
(`read -rs ARYA_TOKEN; export ARYA_TOKEN`), then add:
```json
"mcp_servers": { "arya": { "transport": "streamable_http",
"url": "https://arya.teamnightshift.org/api/mcp",
"headers": { "Authorization": "Bearer ${ARYA_TOKEN}" } } }
```
Keep `"schema_version": 1` in the file. Run `muse` from the same terminal
window and type `/mcp` to see the six tools.
## Limits
- 600 calls per account per hour (`ARYA_MCP_RATE`), shared by the connector,
the Lists API (`docs/API.md` in the repo root) and every other route a token
reaches; over it, 429 with `Retry-After`. 256 KB per request,
50 items per `add_items`, 200 lines per `file_receipt`.
## Tools
| Tool | What it does |
|---|---|
| `get_snapshot` | Running-low items, spend and returns totals, trip count, top items. |
| `get_list` | The current shopping list, with quantities, check-offs and each item's list, plus the names of all lists. |
| `add_items` | Add items by name, as a person would say them, to a named list (`list`). Returns the items added and that list after the add. |
| `check_item` | Check an item off (or uncheck it) by name. |
| `file_receipt` | File a receipt the agent has already read. Arya verifies the arithmetic, files the lines, and learns from them; it spends no perception tokens of its own. |
| `get_receipts` | Recent filed receipts: merchant, date, total. |
Every item goes on a named list. The account's list names come up front, in the `initialize` instructions and in the `add_items` description from `tools/list`, so an agent asks "which list?" once instead of learning the names from a failed add. With one list, adds go there. The Lists API (`/api/v1`, see `docs/API.md`) is list-aware: it reads and writes each list by id.
## The showcased client
Muse Spark is the reference client. In chat, "add oat milk to my
list" files through `add_items`, "what am I low on?" answers from
`get_snapshot`, and a receipt photo read in chat files through
`file_receipt`. The protocol is the standard one. Nothing in the
connector is specific to a single assistant.
---
# Glossary
The words Arya uses, in plain terms. Names in quotes are the exact words on screen.
## In the app
- **Today**: the first tab. What is running low, store alerts and what to use soon. You can change what it shows with Customize your "Today" screen.
- **Navigation tab**: the row of tabs at the bottom of the phone: Today, Spending, Quest, Ask and More. "Customize navigation tab" changes which ones sit there.
- **Lists**: under More, one card per store list, with suggestions underneath. Every item sits on one list.
- **Suggestions**: items Arya thinks you may need, each with a reason. Arya never adds to a list by itself. You tap Add.
- **Running low**: staples that are due again, worked out from how often you buy them.
- **Use soon**: food you bought that should be eaten soon. "Used it" moves an item back to Suggestions.
- **Ask Arya**: the chat, on the tab named "Ask". Ask how to do something in Arya, or about your lists and spending.
- **Help tab**: the ask box, first steps, a how-to topic for each feature, and links to the Arya guide, Arya's powers, the Tour and Send feedback.
- **Arya**: a grocery app that learns how you shop from your receipts and helps shoppers help each other.
- **Arya guide**: the swipeable cards that explain each feature.
- **Tour**: the first-run walk-through, shown as bubbles over a dark screen.
- **Quest**: the optional game. It turns a shopping trip into a hunt through a model of your store and ends on a photo of your receipt. It has no clock and no way to lose.
- **Use Arya in Muse**: the Settings, About Arya row that connects the Muse assistant to your Arya account through the MCP connector.
- **Demo Town**: the simulated neighborhood of invented stores the demo household shops in. It is labelled as simulated.
## How it works
- **Model proposes, code checks**: a model reads the receipt. Plain code then checks every line against the printed total before anything posts.
- **Staple judge**: decides whether an item is a staple you buy again and again. Muse Spark is the main seat and Muse Spark Open a second seat.
- **Curator**: proposes product merges and splits for you to apply.
- **Crowd map**: the aisle layout built from shoppers' own store scans.
- **MCP connector**: lets an AI assistant add items and read lists through Arya's MCP endpoint.
- **Lists API**: a small bearer-token API to read and write your lists from a script. See the API reference.
- **PWA**: a web app you add to your phone's home screen. Arya is a PWA only.
## The Meta models
Model ids, jobs and measured numbers live in [the model map on the judges page](/docs/judges#which-model-does-which-job). This list only explains the words.
- **Muse Spark**: reads receipt photos first, and also runs the staple judge, the Curator, Ask Arya and photo reading. Arya uses the Standard tier.
- **Muse Wonder**: re-reads a receipt only when the arithmetic checks flag it, or when Spark fails.
- **SAM 3.1**: finds shelves and signs in a store scan, boxes items for Find it for me and pixelates the people it detects.
- **Muse Voice Transcribe**: turns speech into text for the list mic and Ask Arya.
- **Muse Image**: made the grab-celebration characters in Quest, once.
- **Muse Spark Open**: the open-weights model, hosted on Baseten. It is the second seat of the staple judge and sees item names only (no prices, dates, stores or account details).
- **Meta Model API**: the one client every server call goes through, with a daily spend ceiling.
## Not live
Community prices, Deals and Arya shoppers (seeing other shoppers nearby) are not live. Muse Glimmer is not used. Muse Code, running Muse Wonder, wrote the first version of the engine. It is not part of the shipped app.
---
# Design decisions, and what comes next
I built Arya as one well-defended single-server app on purpose. The window was for the product, not for a fleet of servers. This page has three parts: what is built and holds up for a few dozen people at once, what I know is weak, and the plan to grow past that. The dated choices are in the [decision log](https://github.com/vibhortayal/arya/blob/main/docs/DECISIONS.md), what was made and when is on the [build page](/docs/what-we-built), and the unbuilt ideas are in the [roadmap](/docs/roadmap). What runs today is in [Architecture](/docs/architecture/overview).
## What is built?
- **Code does money, timing and lists at no model cost.** The ledger, patterns and derived list are rebuilt from the receipts, with no model involved, so they cannot drift. Every model call goes through one module that enforces the personal-data rule. (Architecture 0.3, 0.4)
- **Safe writes on one box.** Atomic JSON writes, a per-file lock, a per-request memo, and a full Postgres copy of every account document on the live site. (Architecture 0.5)
- **Layered spend control.** A daily ceiling for the app, a separate one for the demo, a per-visit share, per-account monthly and hourly limits, and an Ops switch to pause paid reads or change the photo reader without a deploy. Every call writes a ledger row. ([Model policy](/docs/architecture/model-policy))
- **Background work that survives a restart.** Receipt batches and store scans keep a pending list on disk and resume after a deploy.
- **Live sync that already spans processes.** Shared-list changes use Postgres `LISTEN/NOTIFY`, one listener per process. (Architecture 0.8)
- **A demo that cannot touch real data.** Each visit writes to an in-memory sandbox with a size cap.
- **Security basics.** Hashed session and token secrets, a strict script CSP, upload caps at the proxy, throttled sign-in, and people pixelated out of shelf photos before anything is stored.
- **Measured.** A [scale simulator](https://github.com/vibhortayal/arya/blob/main/assistant/docs/SCALE-SIM.md) and a read-only performance pass on 2026-11-02. The Today screen takes about 65 ms alone, about 1.1 s with 10 people at once and about 4 s with 30. That was on a development machine, not the live server.
## What do I know is weak?
These are the findings from my own architecture audit of 2026-11-01 that still hold. Each is tied to a phase of the [scalability design](/docs/architecture/scalability). Two audit items are already closed: `/health` reports the running commit, and the photo reader is Spark first (see the [model map](/docs/judges#which-model-does-which-job)). That second change took the cost pressure off receipt batches.
| What | Why it matters | Fixed in |
|---|---|---|
| One process, one interpreter lock, no cap on model reads in the request thread pool | A burst of photo reads can slow pages that never call a model | Phases 4 and 6 |
| The spend guard reads the whole ledger file on each paid call; password sign-in hashes on the event loop | Both block other requests while they run, and the ledger grows with use | Phase 2 for the ledger; the hashing is not in the plan yet |
| Spend checks are "check, then spend", with no reservation | Parallel reads can overshoot a limit | Phase 6 |
| The ledger is an estimate from a price table, never reconciled with the provider's console | What Arya thinks it spent and what was billed can differ | Not in the plan yet; a manual weekly check |
| Postgres mode stores documents whole, with no version column or pool | Two processes could lose an update silently | Phase 2, before any second process |
| Sessions, demo visits, rate buckets and the receipt queue live in memory or files | A deploy drops every demo visit | Phases 3 and 5 |
| One worker reads every account's receipt batch in order; no cap on batches parked on disk | One big import makes the next person wait | Phase 5 |
| Store-map photos are base64 in one JSON file rewritten per photo | The first wall the simulator found | Phase 7 |
| Saves mirror the whole document to Postgres inside the request | A slow database slows a list tick | Phase 1 |
| `restore.sh` wipes the data folder before extracting; the off-site backup is set up by hand | A bad archive or a lost VM could lose data | Phase 0 |
| Tests run in CI only on demand (my choice, to keep the loop fast) | A broken merge can reach the server unless someone runs the suite | Not a scaling item; revisit after judging |
| Concurrency rules are by convention (each writer must remember the lock) | Easy to get wrong as more people touch the code | Phase 2 |
## What is the plan to grow it?
[Scalability design](/docs/architecture/scalability) has nine phases (0 to 8, the numbers the table above uses). Each phase is switched on by settings. Each has a rollback and a number to measure:
0. Baseline: request timing, a metrics endpoint and a live load test.
1. Hot cache: a short-lived cache, and the Postgres mirror off the request path.
2. Postgres master: Postgres as the master copy, with versioned updates.
3. Shared short-lived state: sessions, demo visits and rate buckets in a shared store.
4. Several processes.
5. Queue and workers: a job queue for receipt, scan and reminder jobs.
6. Model gateway: concurrency caps and spend reservation.
7. Object storage for photos.
8. Cloud provider.
Versioned writes come before a second process, and nothing runs before judging ends. It is a plan, not a promise.
---
# Arya architecture (as built)
Status: living document, revised 2026-11-04. Section 0 is the one-page map with the diagram. Sections 1 to 13 are the design record. SPEC.md states the product rules. This document states
how the built system is put together. Where the two disagree, SPEC.md
wins for behavior and this document wins for structure.
## 0. The whole app on one page
*The diagram is `assistant/docs/architecture.svg` (plain SVG, light and dark),
redrawn and checked against `main` on 2026-11-04. The sections below say what
each box does. Their facts are checked against `main` on 2026-11-04, except the
live-site values in §0.11 and §10, which come from the Ops screen.*
**In one paragraph.** A phone-first PWA (React, built with Vite) talks
to one FastAPI process, which also serves the built app at `/`. Caddy fronts it with TLS. Code does the
money, timing and list logic at $0. Meta models read photos, judge
staples and answer questions, and every call to them goes through one
module (`api/meta_client.py`). State is one JSON document per area, kept
on a Docker volume, in Supabase Postgres, or both.
### 0.1 Front end (`assistant/web/`)
- React 19 and TypeScript, built by Vite into `web/dist`. The API serves
the build at `/` and the static folder `assistant/pwa/` (images, store
models, three.js, `scene.js`, the manifest, icons) beside it; the build
wins on a name clash. `index.html` and `sw.js` are sent with
`Cache-Control: no-cache`. The old address `/next/...` answers 302 to
the same path at `/`.
- One screen per file in `web/src/screens/`, listed in
`screens/registry.tsx`. Navigation is a stack (`app/nav.tsx`); the
session (user, demo, currency, language) is `app/session.tsx`. Shared
parts: `web/src/ui/` (sheet, toast, info sheet, the store view) and
`web/src/lib/` (API calls, formats, stores, language).
- The hunt, Express, Pack, haul, album, share, mapper, group live sync
(`app/groupLive.ts`), push reminders, powers and the install prompt
are ported one to one from the old plain-JS files. The store view is
`scene.js` (three.js), loaded as a classic script.
- The navigation tab is **Today**, **Spending**, **Quest**, **Ask** and **More**
(the default; it can be customized). More holds Lists, Receipts, Stores, Help
and Settings. Settings opens the Profile page (Group, Preferences with
Reminders, Account, About Arya, Sign out last). About Arya is the `about`
view: Arya's powers, Use Arya in Muse, Add to Home Screen, Keeping Arya
running. (Updated 2026-11-02, #416; this replaced the earlier six-tab layout.)
- Service worker `web/src/sw.js`, built to `/sw.js` (vite-plugin-pwa,
injectManifest). It precaches the built shell and the runtime files of
the static folder. It is network first. It sends `/api/`, `/auth/`,
`/decode`, `/import` and `/eyes` straight to the network, never touches
a write (non-GET) or the live stream `/api/group/live`, and shows push
reminders. It takes control at once and deletes every other cache,
the old `arya-v*` caches too.
- Security: no inline script (the CSP has `script-src 'self'`;
`tests/test_csp.py` checks `web/index.html`); no inline handlers.
- Tests: string pins in `assistant/tests/` (they read `web/src` through
`tests/frontend.py`), vitest tests beside the code
(`web/src/**/*.test.ts*`), and browser runs in `web/e2e/`.
- Style: light theme, greyscale structure and one orange accent; see
`DESIGN.md`.
### 0.2 API (`assistant/api/`)
`api/main.py` is a short file: the app, the `require_login` middleware,
a handler that turns a model failure (`RuntimeError`) into a 502 with a
plain sentence, `/health`, one ordered router list, the Lists API mounted
at `/api/v1`, and the static mount for the PWA. Routes live in `api/routes/`, one router per area:
| Router | What it serves |
|---|---|
| `auth`, `account` | sign-in, registration (closed unless `ARYA_REGISTRATION_OPEN=1`; the Ops flag overrides the env, and sign-up is closed on the live site), email code (Supabase) and Google (Arya's own sign-in, flagged), connector and Lists API token, settings, export, feedback, push opt-in |
| `intake`, `review`, `receiptjobs` | receipt photo, PDF, text, order text, structured JSON, Costco JSON, voice transcribe, Curator, review queue; receipt batches that keep reading while the app is closed |
| `lists`, `items`, `expiry` | named lists, add by text or photo, check, move, judge runs, item pages and flags, expiry dates |
| `home`, `chat` | the Today payload, Ask Arya and its apply step |
| `prices` | own-receipt prices, Analysis insights, Arya's powers |
| `groups` | shared home: create, invite, join, leave, live stream |
| `quest`, `questmap`, `receipts`, `shelf`, `storemap`, `stores`, `eyes`, `avatar`, `friends` | hunts, filing items into a store's zones, receipts with originals, shelf scan, Map my store, store directory, pantry and shelf photos kept as sightings, avatar, friends |
| `mcp_connector` | `/api/mcp`, the tool surface for a Muse agent |
| `lists_v1` (mounted at `/api/v1`) | the Open Lists API: read and write your own lists from another app, bearer token only, its own error format |
| `ops` | the owner's Ops screen: models, limits, accounts, freezes, Ask locks (owner only) |
| `town` | Demo Town routes; the demo account only, and only when `ARYA_DEMO_TOWN=1` |
Older siblings stay beside the package: `album_routes`, `hunt_routes`,
`haul_routes`, `pack`. Shared plumbing is in `api/deps.py` (cost ledger,
budget guard, `new_client()`, rate-limit buckets, JSON helpers),
`api/demo_guard.py` (login middleware, demo sandbox, door caps, the list of
routes a token cannot reach), `api/ask_guard.py` (the Ask input and output
guard and its three-strike lock, §6) and `api/settings.py` with
`api/site_flags.py` (a value set in Ops, then the env var, then the default).
### 0.3 Engines in code ($0)
`decode/service.py` (PDF text layer, scanned PDF pages rendered with
`pypdfium2` and sent to the photo reader, arithmetic reconcile),
`ledger/` (returns netted FIFO, allocations), `patterns/` (cadence and
price by store), `list/` (the derived list and its reasons),
`state.running_low` (one rule for Today and Ask), `prices.py` (see §0.7),
`expiry.py`, `dictionary/`, `optimizer.py` (merges duplicate adds and fixes typos before a hunt) and `cart/` (a sandbox cart proposal; approval is a state change only, and the app no longer shows it).
### 0.4 Models
All calls go through `api/meta_client.py`; the personal-data rule is
enforced there. `open_client.py` is the only other model client.
| Job | Model | Notes |
|---|---|---|
| Reasoning, decode fallback, judge, Curator, Eyes, Ask Arya | `muse-spark-1.3` (Standard; `ARYA_SPARK_TIER=contributor` is an operator option, off) | cached prefixes, low effort |
| Receipt photo reader (default, `auto`) | Spark first; `muse-wonder-1.0` (90 s bound) re-reads only a receipt the arithmetic checks flag | `decoded_by` records which read was kept; `ARYA_PHOTO_READER=spark` or `wonder`, or the Ops "Photo reader" switch, changes it with no code change |
| Celebration characters, generated once | `muse-image-1.0` | |
| Voice in | `muse-voice-transcribe-1.0` | |
| Shelf and people segmentation | `sam-3.1` | people SAM detects are pixelated first, and the frame is discarded if that fails (detection is not perfect and has not been scored) |
| Second judge seat, dictionary enrichment | Spark Open, hosted on Baseten | platform data only; the client refuses personal data |
### 0.5 Storage
`api/state.py` keeps one JSON document per former file (receipts,
dictionary, flags, lists, quest, push...). `ARYA_STORE` picks the
backend: `json` (default; files under `data/users//`),
`dual` (files are the truth, each save mirrors to Postgres) or `pg`
(Supabase table `arya.user_docs`, keyed by owner and document name,
role `arya_app`, via `DATABASE_URL`). The demo account always uses files,
and its visitors write only to an in-memory sandbox overlay. Caches and
big blobs (`shelf_matches.json`, `storemap_img.json`) stay on disk in
every mode. A receipt's original photo or PDF is kept per account.
### 0.6 Auth
Password accounts (PBKDF2 hashes, throttled sign-in, cookie sessions
whose tokens are stored as SHA-256 hashes). Account names allow only
letters, digits and `. _ % + - @`. Email-code sign-in goes through Supabase
Auth and Google sign-in is Arya's own (`api/google_oauth.py`, used when
`GOOGLE_OAUTH_CLIENT_ID` and `GOOGLE_OAUTH_CLIENT_SECRET` are set; otherwise
Google goes through Supabase). Either way the provider only proves an email
address; Arya keeps its own account and cookie (`api/supabase_auth.py`, flag
`ARYA_SUPABASE_AUTH`). The Muse connector and the Lists API use a bearer
token whose hash is stored.
### 0.7 Price intelligence
`api/prices.py` is deterministic and uses only the household's own
receipts: per-store unit prices, a cheaper-store hint when the gap is
real, a trend only with enough buys over enough days, the household's own
basket inflation, a split-the-list suggestion, food waste in dollars.
Nothing is fetched from a store, a flyer, an API or the web. Ask Arya
answers price questions from these precomputed facts. Community prices,
Deals and Arya Shoppers are opt-in, recorded only, and not live.
### 0.8 Groups and live sync
With `ARYA_GROUPS=1` (on in compose), an account in a group reads and
writes the shared stores (lists, flags, product knowledge) in the group's
plane; receipts stay with the account that filed them and every member
sees them. Invites are a 6-letter code valid 48 hours. Leaving copies
the lists. Live sync: every shared save calls `live.changed(group_id)`;
each open app holds one Server-Sent Events stream (`/api/group/live`)
and reloads when its group changes. With `DATABASE_URL` set the change is
a Postgres `NOTIFY arya_group` and one listener thread per process wakes
its streams; without it, or with `ARYA_LIVE_PG=0`, the wake stays
in-process.
### 0.9 Push reminders
Flag `ARYA_PUSH_REMINDERS=1` plus VAPID keys. A phone opts in from
Settings; the subscription is saved in `push.json`. A daily cron line
runs `python -m api.reminders`, which sends at most one push per account
per day for staples that run out inside the notice window. No model
calls. Steps: `PUSH-REMINDERS.md`.
### 0.10 Deploy
One Oracle VM, Docker Compose with two services: `api` (uvicorn,
`deploy/Dockerfile.api`) and `caddy` (TLS, security headers, CSP, plain
HTTP redirects to HTTPS). Data lives in the `arya-data` volume. Settings
come from `deploy/.env` (VM only); compose lists every variable the app
reads, and `tests/test_compose_env.py` fails if one is missing. Deploy is
`git pull` then `docker compose up -d --build` in `assistant/deploy`
(`deploy.sh`); never rsync `--delete`. The api image builds the app
(`web/`) in a node stage and copies `web/dist` in. `backup.sh` and `restore.sh`
handle the nightly backup. CI is manual only
(`.github/workflows/tests.yml`).
### 0.11 Feature flags and settings
Most limits and switches can also be set in the owner's Ops screen (Settings,
then Ops and health). A value set in Ops wins over the env var, which wins over
the code default. The table gives the default in compose. The live site (checked with Ops on
Nov 2) differs in these places: `ARYA_STORE=dual`; push reminders on (daily
cron at 13:00 UTC); Supabase Auth and email sign-in on; Demo Town on; the
Kroger aisle lookup on; own-key sign-up off; photo reader `auto`; Spark tier
`standard`; new sign-up closed (the Ops flag wins over the env). Budgets: $10 a
day for the app, $100 a day for the demo (code default $2), $2 per demo visit (code default $0.40).
Per-account limits are $3 heads-up, $20 monthly pause and $10 hourly freeze.
| Variable | Default in compose | Effect |
|---|---|---|
| `ARYA_STORE` | `json` | `json`, `dual` or `pg` storage |
| `DATABASE_URL` | empty | Supabase Postgres connection (role `arya_app`) |
| `ARYA_GROUPS` | `1` | shared households (code default is off) |
| `ARYA_LIVE_PG` | `1` | live sync through Postgres NOTIFY when `DATABASE_URL` is set |
| `ARYA_PUSH_REMINDERS` | `0` | push reminders; needs `ARYA_VAPID_*` keys |
| `ARYA_SUPABASE_AUTH` | `0` | email code and Google sign-in; needs `SUPABASE_URL`, `SUPABASE_ANON_KEY` |
| `ARYA_SUPABASE_SIGNUP` | `1` | `0` limits Supabase sign-in to existing accounts |
| `ARYA_REGISTRATION_OPEN` | `0` | open password sign-up |
| `ARYA_PHOTO_READER` | `auto` | receipt photo reader: `auto` (Spark first, Wonder on a flagged read), `spark` or `wonder` |
| `ARYA_DAILY_BUDGET_USD` | `10` | app model-spend ceiling per day |
| `ARYA_USER_HEAVY_MONTHLY_USD` | `3` | one account's month of AI spend that shows a heads-up on its usage screen |
| `ARYA_USER_MONTHLY_CEILING_USD` | `5` | one account's month of AI spend that pauses its paid reads until the 1st |
| `ARYA_USER_HOURLY_FREEZE_USD` | `1` | one account's hour of AI spend that switches its paid reads off for `ARYA_USER_FREEZE_HOURS` (24); I can lift it with `POST /api/usage/unfreeze` |
| `ARYA_COST_PUBLIC_MIN` | `50` | accounts with spend before the whole-Arya totals show on the usage screen |
| `ARYA_COST_PAID_BY` | `Hackathon credits` | who the usage screen says pays for the AI reading |
| `ARYA_DEMO_DAILY_BUDGET_USD` | `100` (code default `2`) | demo household's day bucket |
| `ARYA_DEMO_SESSION_BUDGET_USD` | `2.00` (code default `0.40`) | one demo visit's share |
| `ARYA_DEMO_DOOR_RATE`, `ARYA_DEMO_AUTH_RATE` | `240`, `120` | per-IP demo door and auth rate limits |
| `ARYA_FEEDBACK_DAILY_CAP` | `5` | feedback messages per account per day |
| `ARYA_TOKEN_GUARDS` | `on` | an API token cannot call the blocked routes (bulk model spend, groups, push, feedback, account export, Kroger aisle hints); `off` lifts the list |
| `ARYA_DEMO_TOWN` | `0` | demo household shops at simulated stores in a simulated neighborhood (`DEMO-TOWN.md`) |
| `ARYA_OWNER_USERNAME` | empty | the owner account: it runs Ops and health (models, limits, accounts, freezes, Ask locks, registration). The owner can hand the role to another account in Ops; the old owner keeps owner rights for 24 hours so a wrong pick can be undone |
| `ARYA_OPS_EMAIL`, `ARYA_ALERT_EMAIL`, `RESEND_API_KEY` | empty | email through Resend: delete-account codes, waitlist and feedback notes, and one alert per Ask lock (at most 20 a day); off until the key is set |
| `GOOGLE_OAUTH_CLIENT_ID`, `GOOGLE_OAUTH_CLIENT_SECRET` | empty | Arya's own Google sign-in; without both, Google goes through Supabase |
| `ARYA_SCAN_DAILY_FRAMES` | `120` | Map my store frames per account per day |
| `ARYA_KROGER_AISLES`, `KROGER_CLIENT_ID`, `KROGER_CLIENT_SECRET`, `ARYA_KROGER_DAILY_CALLS` | `0`, empty, empty, `800` | live Kroger aisle hint (api/kroger.py); never stored |
| `MODEL_API_KEY`, `BASETEN_API_KEY`, `ARYA_OPEN_MODEL`, `GOOGLE_PLACES_API_KEY` | empty | model keys and the store picker |
### 0.12 Cost guards
Every model call writes a row to the cost ledger. The app, the demo
household and each demo visit spend from separate buckets (§10). Live
doors, batch judge runs and hunt creation pass the guard first; the
demo's live doors also have per-IP caps. Deterministic code never calls a
model. An Ask message the guard refuses costs $0 (no model call), and
`max_tokens` bounds each Ask reply (§6).
## 1. The shape: two layers
Arya has two layers. Each layer does the work it is good at.
- **The edge** is the user's own Muse. It sees the media: receipt
photos, PDFs, pasted orders. It reads them and sends
Arya structured data only; on this path no receipt image reaches the
server. A receipt photo filed in the app itself is kept per account
and is viewable in the app ("View receipt", Discussion 42). This is
an accepted trade-off, recorded in SPEC.md.
- **The core** is the Arya server. It verifies, remembers, reasons, and
answers. It spends no tokens on reading media.
The rule between the layers: **the client proposes, Arya disposes.**
Client data is untrusted. Totals must reconcile in code before a receipt
is accepted. Low-confidence lines go to the review queue.
## 2. Intake paths
All paths end in one contract: a decoded receipt with lines, subtotal,
tax, and total.
1. **Structured intake (primary).** POST /api/intake/structured. The
user's Muse posts the decoded receipt as JSON. The server recomputes
every line status. A subtotal that does not match the lines fails
reconciliation and is marked, never silently accepted.
2. **Order text.** POST /api/intake/order-text. A pasted online order
(Instacart, Amazon) is extracted server-side, then follows the same
path. Delivery fees and tips are excluded by rule.
3. **Server decode (fallback).** PDF text and photo decode endpoints
remain for clients that cannot decode at the edge. They use the
Model API under the model policy (MODEL-POLICY.md).
4. **Costco JSON export.** Bootstrap seed only. It is not a product
intake path.
## 3. Verification in code (no model, no cost)
- Arithmetic: line totals must sum to the subtotal within 5 cents.
- Line status: auto at confidence 0.8 or above, else unverified and into
the review queue.
- **Discount lines are structure, not identity.** A discount line
(Costco prints it as "/- ") attaches to its parent item
(named on the receipt, else the previous item line) and is always
auto-verified. It never enters the review queue, on any path.
- Fee lines are classified once and never enter patterns or the list.
## 4. Identity: what things are
Three mechanisms, in order of strength.
1. **The dictionary** maps each store SKU to a canonical name. A name the
user confirms teaches the dictionary. An item whose name cannot be
established is **unresolved**: off the list, out of patterns, waiting
in the review queue. Arya never guesses a name.
2. **Groups.** Products are grouped by an explicit group key when one is
set, else by their normalized name. Costco re-SKUs the same product
under one name; the group is the product everywhere it matters:
patterns, ledger, list. Merges and splits set the explicit key.
3. **The Curator** (server-side reasoning) proposes merges and splits
inside candidate clusters built in code at no cost. The model judges
a cluster under a strict doctrine: same product only; never across
organic vs conventional, form, flavor, fat content, or a size that
makes a different product; when in doubt, keep separate. **A proposal
changes nothing until the user applies it. No silent edits, ever.**
Proposals the dictionary has already overtaken (members already one
group, a duplicate, a split already in effect) retire automatically
as *superseded* and are never shown.
## 5. The Resolver: research at the edge
Arya's server models have no web access. Identification from the outside
world is therefore an edge job, run by the user's Muse:
- Trigger: an unresolved dictionary item, or an unverified receipt line
whose name is abbreviated or unreadable.
- Method: the retailer's own catalog first. A store item number matched
on the retailer's site (for Costco: costco.com and its same-day
storefront) is the strongest evidence. Then the wider web.
- Output: candidates **with source URLs**, posted to Arya
(POST /api/review/suggest) and shown in the review queue next to the
item or line. Research is evidence, not a verdict: the user settles
with one tap, and settling teaches the dictionary.
- A brand initial alone is not evidence. Items with no catalog match
stay open.
## 6. The reasoning agents in the core
Each agent has a narrow job and a named limit.
- **Staple judge.** Input: pattern facts only (buys, cadence, quantities,
all computed in code). Output: STAPLE, SEASONAL, OCCASIONAL, ONE_OFF, or
NOT_CONSUMABLE per product. The verdict gates admission to the list.
Explicit user intent always beats the judge. Unjudged and unresolved
items stay off the list.
- **Curator.** See §4. Proposes identity changes; never applies them.
- **Ask Arya.** Chat over a household snapshot. Actions (add to list,
remove from list, ran out, mark expired, never suggest, snooze) are
validated server-side. A list is found by a near name ("Costco" finds
"Costcozy"), and only a row already on a list can be removed. The
snapshot carries receipt facts (count, last, biggest, trips per
store), spend per store per month, pack dates as words, and the
cheapest store per item. Numbers come from the data, never from the
model. At most 10 actions
run in a turn. More than 3 removes in one turn wait for a yes (Yes, do these
or Not now) on every screen. On the Ask tab the other actions apply at
once; from a list or a hunt, adds apply with Undo and the riskier actions
(out, expired, never, snooze) wait for a yes (`/api/chat/apply`). Demo
sessions can chat; it is a live door under the demo's per-IP and per-visit
caps (§10).
- **Ask guard** (`api/ask_guard.py`). The person's latest message is checked
against high-confidence phrases only (an override with a target, a prompt
leak, secrets, other accounts). A hit gets a fixed refusal with no model
call, and counts a strike. Three strikes in 24 hours pause Ask: a demo visit
for 24 hours, a real account until the owner unlocks it in Ops (the message says to email support). The owner is
never locked. A reply that quotes the system prompt, holds something shaped
like a key or links off Arya is swapped for the refusal, with no strike.
Stored text (item and list names, the screen, a photo) never strikes. Each
Ask reply is capped at 1,500 tokens.
## 7. Money rules (summary; SPEC.md owns the detail)
- Returns are first-class: FIFO netting against buys, refunds reconcile
into spend, patterns count kept items only, a returned item is never
re-suggested.
- Every line carries an allocation: HOUSEHOLD (default) or FOR-OTHERS.
For-others spend is in the ledger, never in patterns, inventory, or
the list.
- The list is a derived view of the inventory model, never the source
of truth.
## 8. Accounts, tokens, and the connector
- Per-user state is a set of JSON documents, one per former file
(receipts, dictionary, flags, lists, quest...). `ARYA_STORE` picks
where they live: `json` (files under data/users//, the
default), `dual` (files are the truth, every save is mirrored to
Postgres), or `pg` (table `arya.user_docs`, see `api/pgstore.py`).
The demo account always stays on files.
- An account name is also a folder name, so it holds only letters,
digits and `. _ % + - @` (`auth.valid_name`). Passwords are PBKDF2
hashes; sign-in is throttled; sessions are cookies, and session
tokens are stored as SHA-256 hashes.
- Email code sign-in (Supabase Auth) is built behind
`ARYA_SUPABASE_AUTH=1`, off by default and on in the live site. Google
sign-in is Arya's own (`api/google_oauth.py`) when its client id and secret
are set, with the Supabase Google provider as the fallback. The provider only
proves the email; Arya keeps its own account (named by the email, no
password) and its own cookie (`api/supabase_auth.py`).
- A connector token is `arya_` plus a random secret. **Only its SHA-256
hash is stored.** It is shown once at mint. Minting a new one replaces
the old, and it can be revoked at any time. Bearer tokens are accepted
alongside session cookies, and the same token works for the Lists API
(`/api/v1`). A token cannot reach paid bulk reads, Ops, groups, push,
export or the Kroger lookup (`demo_guard._TOKEN_BLOCKED`). Demo accounts
cannot mint.
- The Muse connector (custom.arya) carries its own instruction doc:
operations, when to file, the verification rules, the Resolver
procedure, filing hygiene. **The doc is the standing prompt.** The
user writes no prompt.
- Routing is contextual. A receipt, an order, or a question about the
user's own list, spend, or inventory goes to Arya. A general question
does not. "Arya, …" always routes. Ambiguous input: ask, or state
the assumption.
## 9. Onboarding
1. Create an account. A new account opens on Today; there is no
Connect an AI agent step on first run.
2. Connecting Muse is optional, from Settings > About Arya > Use Arya in Muse:
make the code there (shown once).
3. Tell the Muse agent to connect to Arya; it asks for the code and opens a
secure credentials vault. Paste the code there, never into a chat message.
4. Talk to Muse. The screen carries these steps and a copyable fallback setup
prompt for assistant apps that do not load the connector doc automatically.
The Muse Code settings-file route is the backup (`AGENT-CONNECTOR.md`).
## 10. Cost control
- MODEL-POLICY.md is normative: deterministic work in code at $0;
Spark 1.3 Standard, low effort, cached prefixes for volume reasoning
(if the Contributor operator option is on, a client-side 140
requests/min gate applies; its limit is 150); Wonder is banned from
volume paths.
- Every model call is written to a cost ledger. The app's daily
ceiling (`ARYA_DAILY_BUDGET_USD`, $10) stops app model spend; the
demo household spends from a separate bucket
(`ARYA_DEMO_DAILY_BUDGET_USD`, $100/day in compose) and each demo visit from its
own share (`ARYA_DEMO_SESSION_BUDGET_USD`, $2 in compose), so demo use cannot
drain the app. Every live door, batch judge run and hunt creation
passes the guard first.
- A model answer that cannot be read is a 502 with a plain sentence,
never a 500 (`meta_client.parse_json`). Each account has its own limits:
a monthly heads-up, a monthly pause on paid reads, and an hourly freeze (defaults
$3, $5 and $1, see the flag table; the live site uses $3, $20 and $10, and the owner
account is exempt), on top of the daily ceiling for the whole app.
- api/meta_client.py is the only module that calls the Model API. The
personal-data rule is enforced there by construction.
## 11. Deployment
- One Oracle VM (Always Free; no new cloud resources). Docker Compose:
the API and Caddy with Let's Encrypt. No database container: data is
JSON files, mirrored to Supabase on the live site (`ARYA_STORE=dual`); after judging Postgres can become the master copy (`pg`). Public name:
arya.teamnightshift.org (Cloudflare A record, DNS only). The old
arya-pilot.duckdns.org name is a permanent Caddy redirect to it.
- Known limit: one uvicorn process on one VM. Demo sandboxes live in that
process's memory and the JSON files have one writer, so it cannot run as
several processes yet. Dual-mode saves mirror to Supabase inside the
request, so a slow Supabase can slow a save. Roughly a few dozen people at
once is the ceiling (README, Known limits). After judging: Postgres as
master copy, session state out of memory, several processes, a cloud
provider (see `docs/ROADMAP.md`; the phased design is
`docs/SCALABILITY-DESIGN.md`, a plan, not built).
- Deploy is `git pull` on the VM, then a compose rebuild. Never
rsync --delete: it once deleted the VM-only deploy environment.
- Settings come from `deploy/.env` (VM only). Compose passes only the
variables it lists, so every setting the app reads is listed in
`docker-compose.yml`; `tests/test_compose_env.py` fails if one is
missing. `.env.example` names them.
- The full test suite runs before a PR merges. A GitHub Actions workflow
(`.github/workflows/tests.yml`) runs it on demand only, not on
every PR (my choice). Nothing is committed to `main` directly;
a session log goes to `loops/` and `STATUS.md` is updated.
### 11.1 Logs and backups (VM runbook)
- **Log caps.** Compose caps each container's log at 5 files of 10 MB
(`x-logging` in `docker-compose.yml`), so logs cannot fill the disk.
The cap applies when a container is recreated: `docker compose up -d`.
- **Nightly backup.** `deploy/backup.sh` archives the data volume (and
the `arya` Postgres schema when `ARYA_BACKUP_DATABASE_URL` is set in
`deploy/.env`) into `~/arya-backups`, keeping the newest 7 of each.
Install once on the VM with `crontab -e`:
`17 3 * * * /path/to/arya/assistant/deploy/backup.sh >> /path/to/arya-backup.log 2>&1`
Check it with `ls -l ~/arya-backups` the next morning.
- **Off-VM copy.** A backup on the same disk dies with the VM. Install
rclone (`sudo apt install rclone`), run `rclone config` once to add a
remote you own (Oracle Object Storage, Google Drive, any S3), then add
`ARYA_BACKUP_REMOTE=:arya-backups` to `deploy/.env`. The script
then copies each night's files there and exits non-zero (so the log
shows it) if the copy fails.
- **Restore.** `deploy/restore.sh ` stops the api,
replaces the data volume and starts it again. The Postgres dump is
restored by hand with `psql`.
## 12. Quest storage
Per account, in `quest.json`: sessions, Shop-Offs, friends, Spotter
reports, and sightings (item, store, shelf label, time; no image).
`shelf_frames.json` holds one fresh shelf photo per store and item,
already scrubbed of people by SAM (`shelfscan.scrub_people`; no scrub,
no frame). `GET /api/quest/plan` counts purchases per store from the
ledger for Plan a hunt ($0). The demo sandbox overlays all of it in
memory. The store directory (`stores.json`) is platform-level.
## 13. Not built / held
- A fixed per-account spend allowance that resets by itself. Not built; per-account
limits do exist (§10: defaults $3 heads-up, $5 pause, $1 hourly freeze; live $3, $20, $10).
- Public registration is closed by default (`ARYA_REGISTRATION_OPEN`; the Ops flag overrides the env and is closed on the live site). The
owner can open it in Settings. Every door runs on Meta models: Muse in
chat, Spark on the Meta API in the app.
- Forgot-password: not built. Once switched on, email code sign-in
(§8) lets a person in without a password.
---
# Scalability design
Status: a plan, not shipped. Nothing on this page is built unless it says
"today". Written 2026-11-02 against `main` after #489. It extends the
one-line plan in [`ROADMAP.md`](/docs/roadmap) ("Scale out as the community
grows") and the Known limits entry in the [README](https://github.com/vibhortayal/arya/blob/main/README.md#what-are-the-known-limits).
Where this page says more, it does not contradict them. No provider, dates
or costs are named. Each phase is chosen when the number of shoppers asks
for it.
One idea shapes every choice here: Arya is a community app that grows
with its users. The design keeps one codebase and one set of product rules
(`assistant/docs/SPEC.md`). The operator turns each phase on with
settings, one at a time, with a way back.
## 1. What runs today
One container, one process, one VM, as recorded in
[`assistant/docs/ARCHITECTURE.md`](/docs/architecture/overview) sections
0.5, 0.10 and 11.
```
phone (PWA) --https--> CDN / DNS proxy ---> Caddy (TLS) ---> uvicorn, 1 process
| FastAPI routes
| engines ($0 code)
| model client
v
data volume (JSON per account, photos, caches)
Postgres on Supabase (mirror, ARYA_STORE=dual)
Meta Model API, Baseten (model calls)
```
Where state lives today, and why each one matters for scale:
| State | Where it lives today | Shared across processes? |
|---|---|---|
| Account documents (receipts, lists, flags, quest, ...) | JSON files under `data/users//`, mirrored to `arya.user_docs` in Postgres (`dual`) | No: file locks are per process, mirror is write-only |
| Group documents | `data/groups//`, same mirror | No |
| Sign-in sessions | `sessions.json` plus a parsed copy in process memory | Through the file only |
| Demo sessions and sandboxes | Process memory (`auth._demo_sessions`, `demo_guard._demo_sandboxes`) | No |
| Rate-limit and door buckets | Process memory (`api/deps.py`, `api/demo_guard.py`) | No |
| Cost ledger | JSON lines file, scanned whole for "spent today" | Through the file only |
| Receipt originals, shelf frames, map images | Files on the data volume (`storemap_img.json` is rewritten per photo) | No |
| Receipt and scan jobs | One worker thread per process, pending list in a file | No |
| Group live sync | Postgres `NOTIFY arya_group` (one listener thread per process) | Yes, when `DATABASE_URL` is set |
| Push reminders | A daily cron line on the VM | n/a |
### 1.1 Bottlenecks, in the order they will bite
Measured numbers are from the read-only performance pass of 2026-11-02 on a
development machine with a 117-receipt account, not the live server (the
README says the same). The ordering below is a judgment, not a measurement.
1. **One process, one GIL.** The Today payload is computed in Python on each
read (the derived list, ledger and patterns are rebuilt from receipts, with
a per-request memo only). Alone it takes about 65 ms. With 10 people at
once the median is about 1.1 s, and with 30 about 4 s. Threads share the
interpreter, so more threads do not help. The fix is more processes,
which the next three items block.
2. **Process memory holds visitor state.** Demo sessions, demo sandboxes
and rate-limit buckets live in the process. A second process would not
see a demo visitor's session. A restart drops every demo visit.
3. **Files have one writer.** `_file_lock` is a process-local lock. Two
processes writing the same account's `lists.json` would race. Dual mode
mirrors the whole document, so the mirror carries no version and
cannot detect a lost update. `pg` mode (not live) is a whole-document
read-modify-write with the same process-local locks.
4. **Saves wait for the mirror.** In `dual` mode each save upserts the whole
document to Postgres inside the request thread (connect timeout 5 s, one
retry). A slow database slows every tick on a list.
5. **Model calls share the request thread pool.** Reads of receipts, Ask and
shelf scans run in the default thread pool beside sync routes. A burst of
receipt reads can starve plain page loads, and provider rate limits are
handled per call, not per app.
6. **Big blobs rewrite on every change.** `storemap_img.json` is rewritten per
mapping photo; receipt originals live on one disk; the nightly backup
is on the same disk unless `ARYA_BACKUP_REMOTE` is set.
7. **No shared cache.** The Today payload is recomputed per request. The
per-request memo saves repeat work inside one request only.
8. **Observability is logs and `/health`.** There is a cost ledger and an ops
digest, but no request timing, no queue depth and no alert.
## 2. Target architecture
The target is plain on purpose: stateless app processes behind a load
balancer, one database as the master copy, one shared store for short-lived
state, one queue for background work, and one object store for files. Every box
below is a role, not a product.
```
phone (PWA) --> CDN / edge cache (static shell, images)
|
v
load balancer (TLS, health checks, sticky nothing)
|
+------------+------------+------------+
| | | |
app proc app proc app proc app proc (N identical, stateless)
| | | |
+-----+------+-----+------+-----+------+
| | |
v v v
Postgres shared queue object storage
(master copy, key-value (jobs, (receipt originals,
per-doc store reminders, shelf frames, map
versions, (sessions, model calls) images; private
LISTEN/NOTIFY) demo boxes, buckets, signed URLs)
rate buckets,
hot cache)
|
v
worker procs (same image, no HTTP)
|
v
model gateway (one place for
concurrency, retries, provider
rate limits, cost ledger rows)
|
v
Meta Model API, Baseten (unchanged)
metrics + logs + traces from every box --> one dashboard, a few alerts
```
### 2.1 Stateless app processes
- Same container image as today, started with N processes behind the load
balancer. A process keeps nothing between requests except read caches it
can lose. Any process can serve any request, so a crash or a deploy takes
one process out, not the site.
- Health check stays `/health`; the load balancer stops sending to a process
that fails it. Rolling deploys replace processes one at a time, and the
in-memory demo state no longer matters because it has moved out (2.3).
- Code change: none to routes. The seams already exist: `state._load/_save`,
`auth.new_session/session_info`, `demo_guard._demo_sandbox`, `deps` rate
buckets, `receiptjobs`/`scanjobs` queues, `live.changed`. Each gets a
shared backend behind the same function names.
### 2.2 Postgres as the master copy
Today Postgres mirrors files (`dual`). The target makes it the only writer
path (`pg`), with these additions that `pgstore.py` does not have yet:
- **A version per document.** `user_docs` gains a `version` integer. A save
sends the version it read, and the update is `where version = $read`. Zero rows
means someone else saved first. The request then re-reads, re-applies and
retries a few times, then fails plainly. This replaces the process-local
file lock and is the one change that makes several processes safe.
- **Connection pool.** One pool per process with a small cap, instead of one
connection per thread. Timeouts short, so a slow database degrades a
request, never hangs a worker.
- **Rows for the things that are not documents.** Sessions, the cost ledger,
job state and the pending lists become ordinary tables, so "spent today" is
one indexed `sum`, not a whole-file scan. Stockout reports, friends and
store maps can follow later (stage 2 in
`assistant/loops/2026-10-28-supabase-seam.md`); they
can stay as documents for a long time.
- **Group live sync** keeps `LISTEN/NOTIFY`; with N processes each holds one
listener, which is already how `live.py` is written.
Migration from JSON, in order, each step reversible:
1. Keep `dual`. Add `version` with a default of 0 and the versioned update
path behind a flag; files remain the truth. Nothing observable changes.
2. Re-run `assistant/tools/migrate_json_to_pg.py`, compare every document (it reports
mismatches), and fix any drift. Back up both sides (`backup.sh` already
dumps the schema when `ARYA_BACKUP_DATABASE_URL` is set).
3. Flip `ARYA_STORE=pg` on one process while the others stay on `dual`
reads only if the versioned path is on. Otherwise flip the whole site in a
quiet hour. Keep writing files for a while as a shadow copy (a new
`pg+shadow` mode, or the same `dual` logic with Postgres first and the file
second) so rollback is a flag flip, not a restore.
4. Stop the shadow file writes once a full backup cycle has passed with no
mismatch. From here the files are a backup format, not a store.
Rollback at any step: set `ARYA_STORE` back one mode and redeploy. Before
step 4 the files are current; after step 4 rollback means restore the
latest file export from the database (a small export tool is part of the
phase, see 4).
### 2.3 Session and demo state in a shared store
- **Sign-in sessions** move to a table or the key-value store, keyed by the
token hash, with the expiry as a TTL. `session_info` reads one key. The
parsed-file cache (`_SESSIONS_SEEN`) goes away; a short per-process cache
with a TTL of seconds is fine.
- **Demo sessions and sandboxes** move to the key-value store with a TTL of
one visit (hours), keyed by the demo cookie. The sandbox overlay is already
a dict keyed by file path with a 2 MB cap, so it serialises as one value
per visit. Reads load the overlay once per request; writes save it back at
the end of the request. This is the change that lets a demo visit survive a
deploy and lets two processes serve one visitor.
- **Rate-limit and door buckets** move to the same store with atomic
increments and TTLs, so a cap of "240 door hits per IP per hour" holds
across processes.
- Keep the in-process path as the fallback when no shared store is set, so
a laptop run (`compose.yaml`) and the tests stay as they are.
### 2.4 Background jobs and reminders
- Today: one worker thread per process for receipt batches and store scans,
with a pending list in a file, and a cron line for reminders.
- Target: a queue with named job types (`receipt.read`, `scan.read`,
`reminders.daily`, `backup.nightly`, `export.account`) and worker processes
that run the same code as today's thread loops. A job carries the account,
a lease and a retry count. A crashed worker's lease expires and another
worker picks the job up. Idempotency keys (the job id plus the frame
number, which `receiptjobs` already uses) make a retry safe.
- Reminders become a scheduled job that enqueues one `reminders.account` job
per opted-in account. A slow push to one phone then does not delay the rest,
and the cron line on the VM goes away.
- The "Your receipts are ready" and "Your map is ready" cards keep polling
the job list exactly as now; only the producer of that list changes.
### 2.5 Model calls: one gateway, queued, rate-limited
- Every model call already goes through `api/meta_client.py` (and
`open_client.py` for the open model). The target puts a small gateway
behind those two modules: a per-model concurrency cap, a per-provider
token-bucket for the published request limits, retries with backoff on
429 and 5xx, and a timeout per job kind (Wonder re-reads are long, Ask is
short).
- Interactive calls (Ask, a single receipt from the camera) keep a reserved
share of the concurrency so a batch upload never blocks a question.
- Batch calls (bulk receipt reads, scans, judge runs, dictionary enrichment)
go through the queue (2.4) and the gateway's batch share.
- Budgets stay as they are in policy (`ARYA_DAILY_BUDGET_USD`, demo buckets,
per-account limits); their counters move to the database (2.2) so they hold
across processes. A **spend reservation** is taken before the call and
settled after, so two processes cannot both spend the last dollar.
- The ledger row per call stays the audit record; the gateway writes it.
### 2.6 Caching
Three layers, each safe to lose:
1. **Edge.** The built shell and static images already carry immutable
cache headers; the CDN serves them. Nothing under `/api/` is cached at
the edge.
2. **Shared hot cache.** The Today payload per account (and per group plane)
is cached in the key-value store for a few seconds and dropped on any
save to that account or group (`save_at` is the one choke point, so the
invalidation is one line). The same for the price facts and the derived
list. This is the cheapest win on bottleneck 1 and needs no process
change; the per-request memo stays as the last layer.
3. **Per-process.** Short TTL caches for sessions and settings.
Rule: cache derived data only. Receipts, lists and flags are read from the
master copy, never from a cache, on a write path.
### 2.7 Object storage for photos
- Receipt originals, shelf frames and map images move to a private bucket,
one key per account and file (`users//receipts/.`),
with signed, short-lived URLs for "View receipt". The database row keeps
the key, the size and a content hash.
- `storemap_img.json` stops being one rewritten blob: each image is its own
object and the map document keeps keys only.
- Uploads keep going through the API (the people-scrubbing step must run
before anything is stored, as today); the API streams the result to the
bucket. Deletion of an account deletes its prefix.
- Backups: the bucket has versioning or a nightly copy to a second bucket;
the database dump is the other half. `restore.sh` restores the volume
today and would restore the dump plus point at the bucket.
### 2.8 Observability
- **Metrics per process:** request count and latency by route (p50, p95),
in-flight requests, thread pool queue length, database pool wait, cache
hit rate, queue depth and job age by type, model calls by model with
latency, error and 429 counts, spend per bucket. Exposed on a
`/metrics` endpoint the load balancer does not route to the public.
- **Logs:** one JSON line per request with a request id, account hash (not
the name), route, status, duration and the job id when one applies. The
same id goes into model calls and ledger rows.
- **Traces:** optional; a request id in logs covers most of what a trace
would, until there are several services.
- **Alerts, few and plain:** `/health` red for a few minutes; p95 of
`/api/today` above a line; queue age above a line; model error rate above
a line; daily spend near the ceiling. The existing ops digest
(`harness/ops.py`) becomes a reader of the same metrics.
- **Privacy:** no receipt content, no names and no photos in logs or
metrics; the people-scrubbing and personal-data rules in `meta_client`
stay the boundary.
## 3. Phased migration plan
Each phase is turned on by settings on the current image, has a
measurement that says whether it worked, and has a rollback that is a flag
flip or a redeploy of the previous image. Phases go in this order because
each one removes the blocker for the next. None has a date.
| Phase | What changes | Blocker it removes | Rollback | What to measure |
|---|---|---|---|---|
| 0. Baseline | Add request timing and a `/metrics` endpoint; run the Today load script against the live site in a quiet hour; set `ARYA_BACKUP_REMOTE` and prove one restore off the VM | Nothing is known about the live ceiling | Remove the endpoint | p50 and p95 of `/api/today` at 1, 10, 30 concurrent on the live server; restore time |
| 1. Hot cache | Today, price facts and derived list cached per plane for a few seconds, invalidated on save; mirror writes moved off the request path (a background queue in-process, with a circuit breaker on failure) | Bottlenecks 1 (partly), 4 and 7 | Flag off the cache; flag off the async mirror | Same load test: median at 10 and 30 concurrent; cache hit rate; save latency with the mirror slow or down |
| 2. Postgres master | `version` column and versioned updates; connection pool; sessions and ledger as tables; `pg` with shadow file writes; then shadow off | Bottleneck 3 and the one-writer rule | `ARYA_STORE` back one mode; restore from files until shadow is off | Mismatch count from the migration check (must be 0); lost-update retries per hour; database pool wait |
| 3. Shared short-lived state | Demo sessions, sandboxes and rate buckets in a shared key-value store, with the in-process path as fallback | Bottleneck 2 | Unset the store address; the process path returns | A demo visit survives a deploy; door caps hold across two processes |
| 4. Several processes | Start N processes behind the load balancer on the same VM first, then on more than one machine; rolling deploys | Bottleneck 1 | N back to 1 | p50 at 30 concurrent falls in step with N; zero demo visits dropped during a deploy |
| 5. Queue and workers | Receipt, scan and reminder jobs on the queue; worker processes; cron line retired | Bottleneck 5 (partly) | `ARYA_*_JOBS_INLINE` style flags back to the thread loop | Job age p95; retries; jobs lost on a worker restart (must be 0) |
| 6. Model gateway | Concurrency caps, provider rate buckets, spend reservation, interactive reserve | Bottleneck 5 | Caps to unlimited; reservation off | 429s per hour; Ask latency while a batch runs; spend over the ceiling (must be 0) |
| 7. Object storage | Originals, frames and map images in a bucket; `storemap_img.json` split | Bottleneck 6 | Keep files on the volume until the copy is verified; read from the volume on a bucket miss | Bytes on the volume; map save latency; signed-URL errors |
| 8. Cloud provider | Move the whole set (processes, database, store, queue, bucket) to a provider with managed versions of each; the VM becomes a fallback until a full week passes | Hardware ceiling | DNS back to the VM | Same dashboard, same alerts, same load test |
Phases 0 and 1 can happen on the current VM with no new service.
Phase 2 uses the database that already exists. Phase 3 is the first new
service. From phase 4 on, the choice of provider matters, and the design
above lets any of them fit.
## 4. Risks and how the plan answers them
| Risk | Answer |
|---|---|
| Lost updates when two processes save one document | Versioned updates (2.2) before any second process (phase 2 before 4). Never the other order. |
| A migration that silently drops a document | The migration tool already counts mismatches; phase 2 requires zero before `pg`, and shadow file writes keep a second copy. |
| Demo visits dropped during judging | Not a risk here: no phase runs before judging ends. Later, phase 3 removes it. |
| Cost spike from more processes | The budgets are per app, not per process, once counters live in the database (phase 2) and a reservation exists (phase 6). Until then, run one process. |
| Privacy rules weaken as state moves | Each move keeps the same boundary: scrub before store, hashes not names in sessions and logs, personal data only to Meta Standard mode as recorded in `AGENTS.md`. A phase that touches photos (7) needs a privacy review first. |
| Vendor lock-in | Every box is a role with a plain interface (SQL, key-value with TTL, a queue with leases, object keys with signed URLs). Nothing uses a provider-only feature. |
| More moving parts than one person can run | Each phase stays optional and off by default; a laptop run and the tests keep the in-process path. The dashboard and alerts (2.8) come in phase 0, before anything else. |
| Reads stay slow because the engines are Python | The hot cache (phase 1) and N processes (phase 4) are the first answers. Precomputing the derived state on save instead of on read is a later option the memo design already allows. |
## 5. What stays the same
The product rules, the engines in code at $0, the one model client, the
cost policy, the privacy boundary, the PWA, the connector and the API. A
shopper would not see a phase happen, except that Today loads as fast with
hundreds of neighbours as it does alone today.
## 6. Related pages
- [`ROADMAP.md`](/docs/roadmap): the one-line plan this page expands.
- [`../README.md#what-are-the-known-limits`](https://github.com/vibhortayal/arya/blob/main/README.md#what-are-the-known-limits): the limit as
judges read it.
- [`../assistant/docs/ARCHITECTURE.md`](/docs/architecture/overview):
what runs today, section by section.
- [`../assistant/docs/SCALE-SIM.md`](https://github.com/vibhortayal/arya/blob/main/assistant/docs/SCALE-SIM.md): the
simulator and its earlier findings.
- `assistant/docs/OPS.md`: the runbook that each phase updates.
---
# Model policy: cost efficiency (my directive, 2026-10-27)
Rule: every job runs on the cheapest model that holds its golden-set score.
Escalation is by evidence (a scored eval), never by default.
## Prices (per 1M tokens, Meta Model API, verified 2026-10-27)
- muse-spark-1.3 Standard: $1.25 in / $0.15 cached / $4.25 out
- muse-spark-1.3 Contributor: $0.10 in / $0.002 cached / $0.20 out (different data
terms; not used). Limit 150 requests/min. Not the app's tier; see "Spark tier" below.
- muse-wonder-1.0: $4.00 in / $0.20 cached / $20.00 out. This is the expensive
frontier model, not a budget tier. It is banned from volume paths.
- muse-image-1.0: $0.01/image. muse-voice-transcribe-1.0: $0.18/audio-hour.
- sam-3.1: $2.50 per 1,000 images; $0.20 per 1,000 video frames.
## Assignments
1. The cheapest model is no model. Patterns, cadence, run-out dates, ledger
math, reconciliation, and list generation are deterministic code: $0.
2. Volume intelligence (receipt decode fallback, list and date photos,
command parsing, Ask Arya, judge): Spark, reasoning_effort=low, stable
prefixes cached. Spark runs as muse-spark-1.3 Standard (see "Spark tier"
below).
Exception (owner, 2026-11-04): the list photo read sends
reasoning_effort=minimal for speed, and repeats once at low if the
API refuses the value.
3. Eval loops (golden sets, prompt variants): Contributor mode. Gate
scores still count on Standard. (Superseded for app traffic by the
2026-11-01 amendment.)
4. Wonder: zero jobs at baseline (history; the live reader is `auto`, see below). A job earns Wonder only by failing its
golden set on Spark at two effort levels, with the eval logged.
5. Voice: transcribe on demand only; Ask has no voice out (replies are text).
6. Muse Image: painted the 23 characters once and paints share looks (live); a dinner render is planned, not built. muse-image-1.0 costs $0.01 per image. It is the one place a flat
per-unit price beats token metering.
## Measured anchors (default reasoning effort, spike runs)
- PDF receipt decode: ~2.9 cents/receipt. Photo: ~1.5 cents short, ~3.1 long.
- JSON import: $0. Patterns on imported history: $0.
- Policy targets: a full trip decode at or under 1 to 2 cents; total hackathon
API spend well under the $150 credit (plan §8 estimated $57 to $61 at defaults).
## Credits (me, 2026-10-29)
There is no $0 rule. Hackathon credits (Meta Model API, plus Baseten and Modal)
are available, so spending on Meta models (Muse Image, Spark, Wonder, SAM) is
allowed when it improves the product or demo. Stay budget-aware: the cheapest
model that holds its score is still the default, spend goes through the ledger,
and large batches get a capped trial first. The in-code caps (daily and demo
budgets) are unchanged.
## Hackathon amendment (2026-10-27, per WONDER-EVAL.md; changed 2026-11-01)
**My decision 2026-11-01:** receipt photos are read by the **auto
reader** (Ops "Photo reader" = "Spark first, Wonder as backup"): Spark
reads first. Wonder re-reads only when Spark fails or the checks
below flag the read (arithmetic, low confidence on too many lines). Wonder
stays allowed as that second reader through Nov 9. That is meant to cover
judging after submission. The original 2026-10-27 amendment (Wonder first, Spark
fallback) is summarized below.
Original (2026-10-27): through Nov 4, judge-facing photo decode ran Wonder first with a Spark fallback. Wonder scored 97 of 98 lines after hand adjudication (the automatic scorer gives Wonder 86 of 98) against Spark's 92 of 98 on the automatic scorer. The numbers, latency, cost and the staple-judge check are in [WONDER-EVAL.md](/docs/evals/wonder-eval).
**Auto reader (built 2026-11-01; my pick for judging on
2026-11-01; NOT yet measured on photos).** `ARYA_PHOTO_READER=auto` (or the Ops "Photo reader" switch)
reads with Spark first and runs the checks in `api/decode/checks.py`
(lines add up to the subtotal, subtotal plus tax equals the total, the
footer item count, impossible prices, too many unsure lines, a price far
from the same item number's earlier price on the account). Only a
receipt that fails them is re-read by Wonder. A Spark failure goes
straight to Wonder. If Wonder is down, the Spark read is kept and every
line is sent to review. The receipt records `decoded_by` and `second_read`
(the reasons), so it is never silent. Why it exists: Wonder won the eval
on accuracy. Spark is about nine times cheaper per receipt and about
four times faster. Cheap arithmetic can catch most of Spark's mistakes.
Known gap, from the Gate 0 notes: a price rotation (two prices swapped
between lines) keeps the sum, so arithmetic alone misses it. The price
history check covers it only for item numbers seen before. Before the
default changes, run `python3 -m harness.photo_reader_eval --photos DIR`
on the Gate 0 photos and require that every price where Spark and Wonder
differ is on a flagged receipt. After Nov 9 the amendment above ends.
If I extend it, auto keeps Wonder's accuracy only where the checks say it
is needed. Otherwise photo decode is Spark only.
**Demo budget.** Each demo visit may spend `ARYA_DEMO_SESSION_BUDGET_USD`
(code default $0.40; $2 on the live site, owner 2026-11-04) on live
reads, held in that visitor's in-memory sandbox; all demo visits together
stay under `ARYA_DEMO_DAILY_BUDGET_USD` (code default $2; $100 a day on
the live site for the judging window, about 50 full visits a day, and the
top of the Ops range). Ops > Settings > Dollar limits sets both without a
deploy and beats the env file. To go back through the env file, set
`ARYA_DEMO_DAILY_BUDGET_USD=2` and `ARYA_DEMO_SESSION_BUDGET_USD=0.40` in
the VM env file, then run `docker compose up -d` in `assistant/deploy` (a
plain `restart` does not re-read the env file).
**Switch after Nov 9, not before.** The photo reader is an env setting:
`ARYA_PHOTO_READER=spark` in the VM env file, then `docker compose up -d`
in `assistant/deploy` (not `restart`, which keeps the old environment),
puts photo decode on Spark Standard with no deploy. Unset, or `wonder`,
is the amendment path. Receipts record `decoded_by`, so the switch is
visible in the data. Spark-only photo decode scored 93.88% in the Gate 0
eval, with the review queue as the mitigation.
## Spark tier (my decision, 2026-11-01)
Spark and Wonder run in Standard mode. Meta's models page (dev.meta.ai/docs/models):
"In Standard mode, your content is not used to train Meta models." Earlier on
2026-11-01 I had moved the app's Spark calls to
`muse-spark-1.3-contributor` for the lower price; the same day I moved
them back to Standard, so nothing the app sends is used for training.
- Code: `meta_client.resolve_mode()` maps the app's "standard" (Spark) mode to
the tier set on the Ops screen or in `ARYA_SPARK_TIER`; the Ops screen
setting wins. Wonder is always Standard. An explicit `mode="contributor"` with
`personal=True` is still refused.
- Option, not built: a community (Contributor) tier with user opt-in could
support more users at lower cost. It needs an opt-in screen, a per-user
routing rule and updated privacy text first, and my word.
- Rate limits: Standard allows 3,000 requests/min per team. The Contributor tier
allows 150/min, so `meta_client` keeps a sliding-window gate at 140/min for that
case. A call that would wait over 20 s fails with "Arya's AI is busy right now"
(HTTP 502) instead of hanging.
- Receipt photos follow the auto reader above. SAM, Muse Voice and the
open-weights client are unchanged.
---
# Meta products used, and how
Every intelligent step runs on a Meta model. Other tooling (Three.js,
ffmpeg, Postgres) does plumbing only.
| Meta product | Job | Where |
| --- | --- | --- |
| **SAM 3.1** | Shelf segmentation for the store map; Shelf Scan and Find it for me boxes; finding people so they are pixelated before any frame is kept | Mapping pipeline; server |
| **Muse Spark 1.3 Standard** | Staple judge, Curator, Ask Arya, list photos, expiry-date photos, grocery check for unknown names, order-text fallback, shelf labels, Shelf Scan crop verification, receipt fallback | Server (Meta Model API) |
| **Muse Spark Open** (open weights, hosted on Baseten) | Second seat of the staple judge (names only, advisory) and dictionary category labels. It sees item names only (no prices, dates, stores or account details). Numbers: [JUDGES.md](/docs/judges#which-model-does-which-job) | Batch jobs (`api/open_client.py`) |
| **Muse, the user's own assistant (over MCP)** | Connects to Arya as an agent: reads the list, adds and checks off items, answers "what am I running low on?", and files a receipt it read in chat (Arya checks the arithmetic) | The user's Muse session, to Arya's MCP address (live) |
| **Muse Wonder 1.0** | (1) Second reader for receipt photos (rule: see the model map). Numbers: [JUDGES.md](/docs/judges#which-model-does-which-job). | Server |
| **Muse Image 1.0** | The 23 grab-celebration characters; share looks | Characters generated once and shipped as assets; share looks are generated live per tap |
| **Muse Voice Transcribe** | The list mic and Ask Arya's mic | Server |
| **Meta Model API** | The one surface for server calls, through one client module with a daily spend ceiling | Server |
## Muse connects to Arya (live)
Arya has a standard MCP connector, and Muse is the reference client. In Muse, ask the agent to connect to Arya's MCP address. When it asks for your code, paste it into the secure credentials vault Muse opens, never into the chat. I tested exactly that on my own Muse on 2026-11-02, with no terminal and no settings file. Muse then has six tools: `get_snapshot`, `get_list`, `add_items`, `check_item`, `file_receipt` and `get_receipts`. A receipt Muse reads in chat is filed through `file_receipt`. Arya checks the arithmetic itself and spends no photo-reading credits on it. The code acts as your account, but it cannot reach routes that spend model credits in bulk, change settings, or touch groups and friends. It is capped at 600 calls an hour, and you can turn it off in Arya at any time. Setup and limits: [the agent connector](/docs/guide/agent-connector).
## Rules
- Spark and Wonder run in Standard mode (checked 2026-11-01): content is not used to train Meta models. Contributor
is used for eval and synthetic data only; a community Contributor tier with user opt-in is a possible later option, not built.
- Model output is a proposal. Receipts must reconcile, low-confidence
lines queue for review, and identity changes need the user's approval.
- Cheapest model that holds the score. Wonder as the backup receipt reader is
the one exception, time-boxed to the hackathon amendment (expires Nov 9).
- No pictures of people are kept. SAM pixelates the people it detects first, or the frame is
dropped.
**Used to build, not shipped.** Muse Code (Meta's coding agent), running Muse Wonder, wrote the first version of the engine (my account). It is not part of the shipped app; see the README, How this was built.
## Considered, not used
- **Glimmer** (open weights): not run.
- **Gaussian splatting and VGGT 3D models:** tried on a Trader Joe's
walkthrough during the hackathon, then removed (2026-10-29). The
store the video came from was never recorded, so it could not be
any real store's map.
---
# Meta models: available, used, and future ideas
A one-time look (2026-11-02) at every model Meta offered me, what Arya uses
today, and what it could use later. The models in use are described in
[`assistant/docs/MODEL-POLICY.md`](/docs/architecture/model-policy) and
[`docs/quest/META-PRODUCTS.md`](/docs/quest/meta-products). This page adds the
models I did **not** use. The model list comes from Meta's models page
(dev.meta.ai/docs/models) as read on 2026-11-02, and the code from
`assistant/api/meta_client.py`. Where I could not confirm something, it says
"unconfirmed". Everything under "Future" is an idea, not built and not promised.
## Which Meta models does Arya use today?
The models in use, with ids and jobs: [For judges, Meta model map](/docs/judges#which-model-does-which-job).
## Which Meta models are available but not used?
| Model | What it is | Fit for Arya | Cost and effort |
|---|---|---|---|
| **Spark 1.3 Contributor** (`muse-spark-1.3-contributor`) | Same model on a lower-price tier with different data terms (see Meta's models page) | Cuts AI cost by roughly 10x on input and 20x on output (list prices: $0.10 / $0.20 vs $1.25 / $4.25 per 1M tokens). The code supports it (`ARYA_SPARK_TIER`), but Arya runs Standard on purpose. **Future idea:** a community tier where a user opts in, for non-personal jobs only | Needs an opt-in screen, routing rule and privacy text first. Rate limit 150 requests/min vs 3,000 on Standard, which also caps scale |
| **Muse Glimmer** (open weights) | 30B dense, text and image in, text out, Apache 2.0, distilled from Spark; runs on your own hardware via vLLM, SGLang, llama.cpp, Ollama | **Future idea:** a self-hosted "household stays at home" reader, or a cheap pre-filter before Spark for name cleanup and labels. Text and image only, so no audio or PDF. Accuracy on receipts is **not measured** by me | No per-token price from Meta; I would pay for a GPU or a host. Whether Baseten serves it is unconfirmed. Needs its own eval against the 98-line receipt set before any job moves |
| **Muse Spark 1.2 / 1.1** | Older Spark versions. Meta recommends 1.2 for audio input | Not needed: I send audio to Voice Transcribe | None |
| **Muse Wonder Full** | Higher Wonder variant | Muse Code plans only, not on the Model API, so unavailable to the app | n/a |
Smaller API features I have not used: web-search grounding ($2.50 per 1,000
searches; it could ground "is this a good price", but it would send queries
out, which conflicts with the personal-data rule), computer use (no fit), and
the Files API with background responses (a possible fit for very long scan
videos). Image generation is only used offline for the characters. A live
"dinner render" idea sits in MODEL-POLICY and is not built.
## Why can I not use store video and photos?
The best raw material for better store maps and shelf help already exists
online: walkthrough videos and in-store photos on YouTube, TikTok and similar
sites. Arya cannot use them. They belong to their creators and the platforms,
and the platform terms forbid bulk collection. Arya's own rule is that no
store map is built from footage I do not have the right to use. The
Trader Joe's walkthrough experiment was removed for exactly this reason (see
`docs/quest/META-PRODUCTS.md`). Today every store map comes from a shopper
walking their own store with the guided scan.
**Future idea: a data partnership.** A store chain, a licensed-content
provider or a platform that can legally supply in-store video and photos
(aisle walkthroughs, shelf signs, layout) would let Arya:
- start each store with a first map instead of an empty one;
- test and tune the shelf-reading step (SAM plus Spark) on many layouts, not
one household's;
- give community shoppers better "where is it" help from day one.
Any partnership would need written permission, store consent, the existing
people-pixelating rule, and a takedown route. See
[`docs/ROADMAP.md`](/docs/roadmap) ("Store data and community").
---
# Engineering process: how we built Arya
Arya was built in nine days (Oct 27 to Nov 4, 2026) by one person working with a team of AI agents. Each agent has one job. This section shows how that worked and what it left behind. The pages here are the working documents themselves, lightly cleaned for the public. Operator-only steps, server paths and key names are removed.
## How the work moved
I talk to one coordinator. The coordinator hands each piece of work to a role. Each role has its own place in the process:
- **Builders** open the pull requests. UI work starts with a phone-size mockup I approve first.
- **The Tester** runs each change at phone sizes and reports PASS or what to fix.
- **Ops** merges and deploys, checks the app's health afterwards, and rolls back if it fails.
- **The Doc maintainer** keeps the Help tab, guide, Ask knowledge, status and decisions in step with what shipped.
- **The Editor** reads reader-facing docs cold, as a judge would, before they merge.
The order for a change is: idea, name the role, approval, mockup (UI only), pull request, Tester, merge, doc check, smoke test on main, deploy, live checks.
## Where to read more
- [Agent rules](https://github.com/vibhortayal/arya/blob/main/AGENTS.md): the contract every agent follows (hard rules, roles, workflow).
- [Decisions](https://github.com/vibhortayal/arya/blob/main/docs/DECISIONS.md): every choice I made, with the date.
- [Status](https://github.com/vibhortayal/arya/blob/main/STATUS.md): what was live and what was not, batch by batch.
- [Operations](https://github.com/vibhortayal/arya/blob/main/assistant/docs/OPS.md): how the app is watched.
- [Scale simulation](https://github.com/vibhortayal/arya/blob/main/assistant/docs/SCALE-SIM.md): the load test.
- [Submission draft](https://github.com/vibhortayal/arya/blob/main/assistant/docs/SUBMISSION-DRAFT.md): the portal write-up.
- [Quest build plan](https://github.com/vibhortayal/arya/blob/main/docs/archive/quest/BUILD-PLAN.md) (archived on 2026-11-04) and [design discussion](https://github.com/vibhortayal/arya/blob/main/docs/quest/DISCUSSION.md): how the game was planned and argued out.
---
# Evidence behind the measured results
Every number in the repo README's "Measured results" has a file here.
| Claim | File |
|---|---|
| Gate 0: 90 of 98 lines on 9 public receipt photos (Spark), arithmetic 7 of 7 | [GATE-0-REPORT.md](/docs/evals/gate-0) |
| Wonder 97 of 98 vs Spark 92 of 98 on the same set (Wonder after hand adjudication; Spark on the automatic scorer, which gives Wonder 86 of 98), latency and cost | [WONDER-EVAL.md](/docs/evals/wonder-eval) |
| Staple judge precision 0.79 (41 of 52) against a cadence proxy | [JUDGE-EVAL.md](/docs/evals/judge-eval) |
Read the numbers as a gap, not a score. Spark scored 90 of 98 in the Gate 0 run and 92 of 98 in the Wonder comparison on the same set. That difference is run-to-run variance on one small set. So quote the gap between the models, not either number alone. The live reader (`auto`: Spark first, Wonder re-reads a flagged receipt) is not yet measured on photos. A swapped pair of prices keeps the sum, so the review queue stays. On the judge playbook's sample receipt of the time (India Bazaar, Nov 1, 2026), three live runs on 2026-11-02 matched every printed price. One of them was re-read by Muse Wonder.
`gate0-public-set/` holds the hand labels (written before any decode), the raw
model results, the scores, the two scripts (`decode_public.py`, `score_public.py`)
and `manifest.json` with each photo's public source. The photos themselves are
not copied here, because they belong to their publishers. `manifest.json` links
each one.
The scripts and reports were written in a separate workspace, so paths such as
`~/workspace/meta-hackathon-2026/` in them point there, not into this repo.
The reports are dated records; the app has changed since.
---
# Gate 0: public photo-receipt fixture set, report
Date: 2026-10-27. Model: muse-spark-1.3 (Standard), temperature 0.1, SYSTEM prompt taken verbatim from `spike2/decode_photo.py`. Labels were written by hand from the photos BEFORE any decode ran. Each decode call was stateless.
## Verdict
| Set | Lines correct / labeled | Accuracy | vs 90% gate |
|---|---|---|---|
| Public set (identity metric) | 90 / 98 | 91.84% | PASS |
| Public set (strict-string metric) | 80 / 98 | 81.63% | FAIL |
| Combined: my subset (32/32) + public (identity) | 122 / 130 | 93.85% | PASS |
| Combined (strict-string) | 112 / 130 | 86.15% | FAIL |
Two metrics are reported because the task rule ("product identity matches the printed identity, allowing spacing/case differences only") and the decode prompt pull in opposite directions. The SYSTEM prompt itself tells the model to expand brand abbreviations when confident (GG = Green Giant is its own example). The **identity metric** counts a line correct when the decoded product is the same product as printed (abbreviation expansions such as Bh to Boar's Head, BC to Betty Crocker, KS to Kirkland Signature, and word reordering from the brand/product/detail field split are accepted) and the paid price is exact. The **strict-string metric** accepts only spacing and case differences. All 10 lines that separate the two metrics are abbreviation expansions, and nothing else. Recommendation: judge the gate on the identity metric. A strict-string gate would penalize behavior the prompt orders. Note: if my subset's 32/32 was scored strict-string, the two subsets are not comparable, and the combined strict figure (86.15%) is the conservative reading.
## Isolation compliance
- All public-set data lives ONLY under `~/workspace/meta-hackathon-2026/spike2/public-set/` (`photos/`, `labels/`, `results/`, plus `decode_public.py`, `score_public.py`, and this report).
- The only files on my side read were `spike2/decode_photo.py` and `spike2/run_one.py` (to reuse the exact SYSTEM prompt and call pattern) and my subset's result line in `spike2/SPIKE2-REPORT.md` (4 receipts, 32/32) for the combined figure.
- Nothing was written to `spike1/`, `spike2/photos/`, or `spike2/results/`. No public item entered any dictionary, ledger, or pattern file. No shared state exists between calls.
## Sourcing shortfall (stated plainly)
Target was 16 public receipts; **9 were obtained** (98 labelable lines). The limit was whether the lines could be labeled by hand, not search effort (about 25 image searches plus dataset, archive, and blog sweeps). Real receipts that were found but rejected because the labels could not be trusted:
- Trader Joe's, 26 items, $52.18: the holder's thumb and a fold cut off the left edge of nearly every product name. Only about 4 lines could be labeled as printed.
- Fred Meyer (Kroger banner), 15 items, $60.69: photographed at an angle. The item text was too small to transcribe reliably after rotation and upscaling.
- Target, ~7 lines, about $123.90: skewed photo, item names not reliably readable.
- Costco (Roseville) and a second Costco photo: thumbs hide the product names.
- CVS/Walgreens photos found were coupon tails only (no item lines); one WIC slip showed quantities without prices and part of a card number (skipped on both grounds).
- Rejected outright: fake/AI sample receipts, rendered samples, non-US receipts (Canada, UK, Serbia, Russia, Australia, Colombia), staged prop receipts, and photos too blurry to label.
Mix targets. Long receipts (15+ lines): target 4, got 1 (Alexander's, 27 lines); the other long candidates above could not be labeled. Chains beyond Safeway: target 3, got 8. Adverse conditions: hand-held (pub-01, 03, 08, 09), crumpled and rotated 90° (pub-02, kept in original orientation), faded (pub-04, 06), dark and thumb-occluded (pub-08, 09): covered.
## Coverage by store
| File | Store | Lines labeled | Lines correct (identity) | Total | Condition |
|---|---|---|---|---|---|
| pub-01 | Walmart (Meridian, ID) | 13 | 13 | 28.48 | hand-held |
| pub-02 | Publix (Atlanta, GA) | 10 | 10 | 66.58 | crumpled, rotated 90° |
| pub-03 | H-E-B | 12 | 12 | 36.92 | hand-held, outdoors |
| pub-04 | Alexander's Supermarket | 27 | 22 | 39.99 | flat scan, faded dot-matrix |
| pub-05 | Westside Market (New York, NY) | 13 | 10 | 55.55 | flat lay, 2013 |
| pub-06 | Kroger | 3 | 3 | 8.07 | faded, wrinkled |
| pub-07 | Publix (Weston, FL) | 2 | 2 | 4.65 | hand-held; lower receipt in frame |
| pub-08 | Food Lion (Rutherfordton, NC) | 8 (+5 unreadable) | 8 | not in frame | dark, thumb-occluded, PARTIAL |
| pub-09 | Costco (Tigard, OR) | 10 (+5 unreadable) | 10 | not in frame | hand-held, thumb-occluded, PARTIAL |
Chains represented: Walmart, Publix (×2), H-E-B, Kroger, Costco, Food Lion, Alexander's, Westside Market (independent). That is 7 chains (Publix appears twice) plus 1 independent, all beyond my set's Safeway.
## Errors, ranked
1. **Product wrong: 5** (all pub-04, faded vintage print). Digit and code misreads inside abbreviated strings: CHRY48 read as CHRY8, 8PK as SPK, 1884 as 18&4, T/HOUSE16BN as 168N, 96ZPP499 as 962PP499. Four of five carried `needs_user_check`. All five prices were exact.
2. **Price wrong: 3** (all pub-05), a three-way price rotation: ORANGE JUICE 8.99 read as 3.99, NAPOLITANER 2.99 as 4.05, PIZZA POCKETS 4.05 as 7.99. The rotation keeps the line sum, so the arithmetic check alone would not catch it. The ORANGE JUICE error was high-confidence (0.9, no check flag). It is the one silent error in the set.
3. **Item-number wrong: 3 of 46 checked** (reported separately, not in the pass mark): Walmart RED BELL …4688KF read as …4680KF; Westside BOUNTY …81539 as …81599, and NAPOLITANER 0765600721 as 07858000721.
4. **Missed lines: 0. Extra lines: 0.** The model emitted exactly the label line count on all 9 receipts, including the 10 label-unreadable lines on the two partial receipts, which it returned as partial reads with `needs_user_check` (excluded from scoring, not counted as extra).
Arithmetic match: **7/7** receipts where a total is in frame (decoded line sum within 0.05 of the subtotal, and decoded total equal to the printed total). pub-08/pub-09 are not scorable (totals out of frame); on both, the model reported the visible-line sum as its total.
Conventions and notes:
- Savings folds were handled correctly everywhere they occur: pub-02 (0.50, 0.68), pub-03 (25%-off line: list 9.32 / paid 6.99 / discount 2.33), pub-09 (Costco instant savings folded into CRNCHMASTER: paid 5.99, discount 2.00). The pub-09 label records the printed line amount (7.99); the line is scored correct under the established spike-1/spike-2 fold convention, with the 2.00 carried in the discount field.
- `items_sold` matched the label on all 3 receipts that print it (pub-01: 14, pub-03: 12, pub-06: 3). Where not printed, the model inferred a count (pub-02: 10, pub-04: 27, pub-05: 14) instead of returning null. Harmless here, but worth a prompt note.
- pub-07: the model duplicated the brand token into the product field (a schema artifact); identity was unaffected under token-set comparison.
- Decode cost: 72,344 tokens total for 9 photos (23,379 prompt + 48,965 completion).
## Files
- `photos/pub-01.jpg` … `pub-09.jpg`, `photos/manifest.json` (source + store + notes per file)
- `labels/pub-01.json` … `pub-09.json` (hand labels, written before decoding)
- `results/pub-01.json` … `pub-09.json` (raw decode results), `results/scores.json` (full scoring detail)
- `decode_public.py`, `score_public.py` (reproducible pipeline)
---
> **Superseded 2026-11-02.** This report recommends switching receipt reading to Wonder through Nov 4. The rule now in force is in [`assistant/docs/MODEL-POLICY.md`](/docs/architecture/model-policy): Spark reads first, Wonder re-reads flagged receipts, through Nov 9. The measurements below still stand.
# Spark vs Wonder on receipt decode: accuracy and latency
**Headline: on the Gate 0 public set, Wonder reads receipts more
accurately than Spark. Its errors are the kind the system catches,
while Spark's worst error is the kind it cannot. Wonder costs about 9 times more
per receipt and takes about 4 times longer.**
One-sentence result the submission may quote: *"On the 9 public Gate 0
receipt photos (98 hand-labeled lines), Muse Wonder 1.0 scored 97/98
lines correct on identity+price against Spark 1.3 Standard's 92/98,
with zero price errors and zero silent errors; Spark repeated the
set's one known silent failure (a price rotation on Westside Market's
receipt, read at 0.85 confidence with no check flag), while Wonder
read that line correctly at 0.95."*
Quotable form: **"Wonder 98.98% vs Spark 93.88% identity+price
(hand-adjudicated under the Gate 0 definition); price errors 0 vs 4;
silent errors 0 vs 4; arithmetic 7/7 both. Latency: Wonder mean 34.4 s
vs Spark 8.6 s per receipt photo. Cost: $0.118 vs $0.013 per receipt."**
## Method (stated before the numbers)
- **Data.** The Gate 0 public set only: 9 receipt photos
(`~/workspace/meta-hackathon-2026/spike2/public-set/photos/`), 98
hand-labeled lines, labels written before any decode ran. No receipts of mine
anywhere in this eval.
- **Identical conditions.** Both models ran the app's own photo
pipeline: the system prompt from `assistant/prompts/decode_photo_v1.md`
loaded the way `api/decode/service.py: load_photo_prompt()` loads it,
the same message shape as `decode_photo()`, temperature 0.1, one
stateless call per receipt. Spark ran at the app's production
configuration (`muse-spark-1.3`, `reasoning_effort=low`, the
`meta_client.chat` default). Wonder ran at its default
(`muse-wonder-1.0`, no `reasoning_effort` sent). This compares the
configurations the app would actually ship, and the asymmetry is
deliberate: effort is part of a model's production configuration.
- **Scoring, two columns.** Gate 0's verdicts were hand-made under a
written definition (identity = same product as printed, prompt-
sanctioned abbreviation expansions preserve identity, price exact
with the savings-fold allowance). This eval first ran an automatic
scorer (`wonder_score.py`) implementing that definition; the scorer
was **calibrated until it reproduced every Gate 0 hand verdict on
the recorded Spark run exactly** (90/98 identity, 80/98 strict, same
error lines, arithmetic 7/7). The automatic column is that scorer,
unmodified, on both fresh runs. Because the scorer's expansion table
covers the abbreviations the *recorded Spark run* used, every
automatic non-correct line in both fresh runs was then
**hand-adjudicated under the same written Gate 0 definition**. That is the
adjudicated column. Each ruling is listed by name below. Reporting
only the automatic column would penalize Wonder for doing what the
prompt instructs (expanding abbreviations); reporting only the
adjudicated column would hide the judgment calls. Both are printed.
- **Latency** is wall-clock per API call, one run, sequential calls.
**Cost** is computed from each response's usage block at the
MODEL-POLICY prices (Spark Standard $1.25/$4.25, cached $0.15;
Wonder $4/$20 per 1M) via the repo's own `estimate_cost`.
- **Judge spot check.** 20 synthetic grouped items with expected
verdicts set from the judge prompt's rules, mirroring the production
judge-eval archetypes (weekly staples, December eggnog, twice-a-year
spices, gift cards, and the 2-buys-703-days-apart foil case). Run
through the app's own `api/judge.py` batch path, one call per model.
Synthetic because the real judge inputs are my purchase
history; no personal data was sent to either model.
## Results
| Metric (9 receipts, 98 lines) | Spark 1.3 Std (low) | Wonder 1.0 (default) |
|---|---|---|
| Identity+price, automatic scorer | 92/98 = 93.88% | 86/98 = 87.76% |
| Identity+price, hand-adjudicated | 92/98 = 93.88% | **97/98 = 98.98%** |
| Strict-string, automatic | 77/98 = 78.57% | 72/98 = 73.47% |
| Price errors | **4** | **0** |
| Product errors (adjudicated) | 2 | 1 |
| Silent errors (wrong, no check flag) | **4 of 6** | **0 of 1** |
| Arithmetic (of 7 scorable) | 7/7 | 7/7 |
| Latency mean / median / p95 | 8.6 s / 8.8 s / 11.9 s | 34.4 s / 30.0 s / 60.5 s |
| Cost per receipt (total) | $0.0134 ($0.12) | $0.1180 ($1.06) |
Per receipt (identity, adjudicated; latency s; cost $):
| Receipt | Lines | Spark | Wonder | Spark lat | Wonder lat |
|---|---|---|---|---|---|
| pub-01 Walmart | 13 | 13/13 | 13/13 | 9.2 | 30.5 |
| pub-02 Publix (rotated, crumpled) | 10 | 10/10 | 10/10 | 8.7 | 30.0 |
| pub-03 H-E-B | 12 | 12/12 | 12/12 | 7.3 | 26.4 |
| pub-04 Alexander's (faded dot-matrix) | 27 | 25/27 | 26/27 | 11.0 | 47.4 |
| pub-05 Westside Market | 13 | **9/13** | **13/13** | 11.9 | 28.3 |
| pub-06 Kroger | 3 | 3/3 | 3/3 | 6.3 | 15.4 |
| pub-07 Publix | 2 | 2/2 | 2/2 | 4.8 | 11.9 |
| pub-08 Food Lion (partial) | 8 | 8/8 | 8/8 | 8.8 | 59.5 |
| pub-09 Costco (partial) | 10 | 10/10 | 10/10 | 9.6 | 60.5 |
Judge spot check (20 synthetic items, one batch call per model):
**both models 20/20 on class and 20/20 on list admission**, including
the foil archetype (both OCCASIONAL, so the production judge's one clear
error from the 2026-10-27 eval was not reproduced by either model on
the clean synthetic version). Spark: 5.9 s, $0.0067 per batch. Wonder:
12.2 s, $0.0595 per batch. The synthetic items are deliberately
clear-cut, so this instrument has a ceiling: it shows Wonder does not
judge worse. It cannot show that Wonder judges better.
## Where the reads differ, by name
### Lines Spark got wrong that Wonder got right
- **pub-05, the price rotation. All four of Spark's price errors are
on this one receipt, and Wonder made none, here or anywhere.**
ORANGE JUICE $8.99 read as $3.99 (confidence 0.85, **no check
flag**); ROYAL GALA $5.99 as $2.99 (0.80, no flag); NAPOLITANER
$2.99 as $4.05 (0.50, flagged); PIZZA POCKETS $4.05 as $10.99 (0.40,
flagged). Wonder read all four at their true prices, 0.88 to 0.95
confidence. The ORANGE JUICE line is the same line that was Gate
0's one silent error in the recorded run: a rotation preserves the
line sum, so the arithmetic check (7/7 for both models) **cannot**
catch it. This is the most dangerous error class in the product,
and in this run Spark produced it twice silently (0.85 and 0.80, at or
above the app's 0.8 auto threshold, so both would auto-post) and Wonder
produced it zero times.
- **pub-04 idx16, SNK FRT/S CHRY48** (faded dot-matrix): Spark read
"CHRY 8", dropping the 4, at 0.82 confidence and unflagged, so it
would auto-post as the wrong product. Wonder read CHRY48 correctly
and still flagged the line for a user check (0.58), the correct
behavior on a hard line.
### Lines Wonder got wrong that Spark got right
- **None, after adjudication.** Wonder's only surviving error is one
Spark shares (below). Its 11 other automatic-scorer failures were
all the same behavior, expanding or completing what Spark leaves
abbreviated, and every price was exact:
- Brand expansions: LNF to "Light & Fit" (pub-01), PF to
"Pepperidge Farm", KB to "Keebler" and CKS to "Cookies", OS to
"Ocean Spray" (all pub-08), HEB to "H-E-B" (five pub-03 lines:
the same brand, hyphenated, twice with word order normalized).
- Two borderline completions, flagged as judgment calls: RED BELL to
"Red Bell Pepper" (pub-01) and STRKST to "StarKist TUNA …"
(pub-08). The printed abbreviation denotes exactly that product;
the added word names its class. A stricter adjudicator could
score these wrong; scoring both wrong still leaves Wonder at
95/98, ahead of Spark.
- pub-03 idx11 carries the promo text in its detail ("Special
Today- 25.00% Off orig 9.32 Ea."). Gate 0 hand-scored the
identical Spark read of this line correct. Same ruling here.
### Lines both got wrong
- **pub-04 idx21, M&M 15%FREE 1884 read as "18&4"**: the digit-string
misread both models made at Gate 0 too. Wonder flagged it (0.72);
Spark did not (0.85).
### The strict-string inversion
Wonder's strict score (72) is lower than Spark's (77)
because it expands: "Ranch Dressing" is the right product and the
wrong string. Spark's strict number is itself both helped and hurt by
receipt furniture. Its pub-02 reads are verbatim but carry the receipt's
"F" tax-flag glyph in every detail, which strict counts as 0/10 on
that receipt. Strict-string measures transcription style, not
correctness. The identity metric is the one the gate was judged on,
and it is the one to quote.
## Recommendation
- **Photo decode (receipt finish to review queue): flip to Wonder.**
The accuracy gap concentrates exactly where the product is weakest:
silent price rotations that arithmetic cannot catch (Spark 4 price
errors, 2 auto-posting silently; Wonder 0 and 0) and faded-print
digit reads. Decode is asynchronous in the product (the user takes
a photo and reviews a queue), so Wonder's latency can be absorbed
there. A wrong price that posts silently cannot. At $0.118/receipt
the credit cost is not a concern through Nov 4 (about $118 per 1,000
decodes against about $147 of remaining credit).
- **Staple judge: stay on Spark.** The spot check separates nothing
(20/20 vs 20/20), and the only measured production number, 0.79
precision against the cadence proxy, belongs to Spark's run. Paying
9 times more and waiting twice as long for an unmeasured benefit is not justified. Revisit
only with a harder, labeled judge set.
- **Ask Arya / chat: stay on Spark.** Interactive paths depend
on latency. Wonder's profile (12 s for a text batch here, 26 to 60 s
with an image) is the wrong shape for chat. This eval produced
no chat-accuracy evidence for either model.
- **Shelf labeling (Quest, offline batch): flip to Wonder is
defensible but unmeasured head-on.** The supporting evidence is
indirect. Wonder's expansions in this eval were consistently the
correct brand renderings (Keebler, Pepperidge Farm, Ocean Spray,
Betty Crocker, Finlandia), which is the labeling skill, and
labeling runs offline where latency does not matter. If it flips,
say "on decode evidence," not "measured."
- **PDF/text decode: no claim.** The public fixtures are photos; the
text path shares the prompt family but was not run here.
**The latency caveat, quantified:** Wonder's mean is 4.0 times Spark's
(34.4 s vs 8.6 s), its p95 is 60.5 s vs 11.9 s, and its two slowest
calls (59.5 s, 60.5 s) were the two partial receipts. One further
signal: a duplicated dispatch during this eval produced a second,
unscored Wonder run whose latencies averaged about 76 s (max about 104 s) and
which emitted 24 lines for pub-02 (duplicated reads of the rotated
receipt, vs 10 in the scored run). Wonder's tail is heavier and less
stable than the scored run alone shows. Any judge-facing Wonder path
needs a spinner the demo can survive and a timeout with a Spark
fallback behind it.
**Policy note:** the repo's MODEL-POLICY.md bans Wonder from volume
paths, and AGENTS.md reserves policy changes to me. The
recommendation above is scoped to the judge-facing demo paths through
Nov 4 (demo household + my own demo use), not a production
policy change. No production code was changed by this eval.
## Caveats
- One run per model per receipt, temperature 0.1. Variance is real.
Fresh Spark (93.88%) and Gate 0's recorded Spark (91.84%) differ by
3 lines on pub-04 and pub-05, in opposite directions: Spark fixed 3 of
pub-04's 5 recorded digit errors this run and made pub-05's
rotation worse (4 price errors vs 3). Quote the gap between models
(same-day, same-prompt runs), not either absolute as a constant.
- 9 receipts, 98 lines, US grocery only. It is the same small set as
Gate 0, with its known skew (one long receipt; two partials whose
totals are out of frame, hence arithmetic scored on 7).
- The adjudicated column rests on 11 rulings, each named above; the
two borderline completions are marked. The automatic column is
printed alongside so a reader who rejects the rulings still has a
consistent number. On that number Wonder loses, and that is
itself a finding: the models differ in transcription style, and
a scorer built from one model's style misjudges the other's.
- The judge spot check is a ceiling-limited instrument (synthetic,
clear-cut by design). It is evidence of no regression, not of
superiority, for either model.
*Spend: $4.47 total. Spark decode $0.12, Wonder decode (scored run)
$1.06, the duplicated unscored Wonder run $3.22, judge spot checks
$0.07. Runner: ~/workspace/arya-eval/wonder_run.py; scorer:
~/workspace/arya-eval/wonder_score.py (calibration: reproduces Gate 0
hand verdicts exactly); spot check: ~/workspace/arya-eval/
judge_spotcheck.py; raw outputs and per-line scores in
~/workspace/arya-eval/wonder-results/.*
---
> Staple-judge evaluation, 2026-10-27, on my own 114 Costco receipts
> (shared with my approval). The raw receipts and state files are not in this repo.
# Staple-judge accuracy: one defensible number
**In three lines.**
1. Of the 52 items the staple judge admitted to the list on my 114 receipts, 41 (79%) are staples by purchase cadence alone. No human labels, no model calls, $0.
2. The 11 others are mostly slow-cycle items bought 3 times; one is a clear error.
3. The figure moves with the proxy's definition (0.98 with a looser bar, 0.63 with a stricter one), so quote it with the definition, for one household.
**Headline: precision 0.79 (41 of 52).**
One-sentence definition the submission may quote: *"Of the 52 items Arya's
staple judge admitted to the self-writing list, 41 (79%) were independently
corroborated as behavioral staples by purchase cadence alone (bought on at
least 4 separate receipts spanning at least 3 different months, returns
netted), with no human labels involved."*
Quotable form: **"Judge precision 79% against a cadence-derived ground
truth (41/52 admits corroborated); the 11 uncorroborated admits are
dominated by slow-cycle consumables bought 3 times, and only one admit
(Reynolds Foil, 2 buys 703 days apart) is a clear error."**
## Method (stated before the numbers)
- **Data.** My live state, copied read-only from the Oracle VM
(Docker volume `arya-data`, per-user dir): 114 real Costco receipts,
2023-10-25 to 2026-10-26; the dictionary (458 entries); and `judged.json`,
the judge's cached verdicts. No model calls were made; spend $0.
- **Identical pipeline.** The analysis re-ran the app's own code on the
copies: ledger build (discounts folded, fee SKUs excluded), dictionary
grouping (Curator merges applied), and the pattern engine's returns
netting (FIFO, kept quantity only, FOR-OTHERS lines excluded). Item
identities are therefore exactly the identities the judge saw.
- **Admitted set.** Items whose cached verdict has `list_ok = true`
(class STAPLE, or SEASONAL). 156 raw verdicts collapse to 155 distinct
grouped items under the current group map (one Curator merge); 52 of
them are admitted (43 STAPLE, 9 SEASONAL).
- **Ground-truth proxy (behavioral staples).** An item is a proxy staple
if, after returns netting, it was bought on **≥ K distinct receipts**
spanning **≥ M distinct calendar months**. Primary setting **K=4, M=3**.
Justification: the corpus holds ~3 receipts/month over 36 months, so
4 separate trips in 3 different months is the weakest pattern that
excludes one-trip stock-ups, gift/event buys, and two-trip trials, while
still catching a quarterly Costco rhythm. The judge's own eligibility
bar is 2 buys, so the proxy is deliberately stricter than the judge:
it corroborates the judge's rule but does not re-derive it.
- **Metrics.** Precision = admitted ∩ proxy / admitted. Recall =
admitted ∩ proxy / proxy. Disagreements are listed by name below; the
names matter more than the decimals.
- **What this number is NOT.** It is not accuracy against a hand-labeled
gold set, because no such set exists for this household. It measures agreement
between a reasoning judge and an independent cadence proxy. Where they
disagree, either can be right. The disagreement lists say which, case
by case.
## Results
| Proxy | Proxy staples | Precision (all admits) | Precision (STAPLE only) | Recall |
|---|---|---|---|---|
| K=3, M=2 | 88 | 51/52 = **0.98** | 0.98 | 0.58 |
| **K=4, M=3 (primary)** | 55 | 41/52 = **0.79** | 0.74 (32/43) | 0.75 (41/55) |
| K=6, M=4 | 35 | 33/52 = 0.63 | 0.65 | 0.94 |
At the primary setting: all **9 SEASONAL admits are corroborated (9/9)**.
Every disagreement below is at K=4, M=3.
## Disagreements at the primary setting
### Judge admitted, proxy does not corroborate (11 of 52)
Ten of the eleven miss the bar by exactly one receipt or one month. All
eleven are slow-cycle or short-window items where the judge's stated
reason is plausible on its face:
- **Three buys, three months (7):** Avocados; Hand Wash; Kirkland
Signature Bodywash; Kirkland Signature Pink Salt; Lubriderm; Organic
2% Milk; Power Up Mega Omega Trail Mix. Soap, lotion, salt and foil are
bought rarely *because* they last; a cadence bar built for food
under-counts them. Avocados (Nov 2023 to Jan 2024, then nothing) is the
weakest of this group in hindsight.
- **Four buys inside two months (3):** Kirkland Signature Butter Quarters,
Kirkland Signature Organic Whole Milk, Mini Cukes: a real
rhythm of about once every two weeks in Nov and Dec 2023 that does not reappear under the
same SKUs later in the corpus (bought elsewhere, or re-SKU'd).
- **The one clear error (1):** Reynolds Foil (printed "Reynlds Foil" on the receipt): 2 buys, 703 days apart,
judged STAPLE ("long-life foil rebought"). It is also the only
uncorroborated admit under the looser K=3, M=2 proxy.
### Proxy staple, judge kept off (14 of 55)
All fourteen carry the class OCCASIONAL, and every judge reason cites gap
irregularity: Buter Croiss (5 receipts), Cantaloupe (6), KS Lightly
Breaded Chicken Breast Chunks (5), KS Pine Nuts (4), KS Pistachio (5),
KS Towel (5), KS Walnuts (6), Mixed Pepper (4), Nmcholstoff+ (4),
Organic Broc 2Lb (4), Organic Brussels (4), Organic Gre Bean (4),
Pumpkin Seed (5), Red Onions (4). None were left unjudged. The judge
saw all of them and declined. This is the product's stated policy made
visible (precision over recall: a missed item costs the user one tap, an
unnecessary suggestion costs trust). Twice-a-year pantry goods (walnuts,
pistachios, pine nuts, pumpkin seeds) are the arguable cases: a human
might call them staples; the judge calls a 150 to 250 day median gap
"occasional," and under its own prompt that is a defensible call.
## Second signal: my behavior
- **Flags:** my state has no `flags.json`. I have recorded no
list edits (no "never suggest", no dismissals, no manual adds) against
the judge's output. There is therefore no me-vs-judge settlement
signal to report; absence of overrides is noted, not scored.
- **The live list:** the cart prepared from the current smart list holds
5 items. The 4 pattern-surfaced items (Organic Raspberry, Mandarins,
Kirkland Signature Almonds, Slice Almond, the list that went from 16 to 4 in
the first judge sweep) are **all corroborated proxy staples at the
primary setting (4/4)**. The fifth, Power Up Mega Omega Trail Mix, is
judge-admitted (STAPLE, 3 buys at a 23-day median gap) and sits in the
proxy-miss group above by one month of spread. My behavior (it is
on my active list, qty 2) sides with the judge.
## Sensitivity and how to read the headline
The precision figure depends on the definition, by construction. Loosen the
proxy to K=3, M=2 and precision rises to 0.98, with recall falling to
0.58 (the judge is far more selective than a loose cadence bar). Tighten
it to K=6, M=4 and precision falls to 0.63, because a
heavier proxy excludes mid-frequency items the judge reasonably admits.
K=4, M=3 was fixed before the results were seen, as the weakest bar that
still excludes stock-ups and trials; 0.79 is quoted with that definition
attached, never bare.
## Caveats
- One household, one retailer (Costco), 114 receipts. Staples bought at
other stores are invisible to both the judge and the proxy.
- Verdicts are cached at the buy-count current when judged (re-judging
triggers at +2 buys); the proxy uses final corpus counts. A verdict can
therefore predate an item's later purchases. This cuts against the
judge in the recall direction only.
- Grouping is the Curator's current grouping; a wrong merge would move
both sets together, so grouping errors mostly cancel in precision.
- SEASONAL admits are month-gated in the live list logic; this eval
scores the judge's `list_ok`, which is the judge's own decision, not
the seasonal gate's.
*Analysis: analysis script and state copies are kept out of the repo (my data).*
---
# Run Arya on your own computer
For judges and anyone who wants to try Arya locally with their own Meta key.
The live demo needs none of this: https://arya.teamnightshift.org, then "Try it
now".
## What do I need?
- **Docker** with Compose (Docker Desktop, or Docker Engine and the Compose
plugin). Nothing else: no Python, no Node.
- **A Meta Model API key** for the AI parts: receipts, Ask, photos and voice.
Make one at dev.meta.ai (API keys, then Create API key). It starts with
`LLM_`. Without it, lists, the demo household and Quest still run, but
anything that reads a photo, answers in Ask or listens to your voice says
"This copy of Arya has no Meta key, so AI features are off".
- **A Google Places API key** to add stores on your own account. Adding a
store searches the Google Places API. Without the key, the store picker says
"Google search is not set up" and there is no way to type a store in by
hand. The demo household comes with its own stores, so the demo needs no
Google key.
- Everything else is optional.
- A phone-size window. On a laptop, make the browser about 390 px wide. Camera
and microphone steps need a phone, and a phone browser only allows the camera
on https. On a phone reaching your computer over plain `http://` on your
network, use the receipt upload instead of the camera.
## How do I start it?
Get the code with a shallow clone, which is small and fast. A normal
`git clone` also works and brings the full history (much larger):
```bash
git clone --depth 1 https://github.com/vibhortayal/arya.git && cd arya
```
Then, from the repository root:
```bash
cp .env.local.example .env # then put your key on the MODEL_API_KEY line
docker compose up --build
```
Open http://localhost:8080. The first build takes a few minutes. On the first
visit a "Create your account" sheet opens. That makes the one real account on
this copy: a username (lowercase letters, digits and `_ % + @ -`, and `.` after the first character) and a password. Use it
to snap or upload your own receipts and see them read and saved. That is
Arya's core feature. Or tap **Close** on the sheet and then **Try it now** for the demo
household. The demo keeps nothing. It reads a receipt live on the screen but does not
save it. Ask, photos of a list and voice work there with your key. The first start also writes the demo
household into a Docker volume. Later starts keep what is there.
## How do I get a Meta API key?
To run the live model calls from your own checkout, make a Meta Model API key
just for this (dev.meta.ai, API keys, Create API key). Put it on the
`MODEL_API_KEY` line of your local `.env`. Never commit a filled `.env`. The
`MODEL_API_KEY=` line in `assistant/deploy/.env.example` is blank on purpose.
The key is sent only to `api.meta.ai`. Meta keys have no spending cap of their
own, so set a usage limit on your Meta billing page, and delete the key at Meta
when you are done. Photo receipts: see `ARYA_PHOTO_READER` below.
## Settings (`.env`)
Two matter: `MODEL_API_KEY` for the AI, and `GOOGLE_PLACES_API_KEY` to add
stores on your own account. The rest are optional.
| Setting | What it does | Default |
|---|---|---|
| `MODEL_API_KEY` | Your Meta key. Receipts, Ask, photos, voice, notes and store scans use it. Lists, the demo and Quest run without it | empty (AI off) |
| `ARYA_STORE` | Where data is kept. `json` means files in a Docker volume | `json` |
| `ARYA_PHOTO_READER` | `auto`: Muse Spark reads a photo first, Muse Wonder re-reads one the checks flag. Or `spark`, `wonder` | `auto` |
| `ARYA_SPARK_TIER` | Spark runs on Meta's Standard plan | `standard` |
| `GOOGLE_PLACES_API_KEY` | Needed to add a store on your own account: the store picker searches the Google Places API. Empty means the picker says "Google search is not set up". The demo needs no key | empty |
| `BASETEN_API_KEY` | Optional. Muse Spark Open, the staple judge's second seat. Leave empty to skip that check | empty |
| `ARYA_LOCAL_PORT` | Use another port if 8080 is taken | `8080` |
Ignore everything else in `assistant/deploy/.env.example`: Supabase, email,
Google, push and the other settings are for the live site. They default off.
## What does it cost me?
Every model call runs on your key and your Meta account. The demo household
keeps its visit limit locally too: about $0.40 of live reads per visit and
$2 a day for the demo (`ARYA_DEMO_SESSION_BUDGET_USD`,
`ARYA_DEMO_DAILY_BUDGET_USD`). Pricing is in
[`assistant/docs/MODEL-POLICY.md`](/docs/architecture/model-policy).
## How do I stop it, reset it or change the port?
- After editing `.env`, run `docker compose up -d`. `docker compose restart` does
not re-read `.env`.
- Stop: press Ctrl+C, or `docker compose down`. Your data stays.
- Start over (wipes the volume; the next start writes the demo household again):
`docker compose down -v`.
- Another port: set `ARYA_LOCAL_PORT` in `.env`.
## How is it different from the live site?
- One container, plain `http://localhost`, no Caddy and no certificate. The
session cookie is set to work over plain http (`ARYA_INSECURE_COOKIE=1`).
- JSON files in a volume instead of Supabase. No email, so you sign in with the
username and password you made on the first visit.
- Demo Town (the simulated neighborhood, labelled simulated) is on, as on the
live demo, and registration is open, so **Start my own list** on the landing
page makes any number of accounts. Groups are on, so two local accounts can
share a list. There is no waitlist, no Google sign-in and no email code. The
`compose.yaml` here does not touch the live deploy, which uses
`assistant/deploy/docker-compose.yml`.
Next, a step-by-step tour of every feature on your copy:
[`docs/TRY-THE-FEATURES.md`](/docs/try-the-features).
## Can I run it without Docker?
The Tester followed this route step by step from a fresh clone, without a
key (the receipt step was not run). The Docker route above was run twice on a Linux server with no Meta key. A cold
build (about 55 seconds) from a fresh shallow clone, in a browser at 390x844, worked for
start, `/health`, creating an account, signing in again, the demo, the no-key Ask
and receipt messages, and Start shopping. An earlier run with cached
build layers, through direct requests, showed that editing `.env` and the port setting worked.
The cold build needed that test machine's proxy certificate added, because its
network re-signs HTTPS (see "What if something goes wrong?"). Reading a receipt,
photos, voice and a working Ask answer need a real key and were not run on either
route. Still untested: a real Meta key, Docker Desktop on a Mac or Windows
laptop, and a real phone.
You need Python 3.12 or newer (3.13 worked) and Node 22. From the repository root:
```bash
python3 -m venv .venv && .venv/bin/pip install -r assistant/api/requirements.txt
cd assistant/web && npm ci && npm run build && cd ..
../.venv/bin/python -m harness.make_demo_household data
MODEL_API_KEY= ../.venv/bin/python -m uvicorn api.main:app --port 8000
```
In `MODEL_API_KEY=`, put your key in place of ``, or nothing
at all for no-key mode (`MODEL_API_KEY=`). Pasting the angle brackets as they
are fails in bash. The snippet leaves your shell in `assistant/`.
Open http://localhost:8000. The address http://localhost:8000/health answers `ok` when the
server is up. These steps were tested on Linux only. macOS and Windows have not
been tried. Without a key, Ask shows the "no Meta key" message.
On this route its last words, "docker compose up -d", do not apply. Instead, set
`MODEL_API_KEY` and start uvicorn again. To open it from a phone at
`http://:8000`, make uvicorn listen on your network address (its
`--host` option) and set `ARYA_INSECURE_COOKIE=1` before it. Without that, sign-in may not stick
over plain http. I have not tested this on a phone. For front-end work, `npm run dev` in
`assistant/web` starts Vite.
## Let your AI agent set it up
If you use a coding agent that can run commands on your computer, paste this
prompt to it. It uses the no-Docker route, which the Tester followed from a
fresh clone (without a key). Read the prompt first. It asks the agent to run
commands on your machine.
```text
Set up Arya on my computer so I can try it. Repository:
https://github.com/vibhortayal/arya
1. Clone the repository into a new, empty folder, cd into it, and open
docs/RUN-IT-YOURSELF.md.
2. Follow the section "Can I run it without Docker?" step by step. If
something goes wrong, troubleshoot it as you normally would to get Arya
running, and use "What if something goes wrong?" in the guide. Do not
edit any file that git tracks; the folders the guide's commands create
(.venv, node_modules, the web build, the data folder) are expected.
3. Use the no-key demo mode: run the uvicorn command with MODEL_API_KEY= left
empty (nothing after the equals sign). Do not ask me for a
key, a password or any secret, and do not paste or print one.
4. When the server is up, tell me the local address (http://localhost:8000),
that /health answers "ok", and the sign-in steps: tap Close on the "Create
your account" sheet, then tap "Try it now" for the demo household. Tell me
to make the browser about 390 px wide.
```
Without a key, anything that reads a photo or answers in Ask says "This copy of
Arya has no Meta key, so AI features are off". That is expected.
## How do I run the tests?
```bash
(cd assistant && ../.venv/bin/python -m unittest discover -s tests)
(cd assistant/web && npm test)
```
Current counts and known failures: [`STATUS.md`](https://github.com/vibhortayal/arya/blob/main/STATUS.md). One web test,
`pack.test.tsx` "arrow keys pack and skip", is flaky in a full run and passes alone. CI is
manual-only (`.github/workflows/tests.yml`).
## What if something goes wrong?
- **"This copy of Arya has no Meta key, so AI features are off. Put your key from
dev.meta.ai after MODEL_API_KEY= in the .env file, then run: docker compose up -d"**
`MODEL_API_KEY` is empty, or the container was not recreated after you
edited `.env`. Fix `.env`, then `docker compose up -d`.
- **Receipts says "Reading paused" and there is no key.** The same cause as
above: add the key and run `docker compose up -d`.
- **Port already in use.** Set `ARYA_LOCAL_PORT` to a free port.
- **A model error with a key set.** The key was rejected or has no credit.
Check it at dev.meta.ai.
- **The build fails to download packages, with a certificate error (pip or npm).** A
proxy that re-signs HTTPS is the usual cause. Build on a network without one, or
add your proxy's certificate to Docker.
- **Docker Hub answers `429 Too Many Requests`.** The pull limit for your address
is used up. Wait a few minutes and run the build again, or run `docker login`.
- **Status check.** `http://localhost:8080/health` answers when the app is up.
This is not the feature "Use your own Meta key" in the live app's Settings,
which is unfinished, switched off and not used (see [After the hackathon](/docs/roadmap)). Here
the key is the server's key, set in `.env`.
---
# Try every feature on your own copy of Arya
The builder wrote these steps and checked them on a local copy (`docker compose up --build`, http://localhost:8080) on 2026-11-02. "Needs your key" means the step calls a Meta model, so `MODEL_API_KEY` must be set in `.env`. Everything else works without a key.
Arya is a phone app. For the closest view, use your browser's phone mode (for example 390 × 844), or open the address from a phone on the same network.
## 0. How do I start it?
1. Copy the settings file and add your key: `cp .env.local.example .env`, then put your key from dev.meta.ai after `MODEL_API_KEY=`.
2. Run `docker compose up --build`, then open http://localhost:8080.
3. A **Create your account** sheet opens. You can either:
- Fill in a username and a password, then tap **Create account**. Use it for section 9, your own receipts.
- **Tap Close, then Try it now** to open the demo household (sections 1 to 8).
Sign-in on your own copy is a plain username and password (see [Run it yourself](/docs/run-it-yourself)). To make more accounts, use **Start my own list** at the bottom of the landing page.
## 1. What is the demo household?
Tap **Try it now**, then **Skip for now** (or **Show me every screen** for the 2-minute tour).
The household shops in **Demo Town**, a simulated neighbourhood of 800 households and 8 stores: Trader Jill's, Safeday, Costcozy, Whole Meadows, Aldo's, Targot, Sprout Street and Corner Market. Anything from the neighbours is marked **SIMULATED**.
**Today** shows these cards, top to bottom:
- **Next trip**: Trader Jill's, 4 items, with "Hummus 10 oz is out here today".
- **Near you today**: "Hummus 10 oz is out at Trader Jill's. 3 neighbors reported it… Safeday has it at $4.29" and "Whole Milk 1 gal is cheapest at Aldo's, $3.50".
- **Running low**: Whole Milk and Sourdough Loaf, each with how often you buy it.
- **Cheaper elsewhere**: "Pasture Eggs: cheaper at Costcozy, $6.03 there, $6.96 at Trader Jill's".
- **Use soon**: 6 things from your receipts that can expire.
- **Still good?**: one item to check.
- **Spending**: October total.
- **Price watch**: Sourdough Loaf is 6.6% more than in July.
## 2. How do I see a store's news and stock?
1. Tap **More**, then **Stores**. **My stores** lists Costcozy and Trader Jill's (starred), plus Aldo's and Safeday. Each is marked Demo Town (simulated).
2. Tap **Find stores** to search all stores by name, ZIP or address.
3. Tap **More**, then **Lists**, then **Trader Jill's**. The list opens with a heads-up: "Hummus 10 oz is out. Safeday has it…". Hummus is marked **Out here · 3 neighbors**.
4. Under **Suggested for Trader Jill's**, tap an item to see why Arya suggests it, then **+ Add**.
## 3. How do I make things happen in the town (Director)?
1. Open http://localhost:8080/?director=1 and go into the demo.
2. Tap **More**, then **Settings**. A **Director · Stage a moment for the demo** panel appears.
3. Tap **Eggs run out**, **Store closes**, **Saturday rush** or **Incident**, then go back to **Today** and the lists to see the new reports. **Reset** clears them.
## 4. How do lists and adding items work?
- Type into the box at the top of a list. You can type amounts too, like "3 avocados" or "1.5 lb salmon".
- Tap the circle to tick an item. Tap an item to edit its name, amount or note, or to move it to another list.
- Every item lives on a list. Adding from Today with two lists asks which list.
- **Needs your key:** the microphone (talk your list), or a photo of a paper list.
- Turn off your network (browser devtools, Offline), then add, tick and remove items. They save on the phone. Turn the network back on and they sync with a "Synced" note.
## 5. How do I take a shopping trip (Quest)?
1. On the **Trader Jill's** list, tap **Start shopping**. The quest walks you item to item. Tap **Got it** on each.
2. **Needs your key and a camera:** **Find it for me** (SAM 3.1 finds the shelf, and Spark checks it) and **Map my store** on the Quest tab.
## 6. How do I ask Arya something? (needs your key)
Tap **Ask**, then try:
- "We finished the Greek yogurt": the list updates.
- "Where is milk cheaper?": it answers from the receipts.
- "What's running low?"
## 7. How do I see my spending and prices?
- **Spending**: by month, by store and by item. Tap an item to see its price by store.
- **More, Receipts**: the household's 39 receipts. Search by store or amount.
## 8. What should I use soon?
On **Today**, tap **Use soon** and add dates. Arya reminds you before things go off. You can try the date-reading sample without a camera.
## 9. How do I read my own receipts end to end? (needs your key)
1. Sign in with the account from step 0, or make one with **Start my own list**.
2. Go to **More**, then **Receipts**, then **Add receipt**, and snap or upload a receipt photo or PDF.
- Spark reads it first (Wonder is the backup; see [the model map](/docs/judges#which-model-does-which-job)), and Arya checks the lines against the printed total.
- Items land in your data, and a store you choose for a list joins **My stores**.
The demo household reads a receipt live on the screen, but it does not keep it. Stay on the Receipts screen while Arya reads.
## 10. How do I share with my household (groups)?
1. Make a second account: open a private window and use **Start my own list**.
2. In the first account, go to **Settings** and tap **Start a group**. Give it a name, then tap **Invite someone** for a code (it works for 48 hours).
3. In the second account, join with the code.
4. Add an item in one window. It appears in the other.
## 11. How do I use Arya from my own AI app?
Go to **Settings**, then **About Arya**, then **Use Arya in Muse**, and tap **Make my code**. Paste it into Muse or any app that supports MCP connectors.
## How do I reset?
Run `docker compose down -v`, then `docker compose up`. You get a fresh copy with the demo household written again.
---
# Arya Lists API v1
Read and write your own Arya lists from another app or a script, and take
your data with you. The same personal API token that the agent connector
uses works here. Make one in Arya under Settings › Use Arya in another AI
app. The code shows once; a new code replaces it.
Base URL: `https://arya.teamnightshift.org/api/v1`
The same guide, shorter, is the public page at `/developers`, with a
five-minute quickstart in curl, Python and JavaScript. The interactive
reference (try the calls from the browser) is at `/developers/reference`.
`/api/v1` is frozen. Changes after it ships are additive only: new
endpoints, new optional fields, new response fields. Clients must ignore
fields they do not know. The machine-readable schema is at
`/api/v1/openapi.json` (public, no token needed).
```bash
export ARYA=https://arya.teamnightshift.org/api/v1
export TOKEN=... # your code
```
Every call sends `Authorization: Bearer $TOKEN`. The token reaches only
your own account's data. Cookie sign-in does not work on `/api/v1`.
## Lists and items
A list is `{id, name, stores, open_count}`. `open_count` is the number of
items not yet checked. The id `unfiled` is "Everything else": the items
that sit on no named list. An item is
`{id, name, qty, unit, checked, aisle, added_at, list_id}`.
- `aisle` is the department you set yourself, else `null`. Arya's own
guess from the name is not part of the API.
- `added_at` is a UTC time. It is `null` for items added before the API
existed and for items Arya suggested.
- `checked` is for the current shopping cycle, as in the app.
- An item sits on one list at a time.
### GET /lists
```bash
curl -s -H "Authorization: Bearer $TOKEN" $ARYA/lists
# [{"id":"unfiled","name":"Everything else","stores":[],"open_count":1},
# {"id":"l-1a2b3c4d","name":"Costco","stores":["Costco"],"open_count":3}]
```
### GET /lists/{id}
The list, plus `items`.
```bash
curl -s -H "Authorization: Bearer $TOKEN" $ARYA/lists/l-1a2b3c4d
```
### POST /lists/{id}/items
Add items. Send `items` (structured) or `text` (free text), not both.
At most 50 items per call.
```bash
# structured: a name, or an object with qty, unit and aisle
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"items":["oat milk",{"name":"rice","qty":2,"unit":"kg","aisle":"Pantry"}]}' \
$ARYA/lists/l-1a2b3c4d/items
# free text: split the way the app splits typed lists ("eggs x 3, bread")
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"text":"milk\neggs x 3\nbread"}' $ARYA/lists/l-1a2b3c4d/items
```
Reply:
```json
{"added": 2, "already": 1, "already_names": [{"name": "milk", "list": "Costco"}],
"also_on": [], "skipped": [], "items": [ ... ]}
```
- `already`: names that were already on this list. They are not added twice.
- `also_on`: names that are already on a **different** list. The API does
not move them and does not add them to this list; the reply says where
they are. To move one, `PATCH` its `list_id`. (In the app a person can
keep an item on more than one list. The API shows such an item once, under
its main list, and its ticks there are the main list's.)
- `skipped`: pieces of `text` that were not items (a bare number). A line with a sentence in it is skipped whole, so no piece of it is added.
- `items`: the items now on the list for the names you sent.
- `text` is split by fixed rules. The API makes no model calls.
### PATCH /items/{id}
Change any of `checked`, `qty`, `unit`, `list_id` (a list id, `unfiled`, or
`null` for Everything else), `aisle` (`""` clears it).
Nothing changes if any field is bad.
```bash
curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"checked":true}' $ARYA/items/manual-milk
curl -s -X PATCH -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"list_id":"l-9f8e7d6c","qty":2,"aisle":"Dairy"}' $ARYA/items/manual-milk
```
Reply: the item.
### DELETE /items/{id}
```bash
curl -s -X DELETE -H "Authorization: Bearer $TOKEN" $ARYA/items/manual-milk
# {"deleted":"manual-milk"}
```
## Export and import
### GET /lists/{id}/export?format=csv|json|text
`json` (the default) is `{"list": {id, name, stores}, "items": [...]}`.
`text` is a checklist, one item per line: `☐ rice 2 kg`, `☐ eggs x 12`,
`☑ milk`. `csv` has this header and these columns, in this order:
```text
name,qty,unit,checked,department,list
rice,2,kg,false,Pantry,Costco
```
`checked` is `true` or `false`. A cell that starts with `= + - @` gets a
leading apostrophe so a spreadsheet does not run it as a formula. Import
removes it.
```bash
curl -s -H "Authorization: Bearer $TOKEN" "$ARYA/lists/l-1a2b3c4d/export?format=csv" > costco.csv
```
### POST /lists/import
Send the file as the request body. The format is `csv` when the
`Content-Type` contains `csv`, else `text`. `?format=csv|text` overrides that.
`?list=Name` is the list to fill (default `Imported`). It is created if you
do not have it; names match without regard to case. In a CSV, a `list`
column per row wins over `?list=`. Only `name` is required in a CSV. A
text body takes the checklist lines above or any plain list. At most 500
rows.
```bash
curl -s -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: text/csv" \
--data-binary @costco.csv "$ARYA/lists/import?list=Costco"
# {"created":3,"skipped":0,"lists":[{"id":"l-1a2b3c4d","name":"Costco"}]}
```
`created` is the number of new items. `skipped` counts rows that were
empty, not items, repeated, over the 500 limit, or already on a list.
Checked state and department are applied to new items only.
## Your receipts and totals
Your own data only: the receipts **you** filed. Receipts filed by other
members of your household group are not returned. Nothing here is
inferred by Arya. Totals are sums of what your receipts say.
A receipt is `{id, merchant, date, total, subtotal, tax, items, reconciles,
has_photo}`. The detail adds `lines`: `{name, qty, unit_price, line_total,
line_type}` with `line_type` `item`, `discount` or `return` (returns are
negative).
```bash
curl -s -H "Authorization: Bearer $TOKEN" "$ARYA/receipts?limit=20&offset=0" # newest first, max 200
curl -s -H "Authorization: Bearer $TOKEN" $ARYA/receipts/ext-1a2b3c4d5e
curl -s -H "Authorization: Bearer $TOKEN" $ARYA/receipts/ext-1a2b3c4d5e/original > receipt.jpg
curl -s -X DELETE -H "Authorization: Bearer $TOKEN" $ARYA/receipts/ext-1a2b3c4d5e
```
`GET /receipts/export?format=csv|json` returns every receipt you filed.
The CSV has one row per line, with columns
`receipt_id,merchant,date,name,qty,unit_price,line_total,line_type`.
`/receipts/{id}/original` is the photo or PDF you uploaded, if Arya kept one
(`has_photo`). Filing a receipt through the API is not in v1; use the app
or the agent connector.
### GET /analysis/spending
```bash
curl -s -H "Authorization: Bearer $TOKEN" $ARYA/analysis/spending
# {"trips":2,"spend_total":11.5,"returns_total":4.0,
# "by_month":{"2026-10":11.5},"by_store":{"Costco":11.5,"Trader Joe's":0.0}}
```
`spend_total` adds up the receipt totals. `by_store` groups by the merchant
name printed on the receipt.
### GET /analysis/purchases?limit=50
What you bought and when, by the name Arya read from the receipt (max 500):
`{"items":[{"name","times_bought","qty","spent","first_bought","last_bought"}]}`,
most often bought first. Returns and discounts are not counted as buys.
## Errors
Errors are JSON, with the matching HTTP status:
```json
{"error": {"code": "not_found", "message": "no such list"}}
```
| Status | `code` | When |
|---|---|---|
| 400 | `bad_request` | a bad or unknown field, bad JSON, bad format |
| 401 | `unauthorized` | no token, a wrong token, or a revoked token |
| 404 | `not_found` | no such list or item (also for another account's ids) |
| 413 | `too_large` | body over 256 KB |
| 429 | `rate_limited` | over the hourly limit; see `Retry-After` |
| 500 | `internal_error` | our fault; try again |
## Limits
- 600 calls per account per hour, shared with the agent connector and
every other route a token reaches (`ARYA_MCP_RATE`). Over it you get `429` with a `Retry-After` header
in seconds.
- 256 KB per request, 50 items per add call, 500 rows per import.
## Later
Not in v1: webhooks, OAuth, sharing a list with another account, filing a
receipt, and Quest data. Arya's own inference (what is running low,
suggestions, aisle and store maps, prices from other households) is not
part of the API on purpose. The API is for your own data.
---
# After the hackathon
These are ideas for where Arya could go. They are not commitments, and I
need more time to work them out.
**What ships now is a limited-scope MVP:** Arya learns from receipts,
Ask, groups, Quest and the connector, as described in the README and
`STATUS.md`. **This page is what could come later.** Nothing on it is built
unless it says so, and nothing is promised for the submission. "Idea" means
nobody has designed it yet. Sources are named so you can check them.
## Devices
- **Native iOS and Android apps; the PWA stays.** Plan. The PWA was the
fastest way to build in a hackathon. Not designed or scheduled, and not in
any app store today.
- **Ray-Ban Meta glasses.** Idea, raised by me for the store scan.
Spark's check during the hackathon found no public live-camera API for
third-party apps. The flow that works today is to capture on the glasses,
sync to the phone and import the video there. The demo uses the phone camera
throughout. Source: `docs/quest/DISCUSSION.md` (item 38).
- **Meta Ray-Ban Display glasses.** Idea. Not researched yet. What is possible
depends on what Meta offers developers for that device.
## Store data and community
- **Capture more store data.** Item-level locations from scan captures,
reports that close themselves, flags that expire when nobody confirms them,
and mapping quietly from Find it for me frames. Source:
`docs/quest/MAP-MY-STORE.md`, "Not built yet".
- **Arya shoppers and community prices.** Opt-in prices, deals and live
presence (Quiet, Busy, Packed). Today the opt-in records a choice only. A
community figure would need 5 or more households over 30 days, a median and
no names. Source: `assistant/docs/SPEC.md` (FR-11.7), `assistant/docs/SCALE-SIM.md`.
- **Quest community layer.** Leaderboards, badges the data makes funny, team
Shop-Offs settled by receipt time and Clue Hunts. Designed, not built.
Source: `docs/quest/GAME-DESIGN.md`.
- **A store-data partnership.** Online store videos are off limits (copyright,
platform terms). A licensed source of in-store video and photos could power
better maps. See [`docs/MODELS.md`](/docs/architecture/models).
- **Density first.** Cover one metro's stores completely before expanding. A thin
layer over many stores is worth little. Source: `docs/quest/BUSINESS.md`.
- **Store maps at scale.** Maps already merge across shoppers. Not built:
resolving conflicts between maps. Source: `assistant/docs/SPEC.md`, "Not in scope".
- **Shared lists, further.** Not built: split lists inside a group, and sharing a
list with another account through the API (`docs/API.md`, "Later").
- Item and store profile pages; split the list (kids or a partner); friend run
notifications (designed, not built); reimbursement tracking for items bought
for others.
- Invite links work today (`/?join=CODE` opens Join a group). Ideas for growth
as a community product, none designed: shared recipe or meal lists, and a
public page for a store's map status.
## AI and open weights
- **Bring your own Meta key.** Decided against for now. The code is in the repo,
unfinished and switched off (see "Use your own Meta key" below).
- **Replay mode** (recorded answers, so the AI parts work without a key).
Planned, not built.
- **More use of open weights.** Muse Spark Open is a second seat on the staple
judge and labels dictionary names today. Wider use would need the personal
data rule kept (names only).
- **More languages.** Ask answers in the language you write. Spanish and Hindi
were the ones verified. The app's buttons and labels are English only
(`assistant/docs/MULTILINGUAL-ASK.md`). Translating the app itself is a separate project.
- **A hosted open-weights option.** Today one open model (Muse Spark Open) is
hosted on Baseten for two small jobs. A fuller option for people who want
their reads on open weights is an idea only.
- **Models we did not use** (Spark Contributor for a community tier, Muse
Glimmer self-hosted): see [`docs/MODELS.md`](/docs/architecture/models).
- A non-English Ask currently sends the whole feature guide (about 19 KB) on
each question. Trimming that is planned.
## Trust and privacy
- A photo opt-in and a full privacy policy; store permission for mapping.
Source: `docs/quest/ARCHITECTURE.md`.
- Security review items that were deferred, tracked as GitHub issues.
- A community tier with user opt-in is an idea only
(`assistant/docs/MODEL-POLICY.md`).
- The forgot-password flow is held, by choice
(`assistant/docs/ARCHITECTURE.md`, section 13, "Not built / held").
- Arya's own code is MIT licensed (`LICENSE`). The Meta models and third-party
code keep their own terms (`NOTICE.md`).
## Running it yourself
- More for self-hosters: a one-command setup for a small server with TLS (the
laptop setup is in `docs/RUN-IT-YOURSELF.md`).
- Off-site backups with rclone (`ARYA_BACKUP_REMOTE` exists; no bucket is set
up) and a staging address to check a deploy before the live one.
- Cost proposals from the cost watcher, tracked as GitHub issues.
- **Scale out as the community grows.** Today Arya is one server process on
one VM (see [Known limits](https://github.com/vibhortayal/arya/blob/main/README.md#what-are-the-known-limits)). The plan, not a
promise: make Postgres the master copy, move demo and session state out of
memory, run several processes, and move to a cloud provider in step with
the number of shoppers. The design, phase by
phase, is in [`SCALABILITY-DESIGN.md`](/docs/architecture/scalability) (a plan,
not built).
## Developer platform and integrations
The starting point today is the Lists API and the developer portal at `/developers`
(with a quickstart and an interactive reference; `docs/API.md`), plus the MCP
connector that lets an AI agent read and edit your lists
(`assistant/docs/AGENT-CONNECTOR.md`). The API is for your own data, with a
personal token. Desk research on who else is in this space, with its unverified
claims flagged, is in [`MARKET.md`](https://github.com/vibhortayal/arya/blob/main/docs/MARKET.md). Everything below is future work
and not built:
- **A bigger developer portal,** so other platforms can connect to Arya,
with Arya's household data as the shared source. That means more endpoints (receipts,
spend, what is running low), webhooks, OAuth sign-in instead of a personal
token, SDKs and sample apps.
- **Smart fridges, cooking devices and home robots** that add what is empty
or expiring to the list and feed back what was used.
- **Online grocery delivery services and apps,** to send a list for ordering
and to bring back the real receipt and prices. (There is no cart or payment in
Arya today; see `assistant/docs/SPEC.md`, "Not in scope".)
- **Any app that understands a household's food and purchase habits** (meal
planners, nutrition and budgeting apps), connected through the same API and
the import and export below.
- Every integration would be opt-in, per household, with scoped permissions
and a clear way to revoke it.
## Spending, import and export
- **Richer spend tracking and analysis,** with filters (by store, category,
person, period, price per unit). The Spending screen exists today; deeper
analysis and filters are not built.
- **Export to other budgeting apps** (spend and receipts as CSV or a direct
connection). Today the API lists your own data (`docs/API.md`); a budgeting
export is not built.
- **Import lists from other list apps** (a file or a link). Not built.
## More ideas for growing the app and its community
All of these are ideas. None is designed or built. Each would need my go-ahead
and, where it touches personal data, a privacy review first.
**Everyday use**
- Meal planning from what is running low and what to use soon, with a
shopping list built from the plan.
- **Recipe to shopping list, then to stores.** Turn a recipe (link, photo or
typed) into list items, then use the household's receipt history to suggest
where and which brand to buy. Not built.
- Budget goals per store or per month on the Spending screen, with a gentle
heads-up and never a lock.
- Barcode scan to add an item or check a price against your own history.
- Pantry and fridge check-in by photo (outside the current spec).
- Home Screen widgets and watch glances for the list and for use-soon items.
- Calendar-aware reminders, such as "you host on Saturday".
- Voice assistants (Siri, Google Assistant) adding to the list (Alexa is built
but switched off, see below), and
more connectors beyond the Muse agent and MCP.
- Receipt import from email or a store account, so no photo is needed.
- Warranty and return reminders read from receipts (Return Regret already
shows returns are tracked).
- Waste tracking and a monthly "what we threw away" view.
- Gift and event lists with a shared link and no account needed to view.
- Accessibility: a screen-reader review and a calmer Quest mode. (Text size
and spacing already exist in Settings, under Display.)
**Community**
- Neighborhood and store pages with shared tips, labelled as opt-in.
- Verified store news from shoppers, expiring when nobody confirms it.
- Seasonal Quest events, themed skins (escape room) and household leaderboards
that stay private by default.
- A mapper reputation and thanks for people who map stores, with no pay
attached.
- Clubs and teams (families, roommates, offices, school pantries, food banks).
- A public gallery of store maps that are fresh and photo-backed.
- Shared price alerts when a household opts in to share a figure.
**AI and open weights**
- A hosted open-weights reader for receipts, for people who want that choice.
- A smaller on-device model for offline add and quick matches.
- Better product merge and split suggestions in the Curator, and learned
substitutions (the usual brand when one is out of stock).
- Ask Arya acting on more things, with a confirm step for anything that
changes data.
- A public set of evals so others can check the reader scores.
**Trust, privacy and operating it**
- One place for privacy controls. Export my data and Delete my account already
exist in Settings, under Account. The idea is to gather them with the other
privacy choices.
- A public status page and an uptime history.
- A staging address, and automated CI instead of manual runs.
- A published API with webhooks and OAuth (`docs/API.md`, "Later").
- Store permission programme for mapping, and a takedown route for stores.
## Alexa skill: built, switched off, not tested on a device
The code is merged (#515 and #516) behind the `ARYA_ALEXA` flag, which is off in
the live app and the demo. I paused it until after the hackathon. It adds items
by voice only and has not been tried on a real Echo. Amazon has retired its Alexa
list API, so Arya cannot sync with Alexa's own lists.
## Use your own Meta key: unfinished, switched off, not used
The code exists (PRs #391 and #393), it is unfinished, and it is switched off
in the live app and the demo. I decided on 2026-11-02 not to use it. Operator
notes: `assistant/docs/OPS.md`, "Own Meta keys (BYOK)".
---