# Administration overview (/docs/administration)
**For:** Administrators
Admins keep the workspace healthy: confirm new instruments, watch the fleet, issue tokens, manage who is an admin, roll out watcher releases, and connect Linear and Slack. Admins can also delete any run comment and review the feedback people send about Data Hub.
The admin-only pages under **Settings** are **Feedback**, **Integrations**, **Watchers**, and **Members**. Members don’t see them in the settings menu.
Roles and scopes are defined in [Security and permissions](/docs/security). Day-to-day operator setup is in [Set up an instrument](/docs/set-up-an-instrument).
## What you manage [#what-you-manage]
# Architecture (/docs/architecture)
**For:** Engineers and operators
Data Hub has four components that coordinate through S3, a REST API, and a shared
Python library. This page is the conceptual map; the deployment mechanics live in
the [first-time deployment guide](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/first-time-deployment.md)
in the data-hub developer docs.
## System overview [#system-overview]
## Components [#components]
| Component | Package | Description |
| -------------- | ------------------ | -------------------------------------------------------------------------------------------- |
| Web app | `data-hub-web` | Next.js web application, REST API, and MCP server. Deployed on Vercel. |
| Lambda | `data-hub-lambda` | AWS Lambda function triggered by S3 uploads. Runs instrument-specific processing. |
| Watcher | `data-hub-watcher` | CLI on lab instrument PCs. Detects new files, uploads them to S3, reports status to the API. |
| Shared library | `data-hub-shared` | Shared Python library: S3 utilities, instrument enums, test infrastructure. |
## Data flow [#data-flow]
### Automatic upload (auto mode) [#automatic-upload-auto-mode]
1. A lab instrument writes output files to a watched directory.
2. The **watcher** detects new stable files via filesystem events.
3. It groups files into runs and reports each run to the **API**.
4. It uploads raw files to **S3** at the key `{instrument_id}/{run_id}/{filename}`.
5. The S3 upload triggers the **Lambda**.
6. Lambda dispatches to the instrument’s processor, which reads the file for preprocessing. Very large files can be read from a read-only mount of the raw bucket instead of being downloaded, when the deployment turns that option on.
7. Lambda creates/updates the run and files via the **API**, which sends a **Slack** notification once per new run.
8. Users view the run in the **web dashboard**.
### Manual upload (manual mode) [#manual-upload-manual-mode]
Steps 1–3 are the same, but the watcher doesn’t upload immediately:
4. The server adds files to an upload queue.
5. Once a minute the watcher checks the queue and uploads the files someone requested.
6. Steps 5–8 from auto mode follow.
From watcher 1.2.1, auto-mode watchers check the same queue, so **Upload** on a run page works for them too.
## Key design decisions [#key-design-decisions]
* **S3 is the integration boundary.** The watcher and Lambda never talk directly; S3 is the durable hand-off: the watcher writes, Lambda reads.
* **API-driven coordination.** The watcher registers, syncs its YAML config, and heartbeats, which is what powers dashboard health and the upload queue.
* **Pre-signed URLs from the API.** The web app mints pre-signed S3 upload/download URLs so bytes move directly to/from S3 without routing through the API. On Vercel the app assumes an AWS Identity and Access Management (IAM) role via OpenID Connect (OIDC), with no long-lived AWS keys.
* **Lambda-built run archives.** “Download all” delegates to the Lambda, which zips files into a separate archives bucket via S3 multipart upload; the web app then 302s the browser to a short-lived presigned URL. Bytes never traverse Vercel.
* **Shared library for contracts.** Instrument IDs, S3 utilities, and environment config live in `data-hub-shared` so Lambda and the watcher stay consistent.
* **MCP for AI access.** The web app serves an MCP endpoint at `/mcp/v1` exposing tools, resources, and prompts to AI clients. Clients authenticate with OAuth against the same Better Auth instance that signs people in to the dashboard, so an agent acts as the person who approved it. See [MCP overview](/docs/mcp).
## Branch and environment strategy [#branch-and-environment-strategy]
| Branch | Purpose |
| ------------ | ----------------------------------------------- |
| `staging` | Pre-production. Pull requests merge here first. |
| `production` | Live. Changes are promoted from `staging`. |
Feature branches target `staging` via pull requests (PRs). Continuous integration
(CI) runs on every PR and on merges to both branches; merges to `staging` and
`production` deploy each component to its environment automatically. Staging and
production are independent deployments with separate databases.
# Browse and analyze runs (/docs/browse-runs)
**For:** Everyone
After instruments are uploading, the web app is where you find runs and act on them: inspect results, claim work, download files, and leave lab notes. The sections below follow the order you’ll use them in.
## Orient yourself in the app [#find-a-run]
The sidebar and the home page are the two starting points for daily work.
**Home** lists recent runs across instruments, most recent first, with counts for today and this week at the top. Filter by date range, who ran it, and instrument-specific metadata when it’s available, then page through results. Below the runs, Home shows up to four of today’s comments; use the date picker to see comments from a different range.
**Instruments** in the sidebar lists five instruments, and **View all** opens the full list. Each instrument’s page shows the folder its watcher reads from and the file patterns it uploads, then a runs table with column filters that match the instrument type (for example wavelengths on a plate reader).
**My runs** lists runs you’ve claimed, so you can return to your own work without re-filtering the full dashboard.
**Comments** lists every run comment by day. Open it from **View all comments** on Home. Each person’s profile page also shows the comments they wrote and the comments on runs they ran.
### Narrow a long run list [#narrow-a-long-run-list]
Home and the instrument pages share the same four controls:
* **Search runs**: filters the list by run name as you type
* **Date range**: presets from Today through Last 4 weeks, plus a custom range
* **Add filter**: filter by run status, or turn on **Show deleted runs**
* **Column headers**: sort by **Run Started**, **Last Updated**, size, who ran it, or any instrument-specific column
**Run Started** is when the instrument acquired the run. **Last Updated** changes when the run’s files change, or when someone edits, claims, or comments on it. The default order is newest run first. The sort order is saved in the page link, so a link you share opens with the same order.
### What each run status means [#run-status]
A run’s status tells you how far its files have got from the instrument to the dashboard:
| Status | Meaning |
| -------------- | ------------------------------------------------------------------------------------------------------------ |
| Failed | At least one file failed processing. The file’s row on the run page gives the reason. |
| Stalled | At least one file has been processing for more than 20 minutes without finishing. Reprocess it to try again. |
| Pending upload | The watcher found the files but hasn’t sent them yet. |
| Uploaded | The files are in storage. Processing hasn’t started, or this instrument type has no processor. |
| Processing | The server is reading the files, extracting metadata, and building reports. |
| Completed | Processing finished. |
| Empty | The run has no files. |
A run with files in several states shows the first status in this table that applies. For example, a run with one failed file and nine completed files shows **Failed**.
## Search the workspace [#search]
Press **⌘ K** (macOS) or **Ctrl K** (Windows/Linux) to open global search from any page. Results are grouped into runs, files, instruments, users, and comments. That means you can also look up a colleague’s work or the text of a lab note. A file hit opens the parent run with the files table already filtered to that file.
## Open a run and inspect results [#open-run-detail]
Each run has a details page at `/instruments/your_instrument_id/runs/your_run_id`. The arrows beside the run name step you to the next or previous run without going back to the list.
The top of the page lists who ran it, the number of raw files, and their total size. What you see below that depends on the instrument type and whether a processor ran:
* Metadata tags for extracted fields, such as experiment type, wavelength, or temperature
* Processed reports such as tables, images, PDFs, and videos, including interactive plate maps for plate readers and a video player for DishCam runs
* A files table of everything in the run, labeled Raw or Processed, with each file’s size and status
Click a metadata tag to open the instrument’s runs table filtered to that value. For example, clicking a wavelength tag lists every run on that instrument recorded at the same wavelength.
Reports with many images, PDFs, or spectra show one item at a time. Search by filename to jump anywhere in the run, or use the arrows to step through it. The counter beside the arrows shows your position out of the total. Plate maps with more than 12 columns, such as 384-well plates, scroll sideways inside the plate.
Per-type inputs and outputs are in [Instrument data preprocessing](/docs/instrument-preprocessing).
### Filter the files table [#filter-files]
The files table has separate **Type** and **Status** filters. Use them to show, for example, only failed files or only processed CSVs. **Download all** and the run archive include only the files the filters show.
### Live file updates [#live-file-updates]
While files are uploading or processing, the run page checks for updates every 5 seconds for up to 5 minutes. After that it stops and shows **Stopped checking for file updates**. Choose **Refresh** to check for another 5 minutes. Uploading or reprocessing files from the page also starts a new 5 minutes.
## Claim who ran it [#claim-who-ran-it]
Use **Claim** on the run details page to attribute yourself. You can claim or unclaim your own user only, so nobody else can attribute a run to you. Claims show on the run and appear in “ran by” filters on Home and elsewhere. API tokens can’t claim runs; a signed-in person has to.
## Download files [#download-files]
From the files table you can:
* Preview a CSV, TSV, JSON, or other text file by clicking its row. The preview shows the first 1 MB of the file. Use the back and forward buttons to move between previewable files on the same page of the table. **Download** in that window saves the file
* Select files and choose **Download** in the bar that appears above the files table. One file downloads directly. Two or more come as a single zip, which builds in the background like **Download all**
* Choose **Download all** to build a zip of active files in the run, with any active file filters applied
Large zips build in the background. When one isn’t ready yet, Data Hub tells you how long to wait before trying again.
## Act on several runs at once [#bulk-actions]
Select the checkbox on one or more run rows to open an action bar at the bottom of the list. Run rows have no buttons of their own; every run action goes through this bar.
* **I ran these**: claim every selected run in one step
* **Remove my attribution**: unclaim them
* **Upload**: ask the watcher to upload files that are still on the instrument PC
* **Download**: download a zip of each selected run
* **Reprocess**: run server-side processing again, including on runs whose files have stalled
* **Delete**: mark the selected runs deleted
**Upload**, **Download**, **Reprocess**, and **Delete** appear only when every selected run allows them. For example, **Reprocess** is hidden if one of the selected runs belongs to an instrument type with no processor. The checkbox in the header row selects every run on the page, and **Clear** drops the selection.
The selection stays when you move to another page of the same results. It clears when you change the search, a filter, or the sort, so the bar never acts on runs from a list you are no longer looking at.
The files table on a run page works the same way: select files to get **Upload**, **Dismiss**, **Reprocess**, and **Download** for just those files. Its selection also clears when you change the files search, a filter, or the sort.
## Comment on a run [#comment]
The comments thread on run details is for lab notes and follow-ups. Comment activity can notify people who claimed the run or already commented, and clicking a comment notification scrolls to that comment. Set preferences under **Settings → Notifications**; see [Configure notifications](/docs/configure-notifications).
Only the author of a comment can edit it. Authors can delete their own comments, and workspace admins can delete any comment. Comments posted by a script through an API token show the token’s name with a key icon instead of a person.
## Reprocess or delete a run [#reprocess-and-delete]
Reprocessing runs server-side processing again. Use it after a file failed or stalled, or after an engineer fixes the code that reads a file type. The stored result doesn’t change until you reprocess.
* On the run page, hover over a file and choose **Reprocess**.
* To reprocess several files, select them in the files table and choose **Reprocess** in the bar.
* To reprocess whole runs, select them in a run list and choose **Reprocess** in the bar.
Reprocessing sends only the original instrument files. Files Data Hub generated, such as videos, previews, and CSVs, have no **Reprocess** button and are rebuilt from the originals.
**Delete** marks the run deleted without removing stored files. To find a deleted run again, turn on **Show deleted runs** under **Add filter**, then open it and choose **Restore**. Anyone signed in can delete or restore a run.
### Request an upload [#request-upload]
A file that the watcher found but hasn’t uploaded shows **Pending**. Choose **Upload** on that file, or select runs and choose **Upload**, to ask the watcher to send it. The watcher picks up the request within a minute. On instruments that upload automatically, this needs watcher 1.2.1 or later; see [Roll out watcher releases](/docs/watcher-releases#what-each-version-adds).
For instruments in [manual upload mode](/docs/concepts#upload-mode), every file waits for this approval. **Dismiss** tells the watcher not to upload a file. See [Manage the watcher fleet](/docs/manage-watchers#approving-manual-uploads).
## Use the same data outside the UI [#programmatic-access]
You can also list and fetch the same runs and files over the [REST API](/docs/api) or the [Model Context Protocol (MCP) endpoint](/docs/mcp). REST needs a [personal access token](/docs/concepts#personal-access-token); scopes and creation are covered in [Issue and revoke tokens](/docs/manage-tokens). MCP clients sign in through the browser instead, and clients that support MCP Apps show the same interactive run report you see here.
# Watcher CLI (/docs/cli-reference)
**For:** Lab operators
Flags and subcommands for `data-hub-watcher`, generated from the Click CLI in the watcher package. For task walkthroughs see [Install and update the watcher](/docs/install-the-watcher), [Run as a Windows service](/docs/windows-service), and [Roll out watcher releases](/docs/watcher-releases).
## Command reference [#command-reference]
The catalog below lists every command, argument, and option from the installed CLI help text.
## Global flags
These options apply to every `data-hub-watcher` command (watcher 1.2.2).
**Flags**
- `--config PATH` `path` env: `DATA_HUB_CONFIG_PATH` — Override config file path.
- `--verbose` `boolean` — Enable debug logging.
- `--version` `boolean` — Show the version and exit.
## `data-hub-watcher config`
View and manage the watcher config file.
### `data-hub-watcher config edit`
Re-prompt each config field with current values as defaults.
### `data-hub-watcher config open`
Open the config file in your editor, then re-validate.
**Flags**
- `--editor` `text` — Editor command (e.g. --editor code).
### `data-hub-watcher config path`
Print the resolved config file path.
### `data-hub-watcher config set-environment`
Switch the watcher to a different API environment.
**Arguments**
- `ENVIRONMENT` (required) `choice (staging | production | preview)`
**Flags**
- `--api-base-url` `text` — API base URL for the target environment. Required the first time you switch to an environment; reused from config on later switches.
- `--api-key` `text` — API key for the target environment.
- `--show-key` `boolean` — Echo the API key as typed (useful on Windows terminals).
- `--no-register` `boolean` — Fail instead of registering a new watcher if none is stored for the target env.
### `data-hub-watcher config show`
Pretty-print the current config.
### `data-hub-watcher config validate`
Validate the config file (offline, no network calls).
## `data-hub-watcher init`
Interactive setup wizard + API registration.
**Flags**
- `--show-key` `boolean` — Echo the API key as it is typed/pasted. Useful on Windows terminals where hidden input is unreliable for paste.
## `data-hub-watcher self-update`
Check the server for a newer watcher release and upgrade in place. Designed for unattended use — schedule via Windows Task Scheduler (e.g. weekly) so lab PCs converge on the latest published version without operator intervention.
**Flags**
- `--check` `boolean` — Print the server-reported target version without performing the upgrade.
- `--force` `boolean` — Run the upgrade subprocess even if the local version already matches the target.
## `data-hub-watcher service`
Windows service management.
### `data-hub-watcher service install`
Install the watcher as a Windows service.
**Flags**
- `--env-path FILE` `file` — Path to the .env file. Defaults to ~/.data-hub/.env..
### `data-hub-watcher service reinstall`
Stop, uninstall, install, and start the watcher Windows service. Useful after upgrading the watcher wheel from an Administrator shell so the SCM picks up the new ``data_hub_watcher.service`` entrypoint without needing four separate ``service`` invocations.
**Flags**
- `--env-path FILE` `file` — Path to the .env file. Defaults to ~/.data-hub/.env..
### `data-hub-watcher service start`
Start the watcher Windows service.
### `data-hub-watcher service status`
Show the watcher Windows service status.
### `data-hub-watcher service stop`
Stop the watcher Windows service.
### `data-hub-watcher service uninstall`
Uninstall the watcher Windows service.
## `data-hub-watcher state`
Inspect or reset the local upload history.
### `data-hub-watcher state forget`
Forget upload history under a folder so those files are reported again. Clears every state database a watcher on this PC can open: the one `watch` uses and the one the Windows service uses. Stop the watcher first. It only scans the disk when it starts, so a running watcher will not pick the forgotten files back up.
**Flags**
- `--prefix` (required) `text` — Folder under the watch directory to forget, e.g. alice/.
- `--yes` `boolean` — Skip the confirmation prompt.
## `data-hub-watcher upload`
Upload files to Data Hub (manual trigger). One-shot mode: pass both --file and --run-id to upload a single file. Queue mode: omit both to process the server-side upload queue.
**Flags**
- `--file PATH` `path` — Specific file to upload.
- `--run-id` `text` — Associate upload with a specific run.
- `--dry-run` `boolean` — Log what would be uploaded without uploading.
## `data-hub-watcher watch`
Start watching for new files.
**Flags**
- `--dry-run` `boolean` — Validate and log but don't start monitoring.
## Interactive setup notes [#interactive-setup-notes]
`init` and `config edit` prompt for values in the terminal. Those prompts are not CLI flags, so they do not appear in the catalog.
Keys must start with `dhub_`. The watcher strips zero-width and non-breaking-space characters that some Windows clipboards inject. See [Install and update the watcher → Configure](/docs/install-the-watcher#configure).
## Upload examples [#upload-examples]
One-shot upload outside the watch loop:
```sh
data-hub-watcher upload --file /path/to/file.csv --run-id RUN001
data-hub-watcher upload
data-hub-watcher upload --dry-run
```
Pass both `--file` and `--run-id`, or omit both to process the server-side queue in manual mode.
## Switching environments [#switching-environments]
`config set-environment` (and `config edit` when the environment changes) can take `--api-base-url`, `--api-key`, `--show-key`, and `--no-register`. Preview switches require `--api-base-url` the first time. See [Watcher configuration → Switching environments](/docs/configuration-reference#switching-environments).
## Windows service notes [#windows-service-notes]
`service` commands require Windows and the `windows-service` install extra. Use `--env-path` on `service install` and `service reinstall` when the service should load a non-default `.env` file. Full details are in [Run as a Windows service](/docs/windows-service).
## Self-update examples [#self-update-examples]
```sh
data-hub-watcher self-update
data-hub-watcher self-update --check
data-hub-watcher self-update --force
```
Upgrade behavior depends on how you installed the watcher. See [Install and update the watcher → Manual update](/docs/install-the-watcher#manual-update-self-update).
# Concepts (/docs/concepts)
**For:** Everyone
Short definitions of the vocabulary the rest of these docs use. If you’ve arrived mid-page and a term is unfamiliar, it’s defined here. Terms are grouped by what they describe: the equipment, the data, the rules that turn files into runs, and who can do what.
## Equipment and fleet [#equipment-and-fleet]
### Instrument [#instrument]
A piece of lab equipment registered in Data Hub, identified by a kebab-case **instrument ID** (e.g., `akta-fplc`, `bio-rad-cfx96`). The ID is permanent, because it becomes the storage key prefix and the identifier used across the system. A human-readable display name sits alongside it.
An instrument has three states. The dashboard groups instruments into a tab per state and shows connection health separately:
| State | Shown as | Meaning |
| ---------- | --------------------------------------------- | ---------------------------------------------------------------------------- |
| `pending` | **Pending** tab | Registered but not yet confirmed by an admin. A watcher can’t upload for it. |
| `active` | **Active** tab, with Online or Offline health | Confirmed and accepting uploads. |
| `inactive` | **Retired** tab | Deactivated. No longer accepting uploads, but its runs stay browsable. |
See [Set up an instrument](/docs/set-up-an-instrument), [Manage instruments](/docs/manage-instruments), and [Retire an instrument](/docs/retire-an-instrument).
### Instrument type [#instrument-type]
Separate from the instrument ID. The type tells Data Hub which [preprocessor](#preprocessor) to run on files from this instrument, so it controls what metadata, badges, and reports you get.
Admins set it from the instrument page: open the three-dot menu, choose **Edit details**, and pick a supported type. An instrument with no matching type still stores and serves raw files. It gets no extracted metadata and no processed artifacts. See [Instrument data preprocessing](/docs/instrument-preprocessing).
### Watcher [#watcher]
The `data-hub-watcher` command-line program that runs on an instrument PC. It monitors a directory, groups new files into runs, uploads them to cloud storage, and reports status to the API. Each instrument can have at most one active watcher at a time.
A watcher is registered with the API and carries one registration ID per [environment](#environment). Deregistering a watcher is a [soft delete](#soft-delete): its history stays visible for auditing. See [Manage the watcher fleet](/docs/manage-watchers).
### Watcher status [#watcher-status]
What the **Watchers** page shows for each watcher, derived from its most recent [heartbeat](#heartbeat):
| Status | Meaning |
| ---------------- | ------------------------------------------------------------------------------------------------------------- |
| **Online** | A heartbeat arrived in the last 5 minutes. |
| **Unresponsive** | No heartbeat for more than 5 minutes. The process crashed, the PC is off, or the network is blocking the API. |
| **Stopped** | The watcher shut down cleanly and said so in its last heartbeat. |
| **Registered** | The watcher registered but hasn’t sent its first heartbeat yet. |
An unresponsive watcher keeps its registration and history, so you can read its last events to work out what happened. See [Troubleshoot a watcher](/docs/troubleshoot-a-watcher).
Instrument pages and the instruments table sum this up per instrument: **Online** if any of its watchers is online, **Offline** if none is, and **No Watcher** if none has registered.
### Heartbeat [#heartbeat]
A status ping the watcher sends to the API every 60 seconds. It carries the watcher version, instrument ID, watch directory, upload mode, activity counters, and uptime, and it drives the **Last Heartbeat** column and the online health shown on the dashboard. The same tick runs the auto-updater. The upload queue is checked separately, also once a minute, so a slow upload can’t delay a heartbeat.
### Events [#events]
A per-watcher log of lifecycle moments and errors: `watcher_started`, `run_reported`, `file_uploaded`, `update_succeeded`, and `error` (with a `details.kind` discriminator), among others. Events are batched and flushed on each heartbeat and are visible on the dashboard, which makes them the primary tool for diagnosing a watcher remotely. See [Troubleshoot a watcher](/docs/troubleshoot-a-watcher).
### Environment [#environment]
Which Data Hub deployment a watcher talks to: `staging`, `production`, or `preview` (a Vercel preview deployment, which needs an explicit API base URL). Staging and production are separate deployments with separate databases, so a single PC keeps an independent registration and local state per environment and can switch between them.
## Data [#data]
### Run [#run]
A logical grouping of files produced by one acquisition. The watcher assigns each file a **run ID** using its configured [run detection](#run-detection) method, then reports the run to the API. The first file for a run creates it; later files are added incrementally to the same run’s manifest.
### Run status [#run-status]
How far a run’s files have got from the instrument to the dashboard. These are the values you can filter by under **Add filter** on any run list:
| Status | Meaning |
| -------------- | ------------------------------------------------------------------------------------------------------------ |
| Failed | At least one file stopped on an error. [Reprocess](/docs/browse-runs#reprocess-and-delete) after a fix. |
| Stalled | At least one file has been processing for more than 20 minutes without finishing. Reprocess it to try again. |
| Pending upload | The watcher found the files but hasn’t sent them yet. |
| Uploaded | The files are in storage. Processing hasn’t started, or this instrument type has no preprocessor. |
| Processing | The preprocessor is reading the files, extracting metadata, and building reports. |
| Completed | Processing finished. |
| Empty | The run has no files. |
When a run’s files are in different states, the run shows the first status in this table that applies.
### File and file category [#file-and-file-category]
Every file in a run carries a category, shown as a label in the run’s files table:
* **Raw**: what the instrument wrote, uploaded unchanged
* **Processed**: what a [preprocessor](#preprocessor) generated from a raw file, such as a CSV of well values, a contrast-adjusted PNG, or a preview JPG
Both categories download the same way, and file filters on the run page work on this label. See [Download files](/docs/browse-runs#download-files).
### Preprocessor [#preprocessor]
The server-side code that reads a raw file, extracts metadata, and writes processed files. It runs in an AWS Lambda when a file arrives in raw storage, or when you choose **Reprocess**. Which preprocessor runs depends on the [instrument type](#instrument-type), and not every type has one. Per-type inputs and outputs are in [Instrument data preprocessing](/docs/instrument-preprocessing).
### Run archive [#run-archive]
The zip that **Download all** builds for a run. It covers the run’s active files and respects any file filter you have applied. Large archives build in the background as an archive job, so the first request can return a wait time rather than a file.
### Attribution [#attribution]
The record of who ran a run. Use **Claim** on the run to attribute yourself. Attribution is self-only: you claim or unclaim your own user, and you can’t attribute someone else. API tokens can’t claim runs. Claims drive **My runs** and the “ran by” filters, and claiming a run subscribes you to its comments. See [Claim who ran it](/docs/browse-runs#claim-who-ran-it).
## Turning files into runs [#turning-files-into-runs]
### Run detection [#run-detection]
The rule that maps a file to a run ID. Two broad strategies:
* **Prefix**: a regex extracts the run ID from the filename. The default `^([^_]+)` takes everything before the first underscore, so `RUN001_data.csv` → `RUN001`
* **Directory**: each subdirectory under the watch directory is its own run, so `RUN001/data.csv` → `RUN001`
The full set of presets, and the custom-regex option, is in the [Configuration reference](/docs/configuration-reference#run-detection).
### Stability period [#stability-period]
How many seconds a file must stay unchanged (same size and modification time) before the watcher treats it as fully written and eligible for upload. The default is 5 seconds; raise it for instruments that write large files slowly. A file that keeps changing past `max_stability_wait_seconds` (default 300, or 5 minutes) is abandoned with a `stability_timeout` error event.
### Upload mode [#upload-mode]
How files get from the instrument to cloud storage:
* **auto**: files upload immediately after they’re detected and stabilized. With watcher 1.2.1 and later, the watcher also uploads files someone requests from the run page
* **manual**: runs are reported to the server, but files aren’t uploaded until approved through the server-side upload queue, which the watcher checks once a minute. Use this when uploads need human approval
### Initial scan [#initial-scan]
What the watcher does with files already sitting in the watch directory the first time an environment is entered. `production` uploads the existing backlog (`full`). `staging` and `preview` record it as a baseline and skip it (`new-only`), so test environments aren’t flooded with history. See the [Configuration reference](/docs/configuration-reference#initial-scan).
### Dismiss [#dismiss]
For instruments in [manual upload mode](#upload-mode), dismissing a detected file tells the watcher not to upload it. Dismissing differs from deleting: the file never reached Data Hub storage, so there’s nothing to restore. See [Manage the watcher fleet](/docs/manage-watchers#approving-manual-uploads).
### Soft delete [#soft-delete]
Data Hub marks things deleted rather than erasing them, so history survives for auditing. Deleting a run hides it from run lists without removing its stored files, and **Show deleted runs** brings it back into view. Deregistering a watcher keeps its events and heartbeats. Anyone signed in can restore a deleted run from the run page.
## People and access [#people-and-access]
### Role [#role]
Data Hub has two roles, **Member** and **Admin**. Members sign in, browse instruments and runs, claim and comment on runs, send feedback, and read the token list. Admins can also confirm and update instruments, create and revoke tokens, promote or demote other users, delete any comment, review feedback, and manage integrations. Full permission tables are in [Security and permissions](/docs/security#roles).
### Personal access token [#personal-access-token]
A `dhub_`-prefixed bearer token that authenticates the watcher, and other API clients, with the Data Hub API. Tokens carry permission [scopes](#scope) and are created and revoked by admins. The plaintext is shown once, at creation.
A token acts as itself, under its own name. Comments it posts and runs it deletes show the token’s name with a key icon, not a person. Because a token isn’t a person, it can’t claim runs or send feedback. See [Issue and revoke tokens](/docs/manage-tokens).
### Scope [#scope]
A permission string carried by a token, such as `runs:read`. A token-authenticated request is refused with `403 FORBIDDEN` when the token’s scopes don’t cover what the route requires. Dashboard sessions hold every scope, so scope checks only apply to token requests. The scope table is in [Security and permissions](/docs/security#token-scopes).
### Feedback report [#feedback-report]
A bug report or feature request about Data Hub itself, sent from the account menu or by an AI assistant. Each report is stored as an issue in your workspace’s Linear, and its status (Open, Resolved, or Declined) follows that issue. See [Send feedback and see what’s new](/docs/send-feedback).
# Watcher configuration (/docs/configuration-reference)
**For:** Lab operators
The complete reference for the watcher’s configuration: the `config.yaml` schema,
where files live, the environment variables that influence it, and the behaviors
(run detection, upload modes, initial scan) those fields drive.
## File locations [#file-locations]
Everything the watcher persists lives under `~/.data-hub/`:
| Path | Contents |
| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `~/.data-hub/config.yaml` | The main config (below). Override with `--config` or `DATA_HUB_CONFIG_PATH`. |
| `~/.data-hub/.env.` | The API key for an environment (e.g. `.env.staging`, `.env.production`, `.env.preview`). The legacy `~/.data-hub/.env` is still loaded, with the per-environment file taking precedence. |
| `~/.data-hub/watcher-.db` | Per-environment SQLite state (uploaded files, runs, detected files, baseline). |
| `~/.data-hub/watcher.log` | Rotating log (10 MB × 5 backups). On Windows: `C:\ProgramData\DataHubWatcher\watcher.log`. |
| `~/.data-hub/upgrade-worker.log` | Windows uv-tool upgrade transcript. |
## Environment variables [#environment-variables]
| Variable | Purpose |
| ---------------------------- | --------------------------------------------------------------------------------------------- |
| `DATA_HUB_API_KEY` | Supplies the API key, skipping the `init` prompt. Saved into the per-environment `.env` file. |
| `DATA_HUB_CONFIG_PATH` | Override the config file path (same as `--config`). |
| `DATA_HUB_WATCHER_LOG_LEVEL` | Set to `DEBUG` (in the service’s `.env` file) to enable debug logging without a reinstall. |
## `config.yaml` [#configyaml]
```yaml
version: 1
environment: production # "staging", "production", or "preview"
api_base_url: null # required when environment is "preview"
watcher_ids: # one registration id per environment
production:
initial_scan: null # null (default), "full", or "new-only"
instrument:
id: akta-fplc # kebab-case instrument ID
watch_directory: /path/to/data
file_patterns:
- "*.csv"
- "*.xlsx"
enabled: true
upload_mode: auto # "auto" or "manual"
stability_period_seconds: 5 # 1–300
max_stability_wait_seconds: 300 # 1–86400; must be >= stability_period_seconds
run_detection:
pattern: '^([^/]+)/' # regex with one capture group (run ID)
recursive: true
```
### Field reference [#field-reference]
| Field | Type | Description |
| --------------------------------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `version` | int | Config schema version. Currently `1`. |
| `environment` | enum | `staging`, `production`, or `preview`. |
| `api_base_url` | string \| null | Required only for `preview` (the deployment’s `/api/v1` URL). |
| `watcher_ids` | map | One watcher registration id per environment, assigned by the API. |
| `initial_scan` | enum \| null | `null` (env default), `full`, or `new-only`. See [Initial scan](#initial-scan). |
| `instrument.id` | string | Kebab-case instrument ID. Permanent; the storage key prefix. |
| `instrument.watch_directory` | string | Absolute path the instrument writes to. |
| `instrument.file_patterns` | string\[] | Glob patterns; only matching files are uploaded. |
| `instrument.enabled` | bool | Whether the watcher processes this instrument. |
| `instrument.upload_mode` | enum | `auto` or `manual`. See [Upload modes](#upload-modes). |
| `instrument.stability_period_seconds` | int (1–300) | Seconds a file must stay unchanged before it’s considered written. Default `5`. |
| `instrument.max_stability_wait_seconds` | int (1–86400) | Seconds a still-changing file may stay pending before the watcher abandons it with a `stability_timeout` event. Must be ≥ `stability_period_seconds`. Default `300` (5 minutes). |
| `instrument.run_detection.pattern` | regex | One-capture-group regex applied to each file’s path relative to `watch_directory`. |
| `instrument.run_detection.recursive` | bool | `true` watches subdirectories; `false` only the top level. |
A config written by an older watcher used a single top-level `watcher_id`. It’s
migrated transparently on load: lifted into `watcher_ids` under the active
environment and dropped on the next save.
## Run detection [#run-detection]
The `pattern` regex is applied with `re.search` to each file’s path relative to
`watch_directory`, with backslashes normalized to `/`. Capture group 1 is the run
ID, and the pattern must have exactly one capture group. In YAML, single-quote
patterns so backslash sequences like `\d` don’t need escaping.
The `init` wizard offers these presets (or supply a custom regex):
| Preset | Pattern | Recursive | Run ID is… |
| ---------------------- | ------------------------------ | --------- | ----------------------------------------------------------------- |
| Filename prefix | `^([^_]+)` | no | everything before the first `_`: `RUN001_data.csv` → `RUN001` |
| Top subdirectory | `^([^/]+)/` | yes | the top-level folder: `RUN001/data.csv` → `RUN001` |
| Deepest subdirectory | `([^/]+)/[^/]+$` | yes | the immediate parent folder: `plate-a/well-b/data.csv` → `well-b` |
| Timestamp subdirectory | `(?:^\|/)(\d{8}_\d{6}_\d{3})/` | yes | a `YYYYMMDD_HHMMSS_fff` folder anywhere in the path |
| Filename stem | `^(?:.+/)?([^/]+?)\.[^/.]+$` | no | the filename without its extension (each file is its own run) |
## Upload modes [#upload-modes]
* **`auto`**: files upload to cloud storage immediately after run detection. From watcher 1.2.1, the watcher also uploads files requested from the run page, and retries an upload that didn’t finish on its next start (up to three starts).
* **`manual`**: runs are reported without uploading; the server decides which files to upload via a queue the watcher checks once a minute. Useful when uploads need human approval. See [Manage the watcher fleet → Approving manual uploads](/docs/manage-watchers#approving-manual-uploads).
Files of 100 MB or more upload in 64 MB parts from watcher 1.2.0, which is also the first version that can upload files larger than 5 GB.
## Initial scan [#initial-scan]
The first time an environment is entered (via `init` or `config set-environment`),
`initial_scan` decides what happens to files already in the watch directory. It
defaults by environment:
* `production` → `full`: the existing backlog is uploaded (production is the source of truth).
* `staging` / `preview` → `new-only`: the on-disk files are recorded as a baseline and skipped; only files created afterwards are uploaded.
This is why a fresh `init` on `staging`/`preview` doesn’t upload a PC’s history. Set
`initial_scan: full` to opt back in (e.g. to deliberately populate staging), or
`initial_scan: new-only` on production to suppress its backlog. Upgrading an
existing watcher is unaffected: the environment’s local DB already carries history,
so baseline seeding is skipped.
## Switching environments [#switching-environments]
A single PC can move between `staging`, `production`, and `preview`. Because these
are separate deployments with separate databases, each keeps its own registration
in `watcher_ids` and its own credentials in `~/.data-hub/.env.`.
```sh
# Switch to staging (reuses a stored registration, or registers one).
data-hub-watcher config set-environment staging
# Point at a preview deployment (the base URL is required).
data-hub-watcher config set-environment preview \
--api-base-url https://data-hub-git-my-branch.vercel.app/api/v1
```
A switch validates the target API, reuses (or registers) the watcher id for that
environment, pushes the config, and leaves a breadcrumb event in the old
environment. A running `watch` process or service keeps using the old
environment until restarted: on Windows, `service stop && service start`;
elsewhere, restart `watch`.
# Configure notifications (/docs/configure-notifications)
**For:** Everyone
Data Hub can notify you in the app and in Slack when new runs arrive, when someone comments on a run you care about, when a script sends you a message, and when feedback you sent is closed. Open **Settings → Notifications** while signed in. Everything on that page is a personal choice that affects only you.
Connections that the whole workspace shares, such as the Slack channel for new runs and the Slack app behind direct messages, are under **Settings → Integrations** and only admins can change them. See [Set up integrations](/docs/integrations).
In-app and Slack preferences are independent: you can turn on either channel for a type without turning on the other.
## In-app preferences [#in-app-preferences]
| Setting | Effect |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| **Mute all instrument notifications** | Suppresses new-run notifications for every instrument. Per-instrument choices are preserved and resume when you unmute. |
| **Comments on runs you ran** | Notify when someone comments on a run you are attributed to. |
| **Replies to your comments** | Notify when someone comments on a run you previously commented on. |
| **Service notifications** | Show messages that integrations and scripts send you through the API. |
| **Updates on your feedback** | Notify when a [feedback report](/docs/send-feedback) you sent is resolved or declined. |
| **New feedback** | Admins only. Notify when someone sends feedback about Data Hub. |
### Per-instrument subscriptions [#per-instrument-subscriptions]
Under **Per-instrument**, toggle the instruments you want new-run notifications for. Rows are disabled while **Mute all** is on. You can also subscribe from the notifications switch on an instrument’s page, or from the instruments table’s notifications column when that column is shown.
## The notification bell [#notification-bell]
The bell icon in the app lists recent notifications:
* New runs are grouped by day and instrument, and each group expands to show its runs.
* Clicking a comment notification opens the run and scrolls to that comment.
* Messages sent by a script through an API token show the token’s name with a key icon.
## Slack direct messages [#slack-direct-messages]
Connect your Slack account on the same settings page to receive matching events as personal direct messages. After connecting, choose which types also go to Slack:
* New instrument runs (respects your instrument subscriptions and mute)
* Comments on runs you ran
* Replies to your comments
* Service notifications
* Updates on your feedback
* New feedback (admins only)
Disconnect or reconnect from the same card. You have to connect with your organization’s Slack workspace; Data Hub rejects any other workspace.
If **Connect to Slack** is missing or doesn’t finish, an admin hasn’t set up the Slack app yet. See [Slack app for direct messages](/docs/integrations#slack-app).
## Workspace Slack channel [#workspace-slack-channel-admins]
The shared Slack channel that gets a message whenever a new run is created is set by admins under **Settings → Integrations**. See [Slack channel for new runs](/docs/integrations#slack-channel).
## Set up the Slack app for personal DMs [#set-up-the-slack-app-for-personal-dms]
The Slack app setup, including scopes, redirect URLs, and where to enter its credentials, is in [Slack app for direct messages](/docs/integrations#slack-app).
# Configure run detection and uploads (/docs/configure-run-detection)
**For:** Lab operators
Tune three behaviors that decide what becomes a run and when bytes leave the instrument PC: run detection, stability period, and upload mode. Field-by-field schema lives in [Watcher configuration](/docs/configuration-reference); this page is the intent guide.
Change these during `data-hub-watcher init`, with `data-hub-watcher config edit`, or by editing `~/.data-hub/config.yaml` then syncing.
## Group files into runs [#group-files-into-runs]
A **run** is one acquisition: a logical group of files the watcher reports together. **Run detection** maps each file’s path (relative to the watch directory) to a run ID.
Common choices:
* **Prefix from filename**: a regex capture, for example everything before the first underscore so `RUN001_data.csv` → `RUN001`
* **One subdirectory per run**: each folder under the watch directory is its own run, so `RUN001/data.csv` → `RUN001`
Presets and custom regex options are listed under [Run detection](/docs/configuration-reference#run-detection). Set `recursive: true` when files live in subfolders; otherwise only the top level is watched.
When a run gathers files from several folders, two files can share a name, such as a `run.json` in each capture folder. Watcher 1.1.0 and later keep every copy and add a short tag to the later names, such as `sample~3f9a1c2b.tif`. Older watchers kept only the first; see [Files with the same name were skipped](/docs/troubleshoot-a-watcher#files-with-the-same-name-were-skipped).
If files appear on disk but never become runs, check for `pattern_mismatch` events on the watcher detail page and run `data-hub-watcher watch --dry-run`.
## Wait until writes finish [#wait-until-writes-finish]
**Stability period** is how many seconds a file must keep the same size and modification time before the watcher treats it as fully written. Default is 5 seconds (allowed range 1–300). Raise it for instruments that write large files slowly.
A file that keeps changing past `max_stability_wait_seconds` (default 300, or 5 minutes) is abandoned with a `stability_timeout` error event. Raise that cap for instruments whose writes take longer; it must be at least the stability period.
## Choose auto or manual upload [#choose-auto-or-manual-upload]
| Mode | Behavior |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **auto** | Upload as soon as a file is detected and stable. With watcher 1.2.1 and later, the watcher also picks up files someone requests with **Upload** on the run page, and retries uploads that a restart cut off. |
| **manual** | Report the run, but wait for server approval before uploading bytes. The watcher checks the upload queue once a minute. |
Use manual mode when someone must approve which files leave the instrument PC. Admins manage the queue from the dashboard; see [Manage the watcher fleet](/docs/manage-watchers#approving-manual-uploads).
## Existing files on first setup [#existing-files-on-first-setup]
`initial_scan` controls the backlog already in the watch directory when an environment is first entered:
* **`production` default**: upload the backlog (`full`)
* **`staging` / `preview` default**: record a baseline and skip existing files (`new-only`)
Override in config when you need the other behavior. Details: [Initial scan](/docs/configuration-reference#initial-scan).
# Install and update the watcher (/docs/install-the-watcher)
**For:** Lab operators
Install the watcher on an instrument PC, verify it, and apply manual updates. Fleet auto-update (what admins advertise and how PCs upgrade in the background) is covered in [Roll out watcher releases](/docs/watcher-releases).
## Prerequisites [#prerequisites]
* **[uv](https://docs.astral.sh/uv/getting-started/installation/)**: recommended. It installs Python for you; you do not need a separate Python install.
* **A personal access token** starting with `dhub_`. Admins create these at **Settings → Access Tokens**; members cannot mint their own. See [Issue and revoke tokens](/docs/manage-tokens).
* **Watch directory** and **file patterns** for the instrument output folder.
## Install [#install]
Most lab instrument PCs run Windows, so install the `windows-service` extra by default. It adds `pywin32`, which the watcher needs for the [Run as a Windows service](/docs/windows-service) setup:
```sh
uv tool install "data-hub-watcher[windows-service]"
```
This puts the `data-hub-watcher` CLI on PATH in an isolated venv managed by `uv`.
On macOS or Linux, or on a PC you will only run in the foreground, omit the extra:
```sh
uv tool install data-hub-watcher
```
## Configure [#configure]
```sh
data-hub-watcher init
```
The wizard walks through environment, API key, instrument, watch directory, file patterns, run detection, stability period, and upload mode. New instruments start as `pending` until an admin confirms them; see [Set up an instrument](/docs/set-up-an-instrument).
Config saves to `~/.data-hub/config.yaml` and syncs to the server. Intent-level guidance for detection and uploads is in [Configure run detection and uploads](/docs/configure-run-detection); every field is in [Watcher configuration](/docs/configuration-reference).
## Verify [#verify]
```sh
data-hub-watcher watch --dry-run
```
Dry-run validates config, checks API reachability and instrument status, and previews matching files without starting the monitor.
## Run [#run]
On a lab PC, run the watcher as a Windows service so it starts on boot, survives logout, and auto-updates in the background. From an **Administrator** terminal:
```sh
data-hub-watcher service install
data-hub-watcher service start
```
Lifecycle commands, log locations, and recovery settings are in [Run as a Windows service](/docs/windows-service).
For a one-off check, or on macOS or Linux, run in the foreground instead:
```sh
data-hub-watcher watch
```
The watcher monitors the directory, waits for files to stabilize, groups them into runs, uploads (or queues in manual mode), and heartbeats every 60 seconds. Stop a foreground watcher with `Ctrl+C`; stop the service with `data-hub-watcher service stop`.
## Reinstall [#reinstall]
Reinstall the same package (for example after a broken venv or missing service extra):
```sh
uv tool install --reinstall data-hub-watcher
# or with the Windows service extra:
uv tool install --reinstall "data-hub-watcher[windows-service]"
```
On Windows services, follow with `data-hub-watcher service reinstall` from an Administrator shell so the service and upgrade Scheduled Task point at the new install.
## Change configuration later [#change-configuration-later]
```sh
data-hub-watcher config edit # re-prompt each field
data-hub-watcher config open # open the YAML in your editor
data-hub-watcher config show # print the current config
```
Changes sync to the server automatically.
## Manual update (`self-update`) [#manual-update-self-update]
Use this when you want an upgrade immediately, or on a non-service install. Background fleet upgrades are described in [Roll out watcher releases](/docs/watcher-releases).
```sh
data-hub-watcher self-update # check + upgrade if needed
data-hub-watcher self-update --check # report status only
data-hub-watcher self-update --force # re-run even if versions match
```
The command asks the API for the latest published version and upgrades according to how the watcher was installed:
| Install method | Upgrade flow |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Windows + `uv tool install` | Routes through the `DataHubWatcherUpgrade` Scheduled Task. The CLI returns when the task accepts; the service restarts automatically. Tail `~/.data-hub/upgrade-worker.log` for progress. |
| POSIX + `uv tool install` | `uv tool install --reinstall data-hub-watcher==` inline. **Restart the watcher** afterward. |
| Plain venv `pip install` | `pip install -U data-hub-watcher==` inline. **Restart the watcher.** |
| Editable / `uv sync` checkout | Refused; upgrade with `git pull && uv sync`. |
On Windows uv-tool installs, `data-hub-watcher service install` (or `service reinstall`) must have run once from an Administrator shell so the upgrade Scheduled Task exists.
### Pin a specific version [#pin-a-specific-version]
```sh
uv tool install data-hub-watcher==1.2.1
```
Some features need a minimum watcher version, such as uploading files larger than 5 GB (1.2.0) or **Request upload** on instruments that upload automatically (1.2.1). Check [What each version adds](/docs/watcher-releases#what-each-version-adds) before pinning an older version.
Run `self-update --check` to confirm the server’s advertised target. When the server’s latest moves past your pin, the next auto-update tick tries to upgrade again. Fleet-wide pins and rollbacks are admin actions under [Roll out watcher releases](/docs/watcher-releases).
# Instrument data preprocessing (/docs/instrument-preprocessing)
**For:** Lab operators, admins, and engineers
After a [watcher](/docs/concepts#watcher) uploads a raw file, Data Hub can run a **preprocessor** on it: code that reads the vendor’s file, pulls out settings like wavelength or imaging mode, and builds something you can look at in the browser. Nine instrument types have one today. This page covers what each produces, why a file sometimes gets no processing, and what adding a new instrument involves.
Support for a new instrument isn’t a settings change. It takes a new Python module, a database migration, an edit in two repositories, and a Lambda redeploy. See [Add support for a new instrument](#add-support-for-a-new-instrument).
## How a file reaches a processor [#how-preprocessing-works]
1. A file arrives in raw storage, which triggers the Lambda. The bucket notifies on every upload, with no per-instrument filtering.
2. The Lambda checks the filename against a **filename gate**, a cheap test that skips files no processor wants before making any API call.
3. It reads the instrument, then looks up a processor by the instrument’s [type](#set-the-instrument-type). Unmapped types stop here.
4. The processor reads the file, extracts metadata, writes processed files when it has any, and updates the run through the API.
5. The file’s status moves from `processing` to `completed` or `failed`.
### Why an upload sometimes produces nothing [#why-an-upload-produces-nothing]
Two silent no-ops explain a file that uploaded and then produced nothing. Neither marks the file failed, because nothing went wrong:
* **The filename missed the gate.** Each type accepts a narrow pattern, listed in the table below. A TapeStation run only triggers on its PDFs, so the peak-table CSVs it writes are stored but don’t start processing on their own.
* **The instrument has no type set, or a type with no processor.** Files upload and stay browsable; you get no metadata and no reports.
Filename gates apply to uploads only. Choosing **Reprocess** in the web app skips the gate, because clicking it states your intent. That’s why a file that produced nothing on upload can still process when you reprocess it by hand.
### Set the instrument type [#set-the-instrument-type]
Processing is chosen by the instrument’s **type**, not by its instrument ID. The ID (`azure-cielo-qpcr`) names one physical machine. The type (`qpcr`) names a vendor’s file format, and several instruments can share one.
To set it, open the instrument page as an admin, choose **Edit details** from the three-dot menu, and pick an **Instrument type**.
One type means one vendor’s output format. Names like `qpcr` and `fplc` read as generic, but each is bound to a single vendor’s parser: `fplc` expects ÄKTA PDFs, `qpcr` expects Azure Cielo exports. Typing a different vendor’s instrument into an existing type feeds files to a parser that can’t read them. Supporting a second vendor means splitting the type, which is a code change.
## Supported instrument types [#supported-instruments-at-a-glance]
| Type | Vendor and model | Gate for uploads | You get |
| -------------------- | --------------------------- | --------------------------- | --------------------------------------------------------------- |
| `tape_station` | Agilent 4150 TapeStation | `.pdf` | Tape-type badge, PDF reports |
| `fplc` | Cytiva ÄKTA FPLC | `.pdf` | PDF you can read on the run page, no metadata |
| `gel_doc` | Azure 600 Gel Doc | `.tif`, `.tiff` | Contrast-enhanced PNG, imaging badges |
| `qpcr` | Azure Cielo qPCR | `.csv`, `.pdf`, `.aze` | Dye-channel badges, melting curves, the instrument’s PDF report |
| `epson_v700_scanner` | Epson V700 Scanner | `.tif`, `.tiff` | Plate preview JPG, colony measurements |
| `hina_microscope` | Hina Microscope (Nikon ND2) | `.nd2` | Channel overlay JPG, channel badges |
| `plate_reader` | SpectraMax iD3 and iD5 | `.xls` | Well-data CSV, interactive plate maps |
| `dishcam` | DishCam time-lapse camera | `.tif`, `.tiff`, `run.json` | A video of each photo stack, camera-setting badges |
| `aunty` | Unchained Labs Aunty | `.xlsx` | A plate with a curve chart in each well, experiment badges |
## Agilent 4150 TapeStation [#agilent-4150-tapestation]
Reads the tape type out of peak-table CSV filenames and shows TapeStation reports on the run page.
* **Expected files**: CSVs named like `YYYY-MM-DD - HH-MM-SS-_peakTable.csv`, plus matching PDFs
* **Metadata**: `Tape Type`, for example `gDNA` or `HSD1000`, when the filename matches that pattern
* **Processed files**: none
* **On the run page**: tape-type badge, a PDF viewer, and peak-table reports
Uploads trigger on the PDF, not the CSV. A run of CSVs alone stays unprocessed until you reprocess it by hand.
## Cytiva ÄKTA FPLC [#akta-fplc]
Stores ÄKTA chromatography PDFs so you can read them in the browser. The processor reads nothing out of the PDF itself yet, so this type gets no badges or filters.
* **Expected files**: `.pdf` reports from the ÄKTA
* **Metadata**: none
* **Processed files**: none
* **On the run page**: files table, plus each PDF embedded under Report Data
A newly seeded FPLC stays `generic` on purpose until an operator has checked its output format.
## Azure 600 Gel Doc [#azure-600-gel-doc]
Rebuilds each gel TIFF as a viewable image and reads the imaging settings the instrument recorded.
The processor stretches intensities to the 1st to 99th percentile, per RGB channel or as grayscale, then lays every page of the TIFF side by side in one 300 dpi PNG. The stretch is cosmetic: it makes faint bands visible on screen and doesn’t change the raw file.
* **Expected files**: TIFF from the Azure 600
* **Metadata**: `capture_type`, `imaging_mode` (true color, fluorescence, or chemiluminescence), `wavelengths`, `colors`, read from the TIFF `XPComment` tag
* **Processed files**: contrast-enhanced PNG
* **On the run page**: imaging badges, files table, image viewer for the PNGs
Those metadata fields also drive the column filters on the instrument page.
## Azure Cielo qPCR [#azure-cielo-qpcr]
Reads dye channel names, melting curves, and the instrument’s PDF report.
* **Expected files**:
* the Cq Values CSV, for dye channels
* the melting curve CSV (a name ending in `_MeltingCurve.csv`)
* the PDF report the instrument writes
* `.AZE` project files
* **Metadata**: `dye_channels`, the unique names in the Cq Values CSV’s `Fluorescence` column
* **Processed files**: `{run_id}_melting_curve_derivatives.csv` and `{run_id}_melting_curve_plate.json` from the melting curve CSV. An `.AZE` file produces `{run_id}_aze_melting_curve_derivatives.csv`, `{run_id}_aze_melting_curve_plate.json`, and `{run_id}_aze_experiment.json`.
* **On the run page**: dye-channel badges, the PDF report under **Report Data**, and a **Melting Curves** section
The **Melting Curves** section shows the plate for one dye channel at a time. Click a well to see its melting curve and derivative, with a dashed line at the melting temperature. A run with more than one PDF gets previous and next buttons under **Report Data**.
The other CSV files a qPCR run writes upload normally and show as completed, but contribute no metadata.
### `.AZE` project files [#aze-project-files]
Data Hub reads melting curves from the Azure Cielo’s native `.AZE` project files as well as from the CSV export. Melting temperatures match the CSV export, but other values can differ. That’s because `.AZE` files store raw fluorescence, while the CSV export can be adjusted for baseline. A project with no melt data completes with no processed files.
The watcher uploads `.AZE` files only when `*.AZE` is in the instrument’s file patterns. File patterns live in the watcher’s config on the instrument PC, so add it there with `data-hub-watcher config edit` and restart the watcher. Patterns are case-sensitive on some systems, so match the case the instrument uses. See [Watcher configuration](/docs/configuration-reference#field-reference).
## Epson V700 Scanner [#epson-v700-scanner]
Finds agar plates in a flatbed scan and measures the colonies on them.
The processor looks for plates sitting in gold 3D-printed frames. For each plate it finds, it crops the plate, then detects colonies: crop the edge, compare against the plate background, apply a difference-of-Gaussians filter, threshold, and measure whatever survives. Measurements come out in physical units, so colony areas are comparable between scans.
* **Expected files**: TIFF plate scans
* **Metadata**: `dpi`, `color_mode`, `plate_count`, `plate_boxes`, plus `colony_detection` summaries when colonies were measured
* **Processed files**: a web-sized JPG with plate outlines and colony boxes drawn on, and a `_colonies.csv` of per-colony area, centroid, eccentricity, and mean RGB
* **On the run page**: scanner badges, files table, and reports for the processed CSV
When no plates are found, you still get a resized JPG of the whole scan, and colony detection is skipped. An empty `plate_count` usually means the frames weren’t recognized rather than that the plates were blank.
## Hina Microscope [#hina-microscope]
Turns each Nikon ND2 into a single composite image you can view in the browser.
Every channel is reduced to one 2D frame (a maximum projection over Z, the first index over time or position), stretched to the 1st to 99th percentile, then composited onto a brightfield background in each channel’s fluorophore color. The processor reads one frame at a time, so large stacks process without loading the whole file.
* **Expected files**: ND2
* **Metadata**: `sizes`, `channels` (name, excitation and emission wavelength, color), and `dimensions`
* **Processed files**: an RGB overlay JPG for every file in the run
* **On the run page**: channel and size badges, files table, image viewer for the JPGs
Channel layout is identical across a Hina run, so metadata is written once by whichever file finishes processing first. Later files still get their JPG and skip the metadata write. Reprocessing one file therefore won’t correct run metadata if the layout changed mid-run.
An image too large to preview at full size gets a smaller preview instead. For example, a 6221 × 6221 pixel image is previewed at 3110 × 3110 pixels. The raw file is unchanged, and you can download it at full size. A file that showed **Failed** for this reason earlier gets its preview when you choose **Reprocess** on the file or on the run.
Two kinds of image still show **Failed** instead of a preview: an image from a color camera, which Data Hub can’t preview yet, and an image far larger than the Hina Microscope has produced so far. The message on the file says which. In both cases the raw file is stored and you can download it.
## SpectraMax iD3 and iD5 plate readers [#spectramax-id3-and-id5-plate-readers]
Both models share one processor, which converts SoftMax Pro exports into the well data behind the interactive plate maps.
* **Expected files**: SoftMax `.xls`, which is UTF-16 text with a `Plate:` header rather than a real Excel workbook
* **Metadata**: `measurement_mode` (Absorbance, Fluorescence, or Luminescence), `measurement_type` (Endpoint, Kinetic, Spectrum, or Well Scan), and `wavelengths` as bare numbers with no unit suffix
* **Processed files**: `{run_id}_raw_well_data.csv`
* **On the run page**: wavelength, mode, and type badges, plus plate maps built from that CSV
What the plate map shows depends on the read:
* **Kinetic** reads get a time slider. Reads that skip the edge wells keep their time values, and runs that SoftMax Pro stopped early still process.
* **Spectrum** reads (wavelength scans) get a wavelength slider.
* **Luminescence** reads show no wavelength, because luminescence doesn’t use one.
SoftMax Pro writes `#SAT` in a saturated well and `#Low` when the flash lamp missed a well. Those wells show as blank on the plate map, and the rest of the file processes normally.
## DishCam [#dishcam]
Turns each DishCam photo stack into a video you can play on the run page.
* **Expected files**: multi-page TIFF photo stacks, plus the `run.json` file the camera writes next to them
* **Metadata**: camera settings from `run.json`: `fps`, `measured_fps`, `frames`, `duration_seconds`, `quality`, `format`, `started`, and `finished`
* **Processed files**: an MP4 video and a JPEG still frame for each stack
* **On the run page**: a video player at the top of the report, with play, pause, and a timeline you can drag. The still frame shows before you press play.
The video plays at the capture rate or at 10 photos per second, whichever is faster, so slow captures stay watchable. The stored `fps` metadata still records the real capture rate.
A stack processes once both the stack and its `run.json` are in storage, in either order. When a run gathers captures from several folders, each stack uses the `run.json` from its own folder. Watcher 1.1.0 and later keep every folder’s `run.json`, renaming later copies with a short tag such as `run~3f9a1c2b.json`. See [Roll out watcher releases](/docs/watcher-releases#what-each-version-adds).
Large stacks are read one photo at a time. If a stack can’t finish within the [15-minute processing limit](#when-processing-fails), it shows **Failed**, and the message says how far through the photos it got. You can still download the raw file.
## Unchained Labs Aunty [#unchained-labs-aunty]
Reads Aunty exports, which measure protein melting temperature and particle size, and draws a plate of curves.
* **Expected files**: the `.xlsx` export from the Aunty software. Thermal ramps, sizing runs, isothermal holds, and exports that contain only the results table without curves all process.
* **Metadata**: `experiment_type` (`thermal_ramp`, `sizing`, or `isothermal`), `experiment_count`, and the first experiment’s settings: `analysis_mode`, `start_temp_c`, `end_temp_c`, `rate_c_per_min`, `temperature_c`, `analysis_version`, and `experiment_name`
* **Processed files**: `{run_id}_aunty_curves.csv`, with every curve point for every well, and `{run_id}_aunty_plate.json`, which drives the plate view
* **On the run page**: a plate with a small chart in each well. Click a well to open its full chart.
On the instrument page, filter Aunty runs by experiment type, analysis mode, temperature, and ramp rate.
## Types with no processor [#instruments-without-a-matching-processor]
`generic` and `instant_raman` have no processor, and neither does an instrument whose type is unset. Those uploads still work: files are stored, and you can browse, search, claim, comment on, and download them. What you don’t get is extracted metadata, badges, filters, or processed artifacts.
Leaving a new instrument as `generic` is the safe default. It keeps data flowing while someone confirms which processor, if any, can read its files.
## When processing fails [#when-processing-fails]
A file with status `failed` carries an error message on the run page. Open the run, then choose **Reprocess** on that file, or select runs in a run list and choose **Reprocess**. See [Reprocess or delete a run](/docs/browse-runs#reprocess-and-delete).
Processing one file has a 15-minute limit. A file that hits the limit, or is too large to process at all, shows **Failed** with a message that explains why. The raw file stays in storage, and you can download it.
A file that stays in processing for more than 20 minutes without finishing shows **Stalled**. That usually means the processor stopped without reporting back. Reprocess it to try again.
Reprocessing is also the fix after an engineer corrects a parser: the code changed, but the stored result didn’t, until you ask for it again.
## Add support for a new instrument [#add-support-for-a-new-instrument]
**For:** Engineers
Adding an instrument type is a development task spanning the web app and the Lambda, not a configuration change an admin can make in the dashboard. Budget for a database migration and a Lambda image deploy.
**Add the type.** Extend `instrumentTypeEnum` in `web/lib/db/schema.ts` and generate an `ALTER TYPE … ADD VALUE` migration. Reuse an existing type only when the new instrument writes the same vendor’s format.
**Write the processor.** Add a module under `lambda/src/data_hub_lambda/` exposing `process_file(instrument_id, run_id, filename)`, which reports progress through the Data Hub API.
**Register it twice.** Add the type to `PROCESSORS` in `lambda/src/data_hub_lambda/processors.py`, with its filename gate, and to `PROCESSABLE_INSTRUMENT_TYPES` in `web/lib/instruments/processable-types.ts`. These two lists have to agree: the Lambda uses the first to dispatch, and the web app uses the second to decide whether to offer **Reprocess**.
**Add tests** for the processor and for the registry gate.
**Deploy the Lambda image.** No new S3 trigger is needed, since the raw bucket already notifies on every upload.
You can exercise a processor locally without S3 or API access using the `data-hub-process` CLI, and drive the whole handler against a local S3 mirror with its `handler` subcommand.
Full detail, including the archive builder and the Docker build, is in [the Lambda developer docs](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/lambda.md#adding-a-new-instrument--processor).
# Set up integrations (/docs/integrations)
**For:** Administrators and engineers
**Settings → Integrations** holds the connections the whole workspace shares. Only admins see it. The page has three parts:
* **Linear**: where feedback reports go, and where their status comes from
* **Slack app**: the credentials behind personal Slack direct messages and **Connect to Slack**
* **Slack channel**: a webhook that posts a message to a shared channel whenever a new run arrives
Each person’s own notification choices are separate, under **Settings → Notifications**. See [Configure notifications](/docs/configure-notifications).
## How secrets are stored [#integration-secrets-key]
Data Hub encrypts the secrets you paste on this page before saving them: the Linear client secret and signing secret, and the Slack bot token and client secret. It never shows a saved secret again. To change one, paste a new value and save.
Encryption needs the `INTEGRATION_SECRETS_KEY` environment variable on the web app. It is 64 hexadecimal characters; generate one with `openssl rand -hex 32`, set it in Vercel, and redeploy. Until it is set, saving a secret fails, and the Slack app card says so.
If the key changes after secrets were saved, Data Hub can’t read them anymore. The affected field says **Can’t read saved value**. Put the old key back, or paste the value again and save.
## Linear for feedback reports [#linear]
Feedback that people send from Data Hub becomes an issue in your Linear workspace, and the report’s status follows that issue. Until an admin saves a Linear team here, the **Feedback** item stays out of the account menu, and sending feedback through the API or an AI assistant returns an error saying feedback isn’t set up. What reporters see is in [Send feedback](/docs/send-feedback).
The Linear card walks you through four steps.
### 1. Connect your Linear app [#linear-connect]
Data Hub signs in to Linear as an app, so you create one app in Linear for it.
In Linear, go to **Settings → API → OAuth applications** and create a new application.
Name it **Data Hub**. Linear shows this name on every issue it files.
Linear asks for a callback URL. Data Hub doesn’t use one, so paste the URL shown in the step, which is your deployment’s address.
Turn on **Client credentials**, then save.
Copy the client ID and client secret into the Linear card in Data Hub, and choose **Connect**.
Data Hub checks the credentials with Linear before it saves them. If client credentials are turned off for the app, the card says so; turn them on in the app’s settings in Linear and choose **Connect** again.
If the credentials belong to a different Linear workspace than the one already connected, Data Hub asks before switching. Switching clears the team, project, labels, and signing secret. Issues already in Linear stay there.
### 2. Choose where reports go [#linear-destination]
Pick the **Team** that receives reports. Only public teams are listed, because Linear doesn’t show private teams to apps.
Optionally pick a **Project**.
Optionally pick a label for each kind of report: **Bug**, **Feature request**, and **Other**. Labels can come from the team or from the whole workspace. Linear adds the matching label to each new issue.
Choose **Save and turn on feedback**. Everyone sees **Feedback** in their account menu from then on.
Changing the team later clears the project and labels. Status updates also stop until the Linear webhook from step 3 points at the new team.
### 3. Get status updates from Linear [#linear-status-updates]
A webhook tells Data Hub when an issue changes, so it can update the report and tell the person who sent it when the issue closes.
In Linear, go to **Settings → API** and choose **New webhook**.
Paste the **Webhook URL** shown in the step. It ends in `/api/v1/integrations/linear/webhook`.
For **Team**, choose the team from step 2. Under events, turn on **Issues** only.
Save the webhook, then copy its signing secret into **Signing secret** in Data Hub and save.
Data Hub uses the signing secret to check that each update really comes from Linear. The step shows **Waiting for an update from Linear** until the first one arrives. Any change to an issue in the team sends one, including the test report in the next step.
### 4. Send a test report [#linear-test-report]
Choose **Send test report**. Data Hub files a report as you and links to the new Linear issue. Move that issue to **Done** in Linear. The step confirms that the issue closed and that you got a Data Hub notification. You can also skip the test.
### After setup [#linear-after-setup]
Once setup is finished, the Linear card shows a summary: the connected workspace, where reports go, whether status updates are working and when the last one arrived, and what reporters see for each Linear status.
| When the Linear issue moves to | The reporter sees |
| --------------------------------------- | ----------------- |
| **Done**, or any other completed status | Resolved |
| **Canceled** | Declined |
| **Duplicate** | Declined |
| Any other status | Open |
Data Hub notifies the reporter once, when the issue first closes. Moving an issue from one closed status to another, such as **Done** to **Released**, doesn’t notify again. Reopening an issue and closing it again does.
If Data Hub keeps rejecting updates from Linear, **Status updates** shows **Not working**. The usual cause is a wrong signing secret, or Linear turning the webhook off after repeated failures. Paste the signing secret again from the webhook’s page in Linear, and check that the webhook is still turned on.
**Disconnect Linear** turns feedback off. Issues already in Linear stay there.
## Slack app for direct messages [#slack-app]
Personal Slack direct messages and the **Connect to Slack** button use one shared Slack app. Create the app once in the [Slack API portal](https://api.slack.com/apps), then give Data Hub its credentials. You can reuse an existing Slack app, including the one behind the channel webhook.
### Create the app [#create-the-slack-app]
In the Slack API portal, choose **Create New App → From scratch**, or open an existing app.
Under **OAuth & Permissions**, add these scopes:
**Bot Token Scopes:**
| Scope | Purpose |
| ------------ | -------------------------------------------- |
| `chat:write` | Send direct messages with `chat.postMessage` |
**User Token Scopes:**
| Scope | Purpose |
| --------- | ----------------------------------- |
| `openid` | Sign in with Slack (OpenID Connect) |
| `profile` | Display name in the identity token |
Under **Redirect URLs**, add one callback URL per environment. The path is always `/api/v1/settings/slack/callback`:
| Environment | Example URL |
| ----------- | -------------------------------------------------------------------- |
| Local | `http://localhost:3000/api/v1/settings/slack/callback` |
| Staging | `https://staging.datahub.example.com/api/v1/settings/slack/callback` |
| Production | `https://datahub.example.com/api/v1/settings/slack/callback` |
`SLACK_REDIRECT_URI`, or the request origin plus that path when unset, must match a listed URL exactly.
Under **Install App**, choose **Install to Workspace** (or **Reinstall** after changing scopes).
### Give Data Hub the credentials [#slack-app-credentials]
Enter the credentials in the **Slack app** card under **Settings → Integrations**, or set them as environment variables on the web app. A value saved in the card takes effect right away, with no redeploy, and is used instead of the matching environment variable.
| Card field | Environment variable | Where to find it | Required |
| ------------------------ | --------------------- | ------------------------------------------------------- | --------------------------------------------------------- |
| **Bot token** | `SLACK_BOT_TOKEN` | **Install App** → Bot User OAuth Token (`xoxb-…`) | Yes |
| **Client ID** | `SLACK_CLIENT_ID` | **Basic Information** → App Credentials | Yes |
| **Client secret** | `SLACK_CLIENT_SECRET` | **Basic Information** → App Credentials | Yes |
| **Allowed workspace ID** | `SLACK_TEAM_ID` | Workspace URL segment `T…` in `app.slack.com/client/T…` | No, but recommended so connections stay on your workspace |
Each field shows where its value comes from: **Saved here**, **Using** followed by the variable name, or **Not set**. **Remove saved value** makes the environment variable the source again. If Data Hub can’t read a saved value, for example because `INTEGRATION_SECRETS_KEY` changed, it uses the environment variable in the meantime.
Two settings exist only as environment variables, and changing them needs a redeploy:
| Variable | Purpose | Required |
| -------------------- | ------------------------------------------------------------------------------------------------------------------ | -------------------------------- |
| `SLACK_STATE_SECRET` | Signs the short-lived value Data Hub checks while a person connects Slack. Generate with `openssl rand -base64 32` | No (falls back to `AUTH_SECRET`) |
| `SLACK_REDIRECT_URI` | Full callback URL, when it differs from the request origin | No |
Members can then connect under **Settings → Notifications**. Deploy runbooks are linked from [Deploy Data Hub](/docs/self-hosting).
## Slack channel for new runs [#slack-channel]
The **Slack channel** card holds a Slack incoming webhook. Data Hub posts one message to that channel each time a new run is reported. This is separate from personal direct messages, which each person turns on for themselves.
In your Slack app, open **Incoming Webhooks**, turn them on, and add a webhook for the channel you want.
Copy the webhook URL. It starts with `https://hooks.slack.com/services/`.
Paste it into the **Slack channel** card and save. The card shows **Configured** and who last changed it.
To change channels, paste a new webhook URL. To stop the messages, remove the webhook. This setting has no environment variable.
# Manage instruments (/docs/manage-instruments)
**For:** Administrators
When a lab operator registers an instrument from the watcher, it appears as `pending` and cannot upload until you confirm it. Operator-side setup is in [Set up an instrument](/docs/set-up-an-instrument). Retiring and reactivating are in [Retire an instrument](/docs/retire-an-instrument).
## Instrument states [#instrument-states]
The **Instruments** page has tabs for active, pending, and retired instruments.
| Status | UI label | Meaning |
| ---------- | -------- | ------------------------------------------------------------ |
| `pending` | Pending | Registered but not confirmed. Watchers cannot upload. |
| `active` | Active | Confirmed and accepting uploads. |
| `inactive` | Retired | Deactivated; no new uploads. Historical runs stay browsable. |
## Confirm a pending instrument [#confirm-a-pending-instrument]
Open the Data Hub web app and go to **Instruments**.
Open the **Pending** tab (or find the yellow `pending` badge).
Click **Confirm**. Status becomes `active` and the operator’s watcher can upload.
Confirming is admin-only for signed-in users. Under the hood it is `PATCH /api/v1/instruments/:id`. Token automation uses the `instruments:write` scope instead. See the [generated API docs](/docs/api).
## Read an instrument’s page [#view-instrument-health]
The top of each instrument page shows what you need to tell whether its files will reach Data Hub:
* **Watcher status**: a badge next to the instrument name. **Online** means the watcher checked in recently, **Offline** means it hasn’t, and **No Watcher** means none has registered. The watcher checks in once a minute, so a watcher that was just restarted can take up to a minute to show as online. Hover over the badge for details.
* **Watched folder**: the folder the watcher reads from. Hover over it to see which PC it is on.
* **File patterns**: the patterns the watcher uploads, such as `*.csv`.
Files saved anywhere other than the watched folder, such as a shared network drive, don’t appear in Data Hub. Neither do files whose names don’t match a pattern. The folder and patterns come from the watcher’s config on the instrument PC, so change them there with `data-hub-watcher config edit`; see [Watcher configuration](/docs/configuration-reference).
Watcher health comes from heartbeats; see [Manage the watcher fleet](/docs/manage-watchers).
## Edit an instrument [#edit-an-instrument]
The three-dot menu on an instrument page, and the row menu on the instruments table, hold the admin actions:
* **Edit details**: change the **Display name** and the **Instrument type**
* **View watcher**: open the watcher registered for this instrument
* **Retire** or **Reactivate**: see [Retire an instrument](/docs/retire-an-instrument)
The instrument type decides which processor reads the instrument’s files, including DishCam and Aunty; see [Instrument data preprocessing](/docs/instrument-preprocessing). The instrument ID never changes, because it is the storage key prefix.
# Manage members (/docs/manage-members)
**For:** Administrators
Workspace members sign in with Google OAuth. Admins decide who else has the admin role. Member vs admin capabilities are summarized in [Security and permissions](/docs/security#roles).
## Open the members list [#open-the-members-list]
Go to **Settings → Members**. Non-admins who open the URL see an “Admins only” message; the settings sidebar hides the entry for them.
The table lists every signed-in user: name, email, role badge, and an **Admin** toggle.
## Promote or demote [#promote-or-demote]
Toggle **Admin** on a row to grant or revoke admin access. Every privileged action re-checks the database, so the change applies to that user’s next one. Their view of the dashboard can lag by up to 5 minutes, because admin-only buttons and sidebar entries come from their [cached session](/docs/security#session-lifetime).
You cannot demote yourself. That guard prevents locking the workspace out of its last admin. Ask another admin to demote you if needed.
## First admin bootstrap [#first-admin-bootstrap]
The first admins come from the `ADMIN_EMAILS` environment variable (comma-separated, case-insensitive). Listed users are promoted on every sign-in. After that, any admin can promote others from this page.
There is no invite flow in the product: people appear here after they sign in successfully to your deployment. Every Google account your OAuth client accepts can sign in, so that client, not this page, controls who appears here. See [Who can sign in](/docs/security#who-can-sign-in).
# Issue and revoke tokens (/docs/manage-tokens)
**For:** Administrators
Personal access tokens authenticate the watcher, the processing Lambda, and scripts that call the Data Hub API. Creating and revoking tokens requires the workspace admin role. Members can view the list at **Settings → Access Tokens** but cannot create or delete tokens. Roles and scopes: [Security and permissions](/docs/security).
## How a token acts [#how-a-token-acts]
A token acts as itself, under its own name. It doesn’t act as the admin who created it or as any other person.
* Anything a token does is recorded under the token’s name: comments, deleting a run, retiring an instrument, removing a watcher, and notifications sent through the API.
* Run pages, instrument pages, watcher pages, comments, and the notification bell show the token’s name with a key icon.
* A token can post comments, and it can edit and delete only the comments it posted.
Some actions need a signed-in person, so tokens can’t do them at all: claiming runs, sending feedback, and reviewing feedback. AI assistants aren’t affected, because they sign in as the person who approved them; see [MCP overview](/docs/mcp).
The reverse also holds: the API routes that the watcher and the Lambda call accept only tokens. A browser session gets `401 Unauthorized` on watcher registration and heartbeats, upload links, file updates, run creation and updates, and instrument creation.
## Create a token [#create-a-token]
### In the web app [#in-the-web-app]
Sign in as an admin.
Go to **Settings → Access Tokens**.
Click **Create token**.
Enter a **Name** that identifies the client (for example, `FPLC watcher, Lab 201`). This is the name people see wherever the token acts, so make it recognizable.
Pick an **Expiration**: 30 days, 90 days (the default), 1 year, or no expiry.
Under **Scopes**, click a preset, or click **Customize scopes** to pick individual scopes.
Click **Create token**.
The plaintext token is shown once: copy it immediately. It starts with `dhub_` followed by a 64-character hex string.
### Scope presets [#scope-presets]
A preset selects the minimum scopes a common client needs:
* **Read-only**: view instruments, runs, files, watchers, and archive jobs; change nothing
* **Watcher**: what a watcher needs to register, report runs, and upload files
* **Lambda**: what the processing Lambda needs to read instruments and runs, and write run and file results
The **Lambda** preset includes `runs:read`, which DishCam processing needs to pair each photo stack with its `run.json`. A Lambda token created from an older preset gets `403 FORBIDDEN` on DishCam files. Create a new token from the current preset and update the Lambda’s `DATA_HUB_API_KEY`; the [Lambda developer docs](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/lambda.md#supported-instrument-types) describe the swap.
You cannot change a token’s scopes after creation. To change them, revoke the token and create a new one.
### Via the API [#via-the-api]
Creating and deleting tokens is admin-only and needs a signed-in browser session (cookie). Tokens cannot manage other tokens.
```sh
curl -X POST https://datahub.example.com/api/v1/tokens \
-H "Cookie: session_cookie_here" \
-H "Content-Type: application/json" \
-d '{
"name": "FPLC watcher",
"scopes": ["instruments:read", "instruments:write", "watchers:read", "watchers:report", "runs:create", "runs:update", "runs:upload", "files:update"],
"expires_at": "2027-01-01T00:00:00Z"
}'
```
`scopes` is required and must be a non-empty list of explicit scopes. The wildcard `*` and the older coarse scopes `runs:write`, `files:write`, and `watchers:write` are rejected. `expires_at` is optional; if omitted, the token never expires. The request is rejected if it includes `user_id`, because tokens don’t belong to a person. The response includes the plaintext once in the `token` field.
## Use a token [#use-a-token]
### With the watcher [#with-the-watcher]
During `data-hub-watcher init`, paste the token when prompted. Or set it first:
```sh
export DATA_HUB_API_KEY=dhub_your_token_here
data-hub-watcher init
```
### With the REST API [#with-the-rest-api]
```sh
curl https://datahub.example.com/api/v1/instruments \
-H "Authorization: Bearer dhub_your_token_here"
```
More in the [API overview](/docs/api).
Tokens don’t authenticate MCP clients, which sign in with OAuth instead. See [MCP overview](/docs/mcp).
## View tokens [#view-tokens]
**Settings → Access Tokens** lists every active token with these columns:
* **Name**
* **Created by**: the admin who made the token. For tokens created before October 6, 2026, this shows the person the token used to act as.
* **Token**: the prefix only, because the full token is never stored
* **Scopes**
* **Last used**
* **Expires**
* **Created**
## Revoke a token [#revoke-a-token]
Go to **Settings → Access Tokens**.
Click the delete icon on the token’s row and confirm.
The token stops working immediately. Clients using it start receiving `401 Unauthorized`.
```sh
curl -X DELETE https://datahub.example.com/api/v1/tokens/ \
-H "Cookie: session_cookie_here"
```
A revoked token leaves the list, but Data Hub keeps its record. Wherever the token acted, its name stays visible and is marked **(revoked)**.
## After revoking a token [#after-revoking-a-token]
If a running watcher used that token:
Create a new token with the **Watcher** preset.
On the instrument PC, re-run `data-hub-watcher init` and enter the new token. The new token takes over the existing watcher registration, so you don’t need to deregister the old one first.
Restart the watcher (`data-hub-watcher watch` or the Windows service).
## When someone leaves [#when-someone-leaves]
Tokens keep working after the admin who created them leaves or their account is deleted, because tokens act as themselves. When someone leaves, check the **Created by** column and revoke any of their tokens that nobody else should keep using.
## Token hygiene [#token-hygiene]
* Name tokens so you can tell which PC or client each belongs to
* Set expirations for temporary setups
* Scope to least privilege; see [Token scopes](/docs/security#token-scopes)
* Revoke immediately when a watcher is decommissioned or a token may have been exposed
Tokens are hashed with SHA-256 before storage; plaintext is never persisted.
# Manage the watcher fleet (/docs/manage-watchers)
**For:** Administrators
The **Watchers** area is the fleet view: every registered watcher, its health, deregistration, and the upload queue. Cutting releases and advertising versions is separate: [Roll out watcher releases](/docs/watcher-releases).
## Monitor watcher health [#monitor-watcher-health]
Each watcher sends a [heartbeat](/docs/concepts#heartbeat) every 60 seconds with version, instrument, watch directory, upload mode, activity counters, and uptime. The fleet list surfaces:
* **Status**: **Online**, **Unresponsive** (no heartbeat for more than 5 minutes), **Stopped** (shut down cleanly), or **Registered** (no first heartbeat yet). See [Watcher status](/docs/concepts#watcher-status).
* **Last heartbeat**: a stale value is the first sign of a stopped, disconnected, or crashed watcher
* **Hostname and instrument assignment**
* **Events**: per-watcher log (`watcher_started`, `run_reported`, `file_uploaded`, `update_*`, and `error` with a `details.kind` discriminator)
Events are the primary remote-diagnosis tool. For operator-side symptoms, see [Troubleshoot a watcher](/docs/troubleshoot-a-watcher).
Open a watcher to see its event log, a status chart with a connectivity strip of recent heartbeats, and its configuration.
## Deregister a watcher [#deregister-a-watcher]
Each instrument can have at most one active watcher. Deregister the old one when a PC is reimaged or replaced, or to clear “instrument already has an active watcher” on a new install.
* **Web app**: open the watcher under **Watchers** and click **Deregister**, or choose **Deregister** from the watcher’s row menu
* **API**: `DELETE /api/v1/watchers/:watcherId` (requires `watchers:admin`)
Deregistration is a soft-delete: heartbeats, events, and runs stay under **Watchers → Deregistered**. A watcher removed through the API with a token shows that token’s name as the one that removed it.
Retiring an instrument also deregisters its watchers automatically; see [Retire an instrument](/docs/retire-an-instrument).
You don’t need to deregister a watcher just because its token was revoked. Re-running `data-hub-watcher init` with a new token takes over the existing registration; see [After revoking a token](/docs/manage-tokens#after-revoking-a-token).
## Request uploads [#request-uploads]
**Upload** on a run page, or on selected runs in a run list, asks the watcher to send files that are still on the instrument PC. The watcher polls the server’s upload queue once a minute and uploads the files that were requested.
* In **manual** upload mode, every file waits in this queue until someone requests it.
* In **auto** upload mode, files upload on their own, and the queue covers files that need another try. This works with watcher 1.2.1 and later; older watchers in auto mode never pick up requests, so the file shows **Uploading** and stays that way.
## Approving manual uploads [#approving-manual-uploads]
For instruments in [manual upload mode](/docs/concepts#upload-mode), the watcher reports runs but waits for server approval before uploading file bytes. Approve a file with **Upload**, or choose **Dismiss** to tell the watcher not to upload it. To act on several files at once, select them in the files table and use the bar that appears.
Safeguards that keep the queue healthy:
* Changing an instrument’s `watch_directory` reverts pending upload requests to `detected` so stale paths drain instead of erroring
* A queued file that keeps failing (missing on disk or failing to upload) is retried on at most three polls, then cancelled server-side
# Overview (/docs/overview)
**For:** Everyone
Instrument data tends to stay where it was made. Files sit on the PC wired to the plate reader, named by whoever was at the bench, backed up if someone remembers. Finding a result from three months ago means knowing which machine made it and who was running it.
Data Hub moves that data off the instrument PC automatically. A small program called a watcher runs on each instrument PC, monitors its output folder, uploads new files as they appear, and groups them into runs. From then on the data is searchable in a browser, tied to the person who ran it, and reachable from a script or an AI agent without anyone copying files to a shared drive.
## What Data Hub isn’t [#what-data-hub-isnt]
Worth setting expectations early, because the name invites a few wrong guesses:
* It doesn’t change your raw files. Uploads are byte-for-byte, and processing only ever adds new files alongside them
* It isn’t an electronic lab notebook. There’s a comment thread per run for notes, and that’s the extent of it
* It isn’t analysis software. It parses vendor formats into something viewable and downloadable; the analysis is still yours
* It doesn’t run on the instrument. Processing happens server-side after upload, so nothing competes with acquisition software for resources
## How a file becomes a run [#how-a-file-becomes-a-run]
1. A [watcher](/docs/concepts#watcher) on the instrument PC notices a new file in the folder it’s watching, waits for the file to stop changing, and groups it with the other files from the same acquisition into a [run](/docs/concepts#run).
2. The watcher uploads those files to cloud storage. For [supported instrument types](/docs/instrument-preprocessing), server-side code then reads the vendor’s format, pulls out settings like wavelength or imaging mode, and builds viewable versions such as plate maps or contrast-adjusted images.
3. The run appears in the web app, the REST API, and the Model Context Protocol (MCP) endpoint at the same time. They’re three views of one record.
The full picture, including where each piece runs, is in [Architecture](/docs/architecture).
## Where your data shows up [#where-your-data-shows-up]
Five places in the web app, in the order most people use them:
* **Home**: every instrument’s recent runs together, with counts for today and this week, and today’s comments
* **Instruments**: one instrument’s runs, with columns matching what that instrument records
* **My runs**: runs you’ve claimed, so you can find your own work without filtering
* **Comments**: every run comment by day, filtered to your runs or to one instrument if you like
* **Watchers**: which instrument PCs are reporting in, for whoever maintains them
To see what has changed in Data Hub recently, open **Changelog** from the account menu.
If a term on those pages is unfamiliar, [Concepts](/docs/concepts) defines the vocabulary in one page. Run, watcher, run status, and instrument type are the four worth reading first.
## Get at the same data from code [#get-at-the-same-data-from-code]
The web app has no private back door. Everything it shows comes from a REST API that you can call yourself, and an MCP endpoint that AI clients can call on your behalf. Both read the same records, so a script sees exactly what the dashboard sees.
* **REST**: served at `/api/v1/` on your own deployment host. Each deployment publishes its own OpenAPI 3.1 document at `/api/v1/openapi.json`, so you can generate a client rather than hand-write one. See the [API overview](/docs/api)
* **MCP**: served at `/mcp/v1` over Streamable HTTP, and added to Claude Code, Claude Desktop, or Cursor by pasting the URL. Agents can then search runs, read metadata, and fetch files during a conversation. Clients that support MCP Apps also show the same interactive run reports as the web app. See the [MCP overview](/docs/mcp)
The two differ in how you authenticate. REST clients carry a [personal access token](/docs/concepts#personal-access-token) that an admin creates in the web app, scoped and revocable. A token acts under its own name, so anything a script does is labeled with the token rather than with a person. MCP clients send you through a browser sign-in and a consent screen instead, then act as you with your own permissions, so nobody has to paste a credential into an AI client.
Typical uses look like pulling every run from an instrument for a batch reanalysis, writing the latest run’s archive to disk from a script, or asking an agent which plates from last week failed processing.
## What’s supported today [#whats-supported-today]
Nine instrument types have a processor, covering plate readers, gel docs, qPCR, TapeStation, FPLC, flatbed plate scanners, ND2 microscopy, DishCam time-lapse photos, and the Unchained Labs Aunty for protein melting temperature and particle size. [Instrument data preprocessing](/docs/instrument-preprocessing) lists what each one extracts.
An instrument without a matching processor still works. Its files upload, and you can search, claim, comment on, and download them. What you don’t get is extracted metadata, badges, or rendered previews. Adding a new type is a development task, not a settings change.
## Self-hosted by design [#self-hosted-by-design]
Data Hub isn’t a service you sign up for. Your team runs it, which is the point: raw instrument data and everything derived from it stay on infrastructure you control, and the source is open under the MIT license.
Running it means four pieces, deployed once by an engineer:
* The web app, which also serves the API and the MCP endpoint, on Vercel
* A PostgreSQL database for instruments, runs, files, and comments
* AWS S3 for the files themselves, split into raw and processed buckets
* An AWS Lambda that does the per-instrument processing and builds download archives
A Google OAuth client, created during that deploy, is the fifth piece and the one worth knowing about even if you’ll never touch the stack. It decides who can sign in at all: only your Google Workspace organization, or any Google account. Nothing inside Data Hub overrides that choice, so it’s a deployment decision rather than a setting an admin can change later.
Until the stack exists, nobody can sign in and no watcher can upload. The deployment runbooks live in the repository rather than on this site; [Deploy Data Hub](/docs/self-hosting) points to them.
## Getting access [#getting-access]
Sign in with your Google account, assuming it’s one the deployment allows. Everyone who signs in can browse instruments, runs, and files, comment on runs, and [send feedback](/docs/send-feedback) about Data Hub. Admins can additionally confirm instruments, issue tokens, promote other users, and connect Linear and Slack, and only an existing admin can grant you that. See [Security and permissions](/docs/security#roles) for what each role can do.
## Start here [#start-here]
## Reading these docs with AI [#reading-these-docs-with-ai]
Every page has a **Copy Markdown** button and a **View Options** menu. Use them to:
* Copy a page as Markdown into an assistant
* Open raw Markdown (`/docs/.md`) or the dumps at [`/docs/llms.txt`](/docs/llms.txt) and [`/docs/llms-full.txt`](/docs/llms-full.txt)
# Retire an instrument (/docs/retire-an-instrument)
**For:** Administrators
Retire an instrument when it is decommissioned or out of service for a long stretch. Retirement sets status to `inactive` (shown as **Retired**), removes it from the sidebar and dashboard lists of live instruments, and stops new uploads. Nothing is deleted from storage; you can reactivate later.
## What retirement does [#what-retirement-does]
When you retire an instrument:
* Status becomes `inactive` (UI: **Retired**), with `retiredAt` / `retiredBy` audit fields
* It stops accepting new runs and watchers
* All existing runs and files stay browsable
* Every active watcher on that instrument is deregistered in the same request so they stop heartbeating
Instrument IDs stay permanent. Retire rather than trying to rename or delete an instrument that has already produced data.
## Retire from the web app [#retire-from-the-web-app]
Open the instrument’s detail page.
Open the admin actions menu and choose **Retire**.
Confirm in the dialog. The dialog lists run count and how many watchers will be deregistered.
The instrument moves to the **Retired** tab on **Instruments**.
## Reactivate [#reactivate]
From a retired instrument’s detail page, choose **Reactivate**. Status returns to `active` and the instrument reappears in the sidebar and dashboard.
Watchers deregistered at retirement do **not** reconnect automatically. An operator must re-register (usually `data-hub-watcher init` on the instrument PC) after reactivation.
## Via the API [#via-the-api]
```sh
# Retire
curl -X PATCH https://datahub.example.com/api/v1/instruments/bio-rad-cfx96 \
-H "Authorization: Bearer dhub_your_token_here" \
-H "Content-Type: application/json" \
-d '{"status":"inactive"}'
# Reactivate
curl -X PATCH https://datahub.example.com/api/v1/instruments/bio-rad-cfx96 \
-H "Authorization: Bearer dhub_your_token_here" \
-H "Content-Type: application/json" \
-d '{"status":"active"}'
```
Session callers need the admin role; bearer tokens need `instruments:write`.
# Security and permissions (/docs/security)
**For:** Administrators and developers
This page explains who can do what in Data Hub: the two roles, the scope system
that gates token-authenticated requests, and how to handle tokens safely. For the
mechanics of creating and revoking tokens, see [Issue and revoke tokens](/docs/manage-tokens).
## Authentication [#authentication]
The API accepts two kinds of credentials:
* **Session cookies**: used by the web dashboard, which signs people in with Google OAuth via [Better Auth](https://www.better-auth.com). Session callers implicitly hold every scope; scope checks only apply to token requests.
* **Bearer tokens**: personal access tokens (`dhub_…`) used by the watcher, the Lambda, and scripts, sent as `Authorization: Bearer `. A token acts as itself, under its own name; see [How tokens act](#how-tokens-act).
* **OAuth access tokens**: issued to MCP clients after you sign in and approve a consent screen. Covered in [MCP access](#mcp-access).
Tokens are hashed with SHA-256 before storage; the plaintext is shown exactly once, at creation.
### Who can sign in [#who-can-sign-in]
Any Google account that finishes the sign-in flow becomes a member of your deployment. Data Hub itself checks nothing: there is no invite flow, no email allowlist, and no email-domain check, so a first successful sign-in creates the member record.
Restrictions live on the Google side instead, in the OAuth client your team creates at deploy time. A consent screen set to **Internal** accepts only accounts in your Google Workspace organization. Set to **External**, it accepts any Google account, including personal ones. Engineers make that choice when they [create the OAuth client](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/first-time-deployment.md#create-a-google-oauth-client).
Data Hub accepts 10 Google sign-in attempts per minute from one IP address and rejects the rest.
Every new member starts in the member role, never admin. See [First admin bootstrap](/docs/manage-members#first-admin-bootstrap) for how the first admin gets the role.
### Session lifetime [#session-lifetime]
Signing in stores a session in the database and sets a signed cookie. The session expires 7 days after its last refresh, and the dashboard refreshes it at most once a day, so a session in daily use doesn’t lapse on its own.
That cookie also carries a signed copy of the session for 5 minutes, which keeps most page loads off the database. Requests read the copy instead of the row. A change made in the database directly, such as deleting a session, therefore applies when the copy expires rather than on the next request.
Revoking someone’s access on the Google side, by removing them from your Workspace organization for example, blocks their next sign-in but leaves any session already open working until it expires. Delete that user’s rows from the `session` table to cut them off sooner. Signing out from the sidebar ends the session immediately.
## Roles [#roles]
| Role | Can do |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Member** | Sign in to the dashboard; view instruments, runs, files, and the token list; claim runs; comment, and edit or delete their own comments; delete and restore runs; send feedback and see the reports they sent. Cannot create or revoke tokens, confirm or update instruments, or manage other users. |
| **Admin** | Everything members can, plus: confirm and update instruments; create and revoke tokens; promote and demote other users; delete any run comment; review every feedback report; manage **Settings → Integrations**; and set watcher release versions. |
The first admin is bootstrapped from the `ADMIN_EMAILS` environment variable
(comma-separated, case-insensitive); listed users are promoted on every sign-in.
After that, any admin can promote others from **Settings → Members**. Admins
cannot demote themselves: that guards against locking the workspace out of its
last admin.
### Admin-gated operations [#admin-gated-operations]
A subset of mutations require the admin role in addition to (or instead of) a scope
check:
* `PATCH /api/v1/instruments/:id`: session callers must be admin (bearer-token automation still uses `instruments:write`, so watcher/Lambda flows are unaffected).
* `POST /api/v1/tokens` and `DELETE /api/v1/tokens/:id`: admin-only, session-only. Bearer tokens cannot manage other tokens.
* `GET /api/v1/users`, `PATCH /api/v1/users/:userId`: admin-only, session-only. Used by **Settings → Members**.
* `/api/v1/settings/integrations/*`: admin-only, session-only. Used by **Settings → Integrations**.
* `DELETE` on a run comment: admins can delete any comment; everyone else, including tokens, can delete only their own.
* `GET /api/v1/feedback`: admins get every report; members get only the reports they sent.
## How tokens act [#how-tokens-act]
A personal access token acts as itself, not as the admin who created it. Data Hub records everything a token does under the token’s name, and the app shows that name with a key icon: on comments, deleted runs, retired instruments, removed watchers, and notifications sent through the API. A revoked token keeps its record, so its name stays visible and is marked **(revoked)**.
Because a token isn’t a person, three things need a signed-in person instead: claiming runs, sending feedback, and reviewing feedback. Tokens can’t do them, whatever their scopes.
The routes that the watcher and the Lambda call work the other way round. They accept only tokens and return `401 Unauthorized` to a browser session, so a signed-in member can’t act as a watcher. These routes cover watcher registration and heartbeats, upload links, file updates, run creation and updates, and instrument creation.
Tokens keep working when the admin who created them leaves. Revoke a departing person’s tokens yourself; see [When someone leaves](/docs/manage-tokens#when-someone-leaves).
## Token scopes [#token-scopes]
Every personal access token carries an array of permission **scopes**. A token-authenticated request is rejected with `403 FORBIDDEN` when the token’s scopes don’t include the scope a route requires. Issue tokens with the least privilege they need. New tokens must list fine-grained scopes; `POST /api/v1/tokens` rejects the wildcard `*` and the older coarse scopes `runs:write`, `files:write`, and `watchers:write`. Tokens that already carry those still work until you rotate them.
| Scope | Grants |
| ---------------------- | ---------------------------------------------------------------------------------- |
| `instruments:read` | Read instruments, dashboard status, and file patterns |
| `instruments:write` | Create instruments and edit their configuration |
| `runs:read` | Read and search runs, their files, comments, and attributions |
| `runs:create` | Create run records |
| `runs:update` | Update run metadata |
| `runs:delete` | Soft-delete and restore runs |
| `runs:reprocess` | Re-run the processing workflow for a run |
| `runs:upload` | Request presigned S3 URLs to upload files to a run |
| `runs:comment` | Post run comments as this token, and edit or delete the ones it posted |
| `files:read` | Read file metadata and download files and run archives |
| `files:create` | Register file records against a run |
| `files:update` | Update file metadata and upload state |
| `files:delete` | Delete files |
| `files:reprocess` | Re-run the processing workflow for a file |
| `watchers:read` | Read watcher status, heartbeats, and upload queues |
| `watchers:report` | Register, heartbeat, send events, and push watcher config |
| `watchers:admin` | Delete watchers |
| `archive-jobs:read` | Read run-archive job status |
| `archive-jobs:write` | Update run-archive job status (Lambda callback) |
| `notifications:create` | Send notifications to people, optionally about a run |
| `*` | Matches every scope. Reserved for legacy backfilled tokens; rejected on new tokens |
A watcher token typically needs `instruments:read`, `instruments:write`, `watchers:read`, `watchers:report`, `runs:create`, `runs:update`, `runs:upload`, and `files:update`. That is what the **Watcher** preset selects. The **Lambda** preset selects `instruments:read`, `runs:read`, `runs:create`, `runs:update`, `files:create`, `files:update`, and `archive-jobs:write`. A read-only analysis token needs the `:read` scopes for the data you want to query.
The `runs:attribute` and `feedback:admin` scopes no longer exist, because claiming runs and reviewing feedback need a signed-in person. They were removed from existing tokens on October 6, 2026.
## Integration secrets [#integration-secrets]
The Linear and Slack secrets that admins save under **Settings → Integrations** are encrypted in the database with the `INTEGRATION_SECRETS_KEY` environment variable, and the app never shows them again after saving. See [Set up integrations](/docs/integrations#integration-secrets-key).
## MCP access [#mcp-access]
The MCP server at `/mcp/v1` doesn’t accept personal access tokens. An AI client runs an OAuth flow instead: you sign in to Data Hub, approve a consent screen that names the client and the access it wants, and the client receives an access token valid for that deployment’s MCP endpoint alone. A token issued to a client with no person behind it is rejected, so every MCP action attributes to a user.
MCP recognizes two scopes rather than the `:` pairs above:
* `read`: required to connect, and covers every read-only tool, resource, and prompt
* `write`: additionally required by any tool that changes data, such as claiming a run or deleting one
A grant belongs to one client and one person, and it carries that person’s own permissions. Clients register themselves at connect time, which creates an OAuth client record but grants nothing on its own: Google sign-in still decides who gets through, and the consent screen still decides what they hand over.
The dashboard has no screen for reviewing or revoking MCP grants yet. Removing someone from your Google Workspace organization blocks new grants; cutting off a client that already holds a token means deleting the matching rows from the `oauthAccessToken` and `oauthRefreshToken` tables.
Client setup is in [MCP overview](/docs/mcp).
## Token hygiene [#token-hygiene]
* **Name tokens descriptively** (e.g. “FPLC watcher, Lab 201”) so you can tell what each one is for.
* **Set expirations** for temporary setups.
* **Scope to least privilege**: don’t hand out broad tokens for a single read-only integration.
* **Revoke immediately** when a watcher is decommissioned or a token may have been exposed. Revocation is instant; clients using the token start getting `401 Unauthorized`. See [After revoking a token](/docs/manage-tokens#after-revoking-a-token).
* **Revoke a departing person’s tokens.** Tokens don’t stop working when the person who created them leaves. Check the **Created by** column under **Settings → Access Tokens**.
## Public pages, gated bodies [#public-pages-gated-bodies]
Some web pages (dashboard, instruments, run detail, settings) are reachable without
a session so link unfurlers can read page metadata. The page body renders a
sign-in prompt instead of real data when there’s no session, and three independent
layers (`robots` metadata, `robots.txt`, and an `X-Robots-Tag` header) keep the
product app out of search indexes. The `/api/v1/*` surface always requires a
session cookie or bearer token.
# Deploy Data Hub (/docs/self-hosting)
**For:** Engineers
Data Hub is self-hosted, so nobody can sign in or install a watcher until your team has stood up the backend. This page is orientation only: what gets deployed, what access you need, and in what order. It doesn’t walk through the commands.
For the actual deploy instructions, open [First-time deployment](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/first-time-deployment.md) in the data-hub repository. That guide has every command, env var, and checklist. Read this page first if you want the map before you start.
## What you’ll deploy [#what-youll-deploy]
Three pieces, each on infrastructure you choose:
| Piece | What it does | Where it runs |
| ---------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------- |
| PostgreSQL database | Stores instruments, runs, files, watchers, and tokens | Any PostgreSQL host, managed or self-run |
| Web app | Serves the dashboard, the REST API, and the MCP endpoint | Vercel |
| Storage and processing | S3 buckets for raw, processed, and archived files, plus the Lambda that processes uploads | AWS, deployed with AWS SAM |
You’ll also create a Google OAuth client, which is what actually controls who can reach your deployment. Data Hub has no invite flow and no email allowlist of its own, so that client’s setting is the whole access policy. Decide it deliberately and read [Who can sign in](/docs/security#who-can-sign-in) first.
Give `staging` and `production` separate databases and separate AWS stacks. They’re independent deployments holding independent data.
## Access you need before you start [#access-you-need-before-you-start]
The deploy stalls without all four, so confirm them up front:
* **Admin-level AWS credentials.** The first deploy creates the stack, which the automated deploy role can’t do. It can only update a stack that already exists
* **Vercel project access**, including permission to set environment variables for the environment
* **Google Cloud Console access** to create the OAuth client
* **GitHub repository settings access**, to store the secrets that let continuous integration handle every deploy after this one
You’ll also need the AWS CLI and the AWS SAM CLI installed locally.
## How the deploy is sequenced [#how-the-deploy-is-sequenced]
The web app and the AWS stack each need something the other produces, so the order matters. This is the sequence only; [First-time deployment](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/first-time-deployment.md) has the commands for each step:
**Create the database** and keep its connection string. Make sure the host accepts connections from Vercel and from your continuous integration runners.
**Deploy the web app**, set its environment variables, apply the database migrations, then sign in and create the access token the Lambda will use.
**Bootstrap AWS resources**, once per AWS account rather than once per environment. This creates the shared resources both environments draw on.
**Deploy the AWS stack** for this environment, using the web app’s URL and that token.
**Wire the AWS outputs back into the web app**, then redeploy so it picks them up.
Expect the web app to deploy cleanly at step 2 and still not work. Anything touching files, uploads, reprocessing, and run archives stays broken until step 5 connects it to AWS. A green deployment is not the finish line.
## Things that catch people out [#things-that-catch-people-out]
These failures account for most lost time on a first deploy:
* **Set `ADMIN_EMAILS` before you sign in for the first time.** It’s what promotes the first admin. Sign in before setting it and you get an account with no permission to create tokens or confirm instruments
* **Point `BETTER_AUTH_URL` at this deployment’s own public origin.** Data Hub builds the Google callback URL from it, so a value copied from another environment sends people to that other host and sign-in fails. AI clients read the same value when they sign in over MCP, so a wrong one breaks them too
* **Google sign-in on Vercel preview deployments needs `OAUTH_PROXY_URL` and `OAUTH_PROXY_SECRET`.** Google matches each redirect URI exactly and every preview gets a new URL, so previews route the callback through one stable deployment whose URI is registered. Production registers its own and leaves both variables unset
* **Create the Lambda’s token from the Lambda preset.** The Lambda checks an instrument’s type before choosing a processor, so without `instruments:read` every upload fails on a permission error rather than an obvious misconfiguration. DishCam processing also reads the run’s file list, so it fails without `runs:read`. The **Lambda** preset includes both
* **Set `ENABLE_S3_FILES` to `true` or `false` for the AWS stack.** An empty value fails the deploy. Use `false` unless you want the Lambda to read very large files from a mounted copy of the raw bucket, which needs a VPC and costs extra; the runbook covers turning it on later
* **Bootstrapping fails if the AWS account already has a GitHub identity provider** from an unrelated project. Reuse the existing one instead of creating a second
* **Set `INTEGRATION_SECRETS_KEY` before an admin saves Slack or Linear secrets.** Data Hub encrypts those secrets with it, and saving fails without it. Generate one with `openssl rand -hex 32`
* **The Slack channel webhook and the Linear connection aren’t environment variables.** A workspace admin sets them under **Settings → Integrations** after the deploy; see [Set up integrations](/docs/integrations). The Slack app credentials for direct messages can be either environment variables or saved there. Feedback stays turned off until an admin connects Linear
## What finished looks like [#what-finished-looks-like]
Operators can install watchers and instruments start uploading. Hand them [Set up an instrument](/docs/set-up-an-instrument), and create their first tokens through [Issue and revoke tokens](/docs/manage-tokens). To turn on feedback reports and Slack messages, an admin then follows [Set up integrations](/docs/integrations).
From then on, deploys are automatic. Merges deploy the web app, apply migrations, and ship the Lambda, so this manual sequence runs once per environment.
## The runbooks [#the-runbooks]
The deploy commands live next to the code in the data-hub repository. Start with [First-time deployment](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/first-time-deployment.md):
* [First-time deployment](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/first-time-deployment.md): the full sequence above, with every command
* [Continuous integration and deployment](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/ci-and-deployment.md): how later deploys, migrations, and watcher releases happen
* [Local development](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/local-development.md): run the app on a laptop, including a setup that needs no AWS or Google credentials
* [Architecture for contributors](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/architecture.md): internals
For a product-level map of the components and how data moves between them, rather than deploy steps, see [Architecture](/docs/architecture).
# Send feedback and see what’s new (/docs/send-feedback)
**For:** Everyone
Use **Feedback** to tell the team that runs Data Hub about a bug, a missing feature, or anything else about the app itself. Each report becomes an issue in your workspace’s Linear, and Data Hub tells you when that issue is closed.
Feedback is for problems with Data Hub. A problem with one run’s data, such as a wrong plate map, belongs in a comment on that run, where the people who ran it will see it.
## Send a report [#send-a-report]
Open the account menu at the bottom of the sidebar and choose **Feedback**.
Pick a **Type**: **Bug**, **Feature request**, or **Other**.
Enter a **Title** and a **Description**. Say what you expected and what happened instead.
Optionally open **More details** and fill in **Trying to do**, **Tool**, and **Error message**.
Choose **Send feedback**.
Data Hub also records the page you were on, so you don’t need to paste a link. If you send a report with the same title within 24 hours, Data Hub returns the report you already sent instead of creating a second one, unless that report has been closed.
The **Feedback** item appears only after an admin connects Data Hub to Linear. If you don’t see it, ask an admin to follow [Set up integrations](/docs/integrations#linear).
## What happens to a report [#what-happens-to-a-report]
* Data Hub creates an issue in the Linear team your admins chose, with the label they picked for that type of report, if any.
* The team works on the issue in Linear.
* When the issue closes, Data Hub sends you a notification, and a Slack message if you turned that on.
A report’s status follows its Linear issue:
| Linear issue | Report status |
| ---------------------------------- | ------------- |
| Completed, such as **Done** | Resolved |
| Canceled, or closed as a duplicate | Declined |
| Anything else | Open |
The notification names the Linear status too, such as **Resolved (Done)** or **Declined (Duplicate)**. Turn it on or off with **Updates on your feedback** under **Settings → Notifications**; see [Configure notifications](/docs/configure-notifications#in-app-preferences).
You hear about a report once each time it closes. Moving the issue from one closed status to another doesn’t notify you again, but reopening and closing it does.
## Send feedback from an AI assistant [#from-an-ai-assistant]
AI assistants connected to Data Hub over [MCP](/docs/mcp) can send feedback for you, for example when a tool call fails. The assistant shows you the draft first and sends it only after you agree. It can also list the reports you sent and read one in full, by report ID or by Linear issue ID such as `ENG-1476`. The tools are listed under [MCP tools](/docs/mcp/tools).
API tokens can’t send feedback, because every report needs a person behind it. Scripts that call `POST /api/v1/feedback` have to use a signed-in session.
## Who sees which reports [#who-sees-which-reports]
Members see only the reports they sent. They can list them through the API or an AI assistant, without the Linear assignee, priority, or activity.
Admins review every report under **Settings → Feedback**:
* **Open** and **Closed** are separate tabs.
* Each row shows the Linear issue ID, status, label, priority, and assignee.
* A report’s detail includes the Linear status changes and comments, and an **Open in Linear** link.
Status changes happen in Linear. Data Hub has no control for changing a report’s status. Admins can also turn on **New feedback** under **Settings → Notifications** to hear when someone sends a report.
## See what’s new in Data Hub [#changelog]
Choose **Changelog** in the account menu to see what has changed in Data Hub, newest first. It opens on top of the page you’re on. Each entry shows its title, and you select a title to read the entry. Entries you haven’t seen before are marked **New** and start open. If none are new, or every entry is new, only the newest entry starts open. A dot on your account button means there’s an entry you haven’t seen yet, and opening the Changelog clears it. Many entries link to the page on this site that covers the change.
To share an entry, open it and select **Copy link**. Anyone signed in to your Data Hub who opens the link sees the Changelog with that entry open. Someone who isn’t signed in has to sign in first and then open the link again. An AI assistant connected to Data Hub [gives the same links](/docs/mcp#questions-about-data-hub) when it tells you what changed.
# Set up an instrument (/docs/set-up-an-instrument)
**For:** Lab operators and admins
This walkthrough takes you from nothing to an uploading instrument: create an access token, install the watcher, register the instrument, get an admin to confirm it, then start watching. CLI flags, the full config schema, and Windows service setup each have their own page; this guide links them once.
Data Hub is self-hosted. The web app, PostgreSQL database, and AWS stack (S3 plus the processing Lambda) must already be running. If they are not, an engineer needs the [Deploy Data Hub](/docs/self-hosting) guide first.
## Before you start [#before-you-start]
You need:
* **The watch directory**: the folder your instrument writes output into.
* **File patterns**: globs to upload, for example `*.csv`, `*.xlsx`, `*.tiff`.
* **[uv](https://docs.astral.sh/uv/getting-started/installation/)** installed on the instrument PC (it manages Python for you).
To make setup and maintenance easier, we recommend setting up remote access to the instrument PC via [Windows Remote Desktop](https://learn.microsoft.com/en-us/windows-server/remote/remote-desktop-services/remotepc/remote-desktop-allow-access).
## 1. Create an access token [#1-create-an-access-token]
The watcher authenticates to Data Hub with a personal access token. Only workspace admins can create tokens, so ask an admin if you are not one.
In the Data Hub web app, go to **Settings → Access Tokens** and click **Create token**.
Give the token a **Name** that identifies the instrument PC, for example `Plate Reader PC`.
Set **Expiration** to “No expiry”. (The token can always be revoked if necessary.)
Under **Scopes**, click the **Watcher** preset. It grants only what a watcher needs: registering, reporting runs, and uploading files.
Click **Create token**, then copy the token. It starts with `dhub_` and is shown only once.
Keep the token handy: you paste it into the setup wizard in step 3. For rotation, revocation, and the full scope list, see [Issue and revoke tokens](/docs/manage-tokens).
## 2. Install the watcher [#2-install-the-watcher]
On the instrument PC, install the watcher with `uv`. The `windows-service` extra adds `pywin32` so you can follow [Run as a Windows service](/docs/windows-service), which is the default on a lab PC:
```sh
uv tool install "data-hub-watcher[windows-service]"
```
On macOS or Linux, omit the extra:
```sh
uv tool install data-hub-watcher
```
Full install, verify, and update steps live in [Install and update the watcher](/docs/install-the-watcher).
## 3. Run the setup wizard [#3-run-the-setup-wizard]
Start the wizard with `data-hub-watcher init`. It prompts you for one thing at a time:
Choose the environment. For a live lab, this is production.
Enter your Data Hub API URL e.g. `https://datahub.your-domain.com/api/v1`.
Paste the token from step 1 when it asks for the API key. The wizard checks the key, then lists the instruments already registered.
Pick your instrument from the list, or choose **Register a new instrument** if it is not there.
For a new instrument, enter an ID and a display name. The ID is a short, lowercase, hyphenated name like `bio-rad-cfx96`. **It cannot be changed later.**
Enter the full path to the folder your instrument writes into.
List which files to upload, such as `*.pcrd,*.csv`. The watcher ignores files that do not match.
Choose how files get grouped into runs. Pick the preset that matches how your instrument names its output, or write your own. The wizard previews a few matches so you can check the run IDs. For example, the default preset uses the filename prefix before the first underscore, so `RUN001_quant.csv` and `RUN001_melt.csv` both group into run `RUN001`.
Set the stability period. This is how much time (in seconds) a file must stay unchanged before the watcher counts it as finished. The default is 5 seconds; raise it for instruments that write large files slowly.
Choose the upload mode. “Auto” uploads each file as soon as it is stable; “Manual” waits for your approval first.
For run detection options, see [Configure run detection and uploads](/docs/configure-run-detection); [Watcher configuration](/docs/configuration-reference#run-detection) lists every preset.
The wizard then registers the watcher, saves the config to `~/.data-hub/config.yaml` (`C:\Users\\data-hub\config.yaml` on Windows), and syncs it to Data Hub. When it reports that setup is complete, start watching.
## 4. Confirm the instrument’s registration [#4-confirm-the-instruments-registration]
A pending instrument cannot upload to Data Hub until an admin confirms it from the web app.
Open the Data Hub web app and navigate to **Instruments > View all** from the app sidebar.
Select the “Pending” tab and find your instrument. It should show a yellow `pending` badge.
Click **Confirm**. The instrument should move to the “Active” tab and appear as online shortly.
Open the instrument’s page and check the watched folder and file patterns shown under its name. They should match what you entered in the wizard; files outside that folder, or with names that don’t match a pattern, won’t reach Data Hub.
Some instruments also get automated preprocessing (metadata extraction, image conversion, and related outputs). To choose a processor, an admin opens the three-dot menu, selects **Edit details**, and picks a supported instrument type. What each type produces is in [Instrument data preprocessing](/docs/instrument-preprocessing).
Adding a new processor requires code changes; see [Adding a new instrument](https://github.com/Arcadia-Science/data-hub/blob/production/docs/developer/lambda.md#adding-a-new-instrument) in the developer docs.
See [Manage instruments](/docs/manage-instruments) for more info.
## 5. Start the watcher service [#5-start-the-watcher-service]
First, preview what the watcher will do without starting it:
```sh
data-hub-watcher watch --dry-run
```
This validates config, checks that the API is reachable and the instrument is active, and previews which files would be picked up.
When it looks right, install and start the watcher as a Windows service. Run these from an **Administrator** terminal:
```sh
data-hub-watcher service install
data-hub-watcher service start
```
The service starts on boot, keeps running after you log out, and auto-updates in the background. Lifecycle and log details are in [Run as a Windows service](/docs/windows-service).
On macOS or Linux, run the watcher in the foreground instead:
```sh
data-hub-watcher watch
```
The watcher detects new files, groups them into runs, uploads them (in auto mode), and sends a heartbeat every 60 seconds. Runs appear in the dashboard after upload.
## What you get once it is running [#what-you-get-once-it-is-running]
* Raw files upload to cloud storage
* Instrument runs and files appear in the web dashboard
* Watcher service automatically restarts on PC restart
* Watcher health (heartbeats, events) appears under **Watchers**
* Subscribed users can get in-app and Slack notifications for new runs
# Troubleshoot a watcher (/docs/troubleshoot-a-watcher)
**For:** Lab operators and admins
Find your symptom below. When you are stuck, the [logs](#where-are-the-logs) are the fastest way in. Admins can also inspect heartbeats and events under **Watchers** in the web app.
## Setup and registration [#setup-and-registration]
### “Instrument is still pending activation” [#instrument-is-still-pending-activation]
The instrument was registered but an admin has not confirmed it. Ask an admin to open **Instruments** and click **Confirm**. See [Manage instruments](/docs/manage-instruments).
### “Instrument already has an active watcher” [#instrument-already-has-an-active-watcher]
Each instrument can have at most one active watcher. An earlier install (often on another PC, or before a reimage) is still registered. The CLI prints the existing watcher id; deregister it before re-running `init`:
* **Web app**: **Watchers**, open the existing watcher, click **Deregister**
* **API**: `DELETE /api/v1/watchers/` with a bearer token that has `watchers:admin`
Deregistration is a soft-delete: history stays under **Watchers → Deregistered**.
### `init` rejects my API key [#init-rejects-my-api-key]
The wizard rejects values that do not start with `dhub_`, and strips zero-width and non-breaking-space characters that Windows clipboards inject. If a paste keeps failing, pass `--show-key` so you can see what the prompt received.
## The watcher will not start [#the-watcher-will-not-start]
* Run `data-hub-watcher config show` to confirm config and environment
* Run `data-hub-watcher watch --dry-run`: it prints the first failing check
* Confirm the instrument is `active`, not `pending`
* On Windows, if the **service** will not start, see [Windows service](#windows-service)
## The watcher shows Offline right after a restart [#the-watcher-shows-offline-right-after-a-restart]
The watcher checks in once a minute, so a watcher that was just restarted can take up to a minute to show as **Online**. Refresh the page after a minute. If it still shows **Offline** or **Unresponsive**, check that the process is running and see [“Connection error” or “Request timed out”](#connection-error-or-request-timed-out).
## Heartbeats fail with `426 Upgrade Required` [#heartbeats-fail-with-426-upgrade-required]
The watcher is older than the **Minimum supported version** an admin set under **Settings → Watchers**, so the server rejects its heartbeats and it shows as unresponsive. It keeps checking for updates every hour and upgrades itself if **Latest version** is newer. To upgrade now, run `data-hub-watcher self-update`. See [Roll out watcher releases](/docs/watcher-releases#release-config-fields).
## “Connection error” or “Request timed out” [#connection-error-or-request-timed-out]
The watcher cannot reach the Data Hub API. Check:
* Network connectivity from the instrument PC
* Environment in config (`staging`, `production`, or `preview`): `data-hub-watcher config show`
* That the API base URL for that environment is reachable
### A run is missing after a connection problem [#a-run-is-missing-after-a-connection-problem]
Watcher 1.2.2 and later retry a run report that failed because the watcher couldn’t reach Data Hub, or because Data Hub returned a server error. The first retry comes after 5 seconds. The wait then doubles up to 5 minutes between tries, and the watcher keeps trying every 5 minutes until the report goes through. It needs no new file and no restart, so the run appears on its own once the problem clears.
The watcher records one `run_report_failed` event when a run’s report first fails. It doesn’t record another for the same run while it keeps retrying, even if more files in the run finish in the meantime.
A failure that isn’t a connection problem or a server error, such as `401 Unauthorized`, is not retried, because it would fail the same way. Each one is recorded. Fix the cause, then restart the watcher.
Older watchers try again only when another file in the run finishes or the watcher restarts. Restart the watcher to report the run.
## Files are not being detected [#files-are-not-being-detected]
* Confirm the watch directory and file patterns. The instrument’s page in the web app shows both, or run `data-hub-watcher config show` on the PC
* Check that the instrument saves its files into that folder, not a shared network drive or another folder
* Check file patterns. Globs are case-sensitive on some systems (`*.csv` does not match `data.CSV`)
* If files live in subfolders, set run detection to recursive; see [Configure run detection and uploads](/docs/configure-run-detection)
* Run `data-hub-watcher watch --dry-run` to preview matches
* Look for `pattern_mismatch` events on the dashboard
## Files are detected but not uploading [#files-are-detected-but-not-uploading]
* In **manual** mode, files wait for approval in the upload queue; see [Manage the watcher fleet](/docs/manage-watchers#approving-manual-uploads)
* Check the log for `upload_failed`
* A revoked or expired token surfaces as `401 Unauthorized`. Create a new token and re-run `init`; see [Issue and revoke tokens](/docs/manage-tokens#after-revoking-a-token)
### A requested upload stays on Uploading [#a-requested-upload-stays-on-uploading]
On an instrument in **auto** mode, **Request upload** needs watcher 1.2.1 or later. Older watchers in auto mode never read the upload queue, so the file shows **Uploading** and never arrives. Update the watcher; see [Roll out watcher releases](/docs/watcher-releases#what-each-version-adds).
### Files larger than 5 GB don’t upload [#files-larger-than-5-gb-dont-upload]
Watchers older than 1.2.0 can’t upload files larger than 5 GB, and Data Hub refuses the upload instead of letting it fail partway. Update the watcher to 1.2.0 or later, which uploads large files in parts.
### An upload was cut off by a restart or an update [#an-upload-was-cut-off-by-a-restart-or-an-update]
Watcher 1.2.1 and later retry an automatic upload that didn’t finish the next time the watcher starts, on up to three starts. Two events tell you when it stops trying:
* `upload_retries_exhausted`: the upload still failed after three starts. Choose **Upload** on the file’s run page to try again.
* `pending_upload_missing`: the file was no longer on disk when the watcher started, so there was nothing to retry.
Older watchers don’t retry; use **Upload** on the run page instead.
### Files with the same name were skipped [#files-with-the-same-name-were-skipped]
When a run gathers files from several folders, watchers older than 1.1.0 uploaded only the first file with a given name and skipped the rest. Watcher 1.1.0 and later keep every copy, adding a short tag to later names. To upload files an older watcher skipped:
Update the watcher to 1.1.0 or later.
Stop the watcher (`data-hub-watcher service stop`, or `Ctrl+C` in the foreground).
Run `data-hub-watcher state forget --prefix your_folder_name/` on the instrument PC, where `your_folder_name/` is the folder under the watch directory that held the skipped files.
Start the watcher again. It reports the files in that folder again and uploads the ones Data Hub doesn’t have.
## Auto-update never fires [#auto-update-never-fires]
Check, in order:
* Prefer a long-running service (or scheduled) install; a one-off console session may not stay up for the hourly tick
* Environment is not `preview`: auto-update is hard-disabled there
* Activity-window guard is not holding it up: run `data-hub-watcher self-update`, or ask an admin to flag the release **mandatory**
* **Last Heartbeat** on the dashboard is recent; a stale watcher is not ticking
Upgrade failure modes (`update_started` → `update_failed`, editable installs, missing Scheduled Task) are covered in [Install and update the watcher](/docs/install-the-watcher#manual-update-self-update) and [Roll out watcher releases](/docs/watcher-releases).
## Windows service [#windows-service]
### The service exits immediately [#the-service-exits-immediately]
If it stops before writing to `watcher.log`:
* Check `C:\ProgramData\DataHubWatcher\service-bootstrap.log` for a pre-dispatcher crash (missing `pywin32`, moved venv, corrupt install)
* Query the Windows Application event log:
```powershell
Get-WinEvent -FilterHashtable @{LogName='Application'; StartTime=(Get-Date).AddHours(-1)} |
Where-Object { $_.ProviderName -match 'Python|DataHubWatcher' -or $_.Message -match 'DataHubWatcher' } |
Format-List TimeCreated, ProviderName, Id, LevelDisplayName, Message
```
* Run the service in the foreground to see the traceback:
```powershell
& "C:\path\to\venv\Scripts\python.exe" -m win32serviceutil debug DataHubWatcher
```
### `self-update` says “Upgrade dispatched” but nothing happens [#self-update-says-upgrade-dispatched-but-nothing-happens]
On Windows uv-tool installs, the upgrade runs out of process under SYSTEM and takes 30–60s. Track it:
```powershell
Get-Service data-hub-watcher
Get-Content ~/.data-hub/upgrade-worker.log -Tail 20 -Wait
```
### Turn on debug logging for the service [#turn-on-debug-logging-for-the-service]
Add `DATA_HUB_WATCHER_LOG_LEVEL=DEBUG` to the env file the service loads (typically `~/.data-hub/.env.`) and restart the service. No reinstall needed.
## Where are the logs? [#where-are-the-logs]
Rotating logs (10 MB each, 5 backups). CLI `watch` and the Windows service share one file:
| Platform | Log file |
| ------------- | ------------------------------------------- |
| Windows | `C:\ProgramData\DataHubWatcher\watcher.log` |
| macOS / Linux | `~/.data-hub/watcher.log` |
Add `--verbose` for debug console output: `data-hub-watcher --verbose watch`. Windows uv-tool upgrade output lands in `~/.data-hub/upgrade-worker.log`.
## API clients get `401 Unauthorized` [#api-clients-get-401-unauthorized]
The bearer token is missing, mistyped, revoked, or expired. Check **Settings → Access Tokens** and re-issue if needed. See [Issue and revoke tokens](/docs/manage-tokens).
A script that calls the watcher or processing routes with a browser session, instead of a token, also gets `401`. Those routes accept only personal access tokens; see [How a token acts](/docs/manage-tokens#how-a-token-acts).
## MCP clients get `401 Unauthorized` [#mcp-clients-get-401-unauthorized]
MCP doesn’t accept personal access tokens, so a client configured with `Authorization: Bearer dhub_…` gets a 401 on every call. Remove the header, leave only the URL, and let the client run its sign-in flow. Setup per client is in [MCP overview](/docs/mcp#connect-a-client).
An already-connected client that starts failing has usually lost its grant: sign in again from the client. If reads work and one tool fails with a missing-scope message, the client holds `read` but not `write`; reconnect and approve both.
## Ask an agent to diagnose it [#ask-an-agent-to-diagnose-it]
Use the `troubleshoot_instrument` MCP prompt under [MCP prompts](/docs/mcp/prompts).
# Roll out watcher releases (/docs/watcher-releases)
**For:** Administrators
Lab PCs upgrade themselves toward the version you advertise per environment. You control that from **Settings → Watchers**. How an individual PC installs or runs `self-update` is in [Install and update the watcher](/docs/install-the-watcher).
## How fleet upgrades reach a PC [#how-fleet-upgrades-reach-a-pc]
The in-process updater runs from the heartbeat tick on every platform (not only Windows services). Roughly once an hour the watcher:
1. Asks the server for the latest version and compares it to its own
2. Upgrades only if a newer version is available, no files have uploaded for several heartbeats, and no run was reported recently (the activity-window guard). Releases flagged **mandatory** skip this guard
3. Installs the new wheel and restarts into it (Windows services restart via failure-actions policy)
4. Emits `update_started`, then `update_succeeded` or `update_failed`, visible on the dashboard
Auto-update is disabled in the `preview` environment so pull-request preview deployments cannot push code to lab PCs.
## Cut a new release [#cut-a-new-release]
Releases are tag-driven and publish from a `production` commit.
Bump the version in `watcher/pyproject.toml` and merge through `staging` to `production`.
Tag and push from `production`:
```sh
git checkout production && git pull
git tag watcher-v1.3.0 && git push origin watcher-v1.3.0
```
Approve the `pypi` deployment under **Actions → Publish watcher** in GitHub.
In **Settings → Watchers**, pick the new version under **Latest version**. The list comes straight from PyPI, so a release shows up as soon as PyPI publishes it. If PyPI can’t be reached, the field accepts a typed version instead. Each environment has its own database, so set `staging` and `production` separately to roll the fleet gradually.
Always tag → publish → verify on PyPI → save. Saving **Latest version** before the wheel is live triggers a wave of `update_failed` events across the fleet.
## Release settings [#release-config-fields]
| Field | Purpose |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Latest version** | The version watchers upgrade to. Leave it blank to pause self-updates. |
| **Minimum supported version** | Optional. A watcher older than this has its heartbeats rejected with `426 Upgrade Required`, so it shows as unresponsive until it updates. It still checks for updates every hour, so it upgrades itself as long as **Latest version** is newer. Leave it blank for no minimum. |
| **Mandatory update** | Skips the activity-window guard; see below. Has no effect while **Latest version** is blank. |
## Mandatory updates [#mandatory-updates]
Toggling **Mandatory update** skips the activity-window guard and fires the upgrade on the next hourly check. The server still compares the running version to **Latest version**, so a correctly pinned PC is not forced past its pin.
Use sparingly: a forced upgrade mid-acquisition can lose data. Reserve it for security fixes, wire-protocol breaks, or cases where leaving the bad version running is worse than restarting in flight.
## Roll back [#roll-back]
Rollback is another release: set **Latest version** back to the older tag for the affected environment(s), turn **Mandatory update** on if you need to bypass the activity-window guard, and wait for the next hourly tick. Turn **Mandatory update** off once the fleet converges. The older wheel remains on PyPI; it is no longer advertised.
If **Minimum supported version** is set above the version you roll back to, lower it first. Otherwise the rolled-back watchers can’t check in.
## What each version adds [#what-each-version-adds]
Some Data Hub features need a minimum watcher version. Set **Latest version** to at least the version below to turn a feature on across the fleet.
| Version | What it adds |
| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1.2.2 | A run report that fails because the watcher can’t reach Data Hub, or because Data Hub returns a server error, is retried until it goes through. Older watchers leave the run unreported until another file in the run finishes or the watcher restarts. |
| 1.2.1 | **Request upload** works on instruments that upload automatically; older watchers in auto mode leave the file showing **Uploading**. The watcher also retries an automatic upload that didn’t finish, such as one cut off by a restart or an update, the next time it starts. It tries on up to three starts. |
| 1.2.0 | Files of 100 MB or more upload in 64 MB parts, and a failed part is retried on its own. Files larger than 5 GB can upload; older watchers are refused for files that large. |
| 1.1.0 | When a run gathers files from several folders, files with the same name are all kept instead of only the first. Later copies get a short tag in the name, such as `sample~3f9a1c2b.tif`. Adds the `data-hub-watcher state forget` command. |
After upgrading to 1.1.0 or later, files that an older watcher skipped because of a duplicate name aren’t uploaded on their own. To upload them, stop the watcher, run `data-hub-watcher state forget --prefix your_folder_name/` on the instrument PC, and start the watcher again. See [Troubleshoot a watcher](/docs/troubleshoot-a-watcher#files-with-the-same-name-were-skipped).
## Operator-side failures [#operator-side-failures]
If `update_started` is followed by `update_failed`, or auto-update never fires, point operators at [Troubleshoot a watcher](/docs/troubleshoot-a-watcher#auto-update-never-fires) and [Install and update the watcher](/docs/install-the-watcher#manual-update-self-update).
# Run as a Windows service (/docs/windows-service)
**For:** Lab operators
On a lab instrument PC, run the watcher as a Windows service. This is the default setup: the service starts on boot, keeps running after you log out, restarts on failure, and participates in [fleet auto-update](/docs/watcher-releases). Foreground `data-hub-watcher watch` is for quick checks and non-Windows hosts.
## Install the service extra [#install-the-service-extra]
The service depends on `pywin32`, which ships in the `windows-service` extra. The [default install](/docs/install-the-watcher#install) already includes it. If you installed without it, add it now:
```sh
uv tool install "data-hub-watcher[windows-service]"
```
Run [`data-hub-watcher init`](/docs/install-the-watcher#configure) first if you have not configured the watcher yet. The service uses the same `~/.data-hub/config.yaml` and per-environment `.env` file.
## Install and start [#install-and-start]
From an **Administrator** terminal:
```sh
data-hub-watcher service install
data-hub-watcher service start
```
`service install` registers delayed auto-start (so it does not race the network stack at boot), configures restart-on-failure recovery, and registers the `DataHubWatcherUpgrade` Scheduled Task used by the auto-updater.
The Windows auto-update path requires that `service install` (or `service reinstall`) has been run from an Administrator shell at least once. Fleet PCs upgrading into an auto-update-capable build for the first time need a one-time `service reinstall` to pick up the task.
## Lifecycle commands [#lifecycle-commands]
| Command | What it does |
| ------------------------------------ | --------------------------------------------- |
| `data-hub-watcher service install` | Register the service |
| `data-hub-watcher service start` | Start the service |
| `data-hub-watcher service stop` | Stop the service |
| `data-hub-watcher service status` | Show whether the service is running |
| `data-hub-watcher service uninstall` | Remove the service |
| `data-hub-watcher service reinstall` | Stop, uninstall, install, and start in one go |
`service reinstall` is the right command after an out-of-band wheel swap (for example a manual `uv tool install -U` from an Administrator shell). Stop and uninstall are best-effort, so it works even if the service is already gone.
`service install` and `service reinstall` accept `--env-path PATH` to override which `.env` file the service loads (defaults to `~/.data-hub/.env.`).
## Do not run `watch` alongside the service [#do-not-run-watch-alongside-the-service]
Running `data-hub-watcher watch` interactively while the service is also running is not supported: both processes would race on the same log file and watch directory. Stop the service first if you need the CLI for debugging:
```powershell
data-hub-watcher service stop
data-hub-watcher --verbose watch
data-hub-watcher service start
```
## Where the service writes logs [#where-the-service-writes-logs]
The service and the CLI share one rotating log:
* `C:\ProgramData\DataHubWatcher\watcher.log`
A separate `service-bootstrap.log` next to it captures crashes before the service control dispatcher takes over (missing `pywin32`, moved venv, corrupt install). To turn on debug logging without a reinstall, add `DATA_HUB_WATCHER_LOG_LEVEL=DEBUG` to the env file the service loads and restart the service.
# Overview (/docs/api)
**For:** Developers and integrators
The Data Hub API is served by the web application at `/api/v1/`. The watcher, Lambda, and dashboard all call it. Data Hub is self-hosted, so the base URL is your deployment host plus `/api/v1`. Examples use `https://datahub.example.com`; substitute your host.
Per-endpoint request and response shapes live in the generated pages under this section (same sidebar). Each deployment also publishes OpenAPI 3.1 at `/api/v1/openapi.json`.
## Authentication [#authentication]
Two methods (full model in [Security and permissions](/docs/security)):
* **Session cookies**: web dashboard (Google OAuth). Session callers implicitly hold every scope.
* **Bearer tokens**: `Authorization: Bearer dhub_…` for the watcher, the Lambda, and scripts
```sh
curl https://datahub.example.com/api/v1/instruments \
-H "Authorization: Bearer dhub_your_token_here"
```
Every token request needs a [scope](/docs/security#token-scopes). A missing scope returns `403 FORBIDDEN`. Some mutations also require the admin role; see [Admin-gated operations](/docs/security#admin-gated-operations).
Some routes accept only one of the two methods:
* The routes the watcher and the Lambda call accept only tokens, and return `401 UNAUTHORIZED` to a session. These cover watcher registration and heartbeats, upload links, file updates, run creation and updates, and instrument creation.
* Claiming runs, sending feedback, reviewing feedback, and managing tokens need a signed-in person, so tokens can’t call them.
Mint tokens in the UI: [Issue and revoke tokens](/docs/manage-tokens).
## First request [#first-request]
List instruments:
```sh
curl https://datahub.example.com/api/v1/instruments \
-H "Authorization: Bearer dhub_your_token_here"
```
Fetch the OpenAPI document (no auth required):
```sh
curl https://datahub.example.com/api/v1/openapi.json
```
The document covers the integrator surface: instruments, runs, comments, files, watchers, archive jobs, search, and `POST /notifications/dispatch`. Routes that only the dashboard uses are left out: `/tokens`, `/users`, `/feedback`, `/settings/*`, and the notification inbox. The MCP transport is documented separately.
## Actions taken by a token [#actions-taken-by-a-token]
A token acts under its own name, not as a person, so responses name the token when it did something:
* A comment posted by a token has `user: null` and a `token` object with the token’s `id` and `name`. A comment posted by a person has `token: null`.
* A run deleted by a token has `deleted_by_token` set, alongside `deleted_at`.
Read both fields when you display who did something. See [How a token acts](/docs/manage-tokens#how-a-token-acts).
## Send notifications [#send-notifications]
`POST /notifications/dispatch` sends a short message to chosen people, with an optional link to a run. It needs a token with the `notifications:create` scope. Each person’s **Service notifications** setting decides whether the message appears in the notification bell, as a Slack direct message, or both. The message shows the token’s name as its sender. Request fields are on the generated [dispatch notifications](/docs/api/notifications/dispatchNotifications) page.
## Pagination and filtering [#pagination-and-filtering]
List endpoints that return pages (notably run search) accept:
* `page`: 1-based page index
* `per_page`: page size, 1 to 100
Responses include pagination metadata such as `page`, `per_page`, `total`, and `total_pages`.
Run report items page by index instead, with `offset` and `limit` (max 200). Their responses return `offset`, `limit`, and `total`.
Run search also supports filters (instrument, date range, attribution via `ranBy`, and instrument-specific metadata). Exact query parameters are on each generated operation page in the sidebar under this **API** section.
## Error responses [#error-responses]
Errors use a nested envelope:
```json
{
"error": {
"code": "NOT_FOUND",
"message": "Instrument not found.",
"details": null
}
}
```
Common codes: `VALIDATION_ERROR` (400), `UNAUTHORIZED` (401), `FORBIDDEN` (403), `NOT_FOUND` (404), `CONFLICT` (409). Scope failures use `code: "FORBIDDEN"` with a message naming the missing scope.
Feedback routes have two more. `LINEAR_UNAVAILABLE` (502) means Data Hub couldn’t reach Linear, where reports are stored. `FEEDBACK_NOT_CONFIGURED` (503) means no admin has connected Linear yet; see [Set up integrations](/docs/integrations#linear).
## MCP [#mcp]
`GET` and `POST` `/mcp/v1` expose the MCP server over Streamable HTTP, outside the `/api/v1` tree. Clients authenticate with OAuth rather than a personal access token, and hold `read` to connect plus `write` to call any tool that changes data. See [MCP overview](/docs/mcp).
# Overview (/docs/mcp)
**For:** Developers and integrators
Data Hub exposes a Model Context Protocol (MCP) server at `/mcp/v1` so AI clients can query instruments, runs, files, and watcher health. Clients authenticate by sending you through a browser sign-in.
Catalogs for [tools](/docs/mcp/tools), [prompts](/docs/mcp/prompts), and [resources](/docs/mcp/resources) are generated from the live server schema, so they match what your client discovers when it connects.
## Authentication [#authentication]
The MCP server authenticates with OAuth. All you need to provide is the endpoint URL; for example, `https://datahub.example.com/mcp/v1`.
The first connection opens your browser:
Sign in to Data Hub with Google, unless you already have a session open.
Read the **Authorize application** screen. It names the client, the host it redirects to, and the access it’s asking for.
Click **Allow**.
Your client stores the access token it receives and renews it without asking again. **Allow** grants everything on the screen: there are no per-permission checkboxes, so click **Deny** if any line looks wrong.
### Scopes [#scopes]
MCP uses two scopes rather than the `:` scopes that gate REST requests:
* **`read`**: required to connect at all, and covers every read-only tool, resource, and prompt
* **`write`**: additionally required by tools that change something, including claiming runs, commenting, requesting uploads, reprocessing, and deleting
Most clients ask for both, because the server advertises both. A client granted only `read` still connects and still reads; its first mutating tool call fails with a message naming the missing scope.
Each entry in the [tools catalog](/docs/mcp/tools) lists the scope its REST equivalent requires, which tells you what data the tool touches. Over MCP, only `read` and `write` are enforced.
The client acts as you, with your role. Admin-only actions, such as deleting someone else’s comment, work over MCP only when you are an admin. Claiming runs and sending feedback work because a signed-in person is behind the client, even though API tokens can’t do either.
## Connect a client [#connect-a-client]
Examples use `https://datahub.example.com`; substitute your deployment host.
### Claude Code [#claude-code]
Register the server with [`claude mcp add`](https://code.claude.com/docs/en/mcp-quickstart). Run this in your terminal, not inside a `claude` session:
```bash
claude mcp add --transport http --scope user data-hub https://datahub.example.com/mcp/v1
```
Confirm that the server was added by running `claude mcp list`. Then, start a session with `claude` and run `/mcp` to authenticate.
### Claude Desktop [#claude-desktop]
Remote servers go through **Settings → Connectors → Add custom connector**. The `claude_desktop_config.json` file handles local servers only. Paste `https://datahub.example.com/mcp/v1` as the URL, click **Add**, then click **Connect** to sign in. Leave the OAuth client fields under **Advanced settings** empty, since Data Hub registers clients on demand.
Claude reaches your server from Anthropic’s infrastructure rather than from your own machine, so the deployment has to be reachable from the public internet. A deployment running on `localhost` can’t be added this way; use Claude Code or Cursor for local work.
### Cursor [#cursor]
Add the server to `.cursor/mcp.json` in your project, or to `~/.cursor/mcp.json` for every project:
```json
{
"mcpServers": {
"data-hub": {
"url": "https://datahub.example.com/mcp/v1"
}
}
}
```
Open **Settings → Tools & MCP**. The server appears with a **Needs authentication** label; click it to start the browser flow. The tools show up once you approve the grant.
### Other clients [#other-clients]
Any client that speaks Streamable HTTP and the MCP authorization spec needs one setting, the URL `https://datahub.example.com/mcp/v1`. For local development, point at `http://localhost:3000/mcp/v1` instead.
Clients that speak MCP but not OAuth can bridge through [`mcp-remote`](https://www.npmjs.com/package/mcp-remote), which runs the browser flow on their behalf.
## Questions about Data Hub [#questions-about-data-hub]
The assistant can answer questions about Data Hub itself, such as how to install a watcher, what a run status means, or what changed last week. It searches the documentation or the [changelog](/docs/send-feedback#changelog), reads what fits, and links it in its answer. Three tools do this, and all need only the `read` scope:
* `search_docs` returns up to five matching sections of the docs, each with a short excerpt and a link
* `read_doc` returns one docs page, or one section of a page, as Markdown
* `search_changelog` lists changelog entries by date, or finds them by topic, and returns each one in full with a link that opens it in the Changelog
Ask it what changed this week, what’s new since a date, or when something changed. Dates are calendar days, and both ends count. The answers come from the docs and changelog that ship with your deployment, so they match the version you run, including a staging deployment that is ahead of the published site. A change appears in the changelog once it has reached your deployment. Clients that let you attach resources by hand can also attach any docs page as `datahub://docs/`, for example `datahub://docs/manage-tokens`.
The assistant reads the guides and the changelog. It doesn’t read the generated [API](/docs/api) endpoint pages or the [tools](/docs/mcp/tools), [prompts](/docs/mcp/prompts), and [resources](/docs/mcp/resources) catalogs, because the client already receives the catalog when it connects. When the docs don’t cover a question, the assistant says so and offers to [send feedback](#feedback) about the gap.
## Interactive run reports [#interactive-run-reports]
Clients that support MCP Apps, an extension of the Model Context Protocol, show the same interactive run report as the web app when the assistant calls `get_run_report`. The report includes image carousels, plate maps, and the Aunty plate with its well charts. Clients without MCP Apps support get the same data as text and describe it instead.
## Feedback [#feedback]
The assistant can send [feedback about Data Hub](/docs/send-feedback) for you, for example after a tool call fails. It shows you the draft first and sends it only after you agree. Sending feedback needs only the `read` scope. The assistant can also list the reports you sent and read one in full, by report ID or by Linear issue ID such as `ENG-1476`. Feedback tools return an error saying feedback isn’t set up until an admin [connects Linear](/docs/integrations#linear).
Problems with one run’s data belong in a run comment instead, which the assistant can also post.
## Machine-readable catalog [#machine-readable-catalog]
Fetch the same catalog these pages render from:
```bash
curl https://datahub.example.com/mcp/v1/schema.json
```
It lists every tool, prompt, and resource, with JSON Schema for arguments and each tool's response. The schema document needs no authentication.
# Prompts (/docs/mcp/prompts)
**For:** Developers and integrators
Prompts are scripted workflows the client surfaces to the user. Each prompt assembles a multi-step instruction that the model then executes using MCP tools. Argument schemas below match what the server advertises over the protocol.
See [MCP overview](/docs/mcp) for authentication and client setup.
### `daily_summary`
Summarize all instrument activity for a given day, including run counts, failures, and system health.
**Arguments**
- `date` (optional, `string`) — Date to summarize (YYYY-MM-DD, UTC calendar day). Defaults to the current UTC day.
### `troubleshoot_instrument`
Diagnose connectivity or processing issues for an instrument by inspecting its status and watcher health.
**Arguments**
- `instrumentId` (required, `string`) — Instrument identifier to troubleshoot
### `compare_runs`
Compare the results of two runs side by side, highlighting differences in experimental outcomes.
**Arguments**
- `instrumentId` (required, `string`) — Instrument identifier (both runs must be on the same instrument)
- `runId1` (required, `string`) — First run identifier
- `runId2` (required, `string`) — Second run identifier
### `find_my_runs`
List runs claimed by the authenticated user, optionally scoped to an instrument and date range.
**Arguments**
- `instrumentId` (optional, `string`) — Optional instrument to narrow results
- `dateFrom` (optional, `string`) — Start date (YYYY-MM-DD, UTC), inclusive
- `dateTo` (optional, `string`) — End date (YYYY-MM-DD, UTC), inclusive
### `explain_failed_run`
Diagnose why a run failed and suggest reprocess or upload fixes.
**Arguments**
- `instrumentId` (required, `string`) — Instrument identifier
- `runId` (required, `string`) — Run identifier within the instrument
### `claim_unattributed_runs`
Find unattributed runs on an instrument and claim them for the authenticated user after confirmation.
**Arguments**
- `instrumentId` (required, `string`) — Instrument identifier
- `dateFrom` (optional, `string`) — Optional start date (YYYY-MM-DD, UTC)
- `dateTo` (optional, `string`) — Optional end date (YYYY-MM-DD, UTC)
### `summarize_instrument_week`
Summarize one instrument's activity over the last seven days (or a provided window).
**Arguments**
- `instrumentId` (required, `string`) — Instrument identifier
- `dateFrom` (optional, `string`) — Start date (YYYY-MM-DD, UTC). Defaults to 7 days ago (UTC).
- `dateTo` (optional, `string`) — End date (YYYY-MM-DD, UTC). Defaults to today (UTC).
# Resources (/docs/mcp/resources)
**For:** Developers and integrators
Resources are reference context that clients can attach to prompts without an explicit tool call. URI templates (for example filter-options) are listed alongside static URIs.
See [MCP overview](/docs/mcp) for authentication and client setup.
### `instruments`
List of all instrument IDs, display names, and types. Use as reference context when constructing tool calls.
*application/json*
**URI**
`datahub://instruments`
### `me`
Authenticated user's identity (id, name, email, image, isAdmin). Same payload as the get_me tool.
*application/json*
**URI**
`datahub://me`
### `glossary`
Static reference: run status derivation, instrument types, ranBy literals, UTC date semantics, and archive polling.
*application/json*
**URI**
`datahub://glossary`
### `instrument-filter-options`
Available filter values for an instrument. Values map directly to search_runs metadata filter arguments (wavelength/measurementMode/measurementType for plate readers; captureType/imagingMode/gelWavelength/gelColor for gel-doc; dyeChannel for qPCR; hinaChannel/hinaDimension/hinaSize for Hina; dpi/colorMode for Epson; auntyExperimentType/auntyAnalysisMode/auntyTemperature/auntyRampRate for Aunty).
*application/json*
**URI**
`datahub://instruments/{instrumentId}/filter-options`
### `docs-page`
A page of Data Hub's own documentation as Markdown, such as datahub://docs/manage-tokens. Same text as the read_doc tool. Use search_docs to find the right page.
*text/markdown*
**URI**
`datahub://docs/{page}`
### `run-report`
MCP Apps view for a Data Hub instrument run report. Linked from get_run_report.
*text/html;profile=mcp-app*
**URI**
`ui://data-hub/run-report`
# Tools (/docs/mcp/tools)
**For:** Developers and integrators
All tools return JSON encoded as a single text content block. Error cases set `isError: true` and return a plain-text message. Each tool below carries its annotations (`readOnly`, `idempotent`, `destructive`) and the schema of its JSON response.
Connecting requires the `read` scope. Every tool without a `readOnly` annotation also requires `write`. See [MCP overview](/docs/mcp) for authentication and client setup.
## Instruments
### `list_instruments`
List all registered lab instruments with run counts, watcher status, and file patterns. Optionally filter by status.
*readOnly*
**Parameters**
- `status` (optional, `"pending" | "active" | "inactive"`) — Filter instruments by status
**Response**
- `instruments` (`array