tane/docs/design/chat-storage.md

Chat storage — spec

Where 1:1 chat history lives on each device, and why it is a separate encrypted database from the inventory.

Problem

Chat history was persisted as a JSON blob per conversation in the OS keystore (flutter_secure_storage), via MessageStore. That is the wrong tool once a conversation is used:

• Read-modify-write on every message. Each incoming/sent message read the whole conversation JSON, decoded it, checked duplicates in O(n), re-encoded the whole list, and wrote it back — O(n) per message, O(n²) over a conversation.
• Silent data loss. A hard cap of 200 messages per peer dropped the oldest without warning. It was a recent-cache, not storage.
• Keystore misuse. The OS keystore (Android EncryptedSharedPreferences, iOS Keychain) is for small secrets (keys, tokens), not growing blobs.
• Manual index. Peers were tracked in a \n-joined string because the keystore can't enumerate keys; the inbox (conversations()) then read every peer's full blob just to get its last message.

Decision

Move chat history to an encrypted Drift/SQLCipher database, in its own file (tane_chat.sqlite) — ChatDatabase, separate from the inventory AppDatabase (tane_inventory.sqlite). Same SQLCipher key (from the OS keystore), so "no plaintext at rest" holds.

Why a separate database, not a new table in AppDatabase

Chat is not inventory. It is an ephemeral, per-device cache of what the relays delivered; it never enters an inventory snapshot, a .tanemaki backup, or the device-to-device sync. Keeping it out of AppDatabase means:

• It carries no CRDT metadata (no HLC updatedAt, tombstones, lastAuthor, schemaRowVersion) — those are meaningless for a network cache.
• It stays out of the inventory's versioned migration series (no v9 bump, no drift_schema_v9.json, no new migration_test case). ChatDatabase is schemaVersion = 1, created fresh; there is nothing to migrate.
• It never trips the inventory sync stream (AppDatabase.tableUpdates() feeds SyncService); a message arriving can't spuriously republish the inventory.

Schema — Messages

One flat, indexed table (see apps/app_seeds/lib/db/chat_database.dart):

column	notes
id	autoincrement — local rowid, also the stable tie-break for equal timestamps
accountScope	namespaces rows per social identity (was the keystore key prefix)
peerPubkey	the conversation
fromPubkey	sender (self vs peer → bubble side)
body	message text
sentAt	ms since epoch (NIP-17 timestamp) — ordering + read mark

• Index (accountScope, peerPubkey, sentAt) serves history and the inbox.
• Unique key (accountScope, peerPubkey, fromPubkey, sentAt, body) makes de-duplication a DB invariant: a relay re-delivers the same gift wrap on every (re)subscribe, and an identical row simply isn't inserted twice.

Behaviour vs. the old store

MessageStore keeps its public API (history / append / conversations), so MessagesCubit, InboxService, UnreadService and the inbox screen are unchanged. What changes:

• append is a single indexed insert (dedup-checked in a transaction), not a whole-conversation rewrite. Returns true only when newly stored.
• history is uncapped, ordered by sentAt then id (stable insertion tie-break for equal timestamps).
• conversations is one indexed scan; the newest message per peer is picked in Dart (peer counts are small).

Not migrating old data

Pre-release: there are no users, so the old keystore blobs are not migrated. The keystore keys (tane.social.chat.*, tane.social.chats) are simply abandoned. If that changes before release, a one-shot import can be added.

Related, deliberately not changed

The same keystore-blob anti-pattern exists, at trivial scale, in OfferOutbox (pending lot ids), TrustReferents (WoT referent set) and SocialSettings (relay URLs). These are small, bounded, rarely-written lists — the keystore is acceptable there. Only chat grows with everyday use, so only chat moved. If any of those grows unexpectedly, the same Drift treatment applies.