Cursors Architecture

Purpose: Abstract cursor pattern for stateless pagination
Location: src/lib/cursor/
Last Updated: 2024-11-26

Overview

The cursor system provides an abstract, implementation-agnostic pattern for token-based pagination. This architecture separates pagination concerns (Cursor) from transformation concerns (Stream), enabling clean composition and reuse across different data sources.

Core Principle: Cursors handle pagination state; Streams handle data transformation.

Architecture

Separation of Concerns

Cursor

• Token-based pagination only
• Manages position in result set
• Provides AsyncIterable<T> interface
• No transformation logic

Stream

• Lazy transformations and collection
• Functional composition via generators
• Built on top of cursors
• See Streams Documentation

Source Code

File	Purpose	Source
Abstract cursor implementation	Base cursor class	src/lib/cursor/abstract-cursor.ts
TypeScript interfaces	Cursor and token types	src/lib/cursor/types.ts
Public exports	Module exports	src/lib/cursor/index.ts

Core Interfaces

For the complete type definitions, see src/lib/cursor/types.ts.

Cursor Interface

interface Cursor<T> extends AsyncIterable<T> {
  fetch(limit?: number): Promise<T[]>;
  hasMore(): Promise<boolean>;
  getToken(): string | null;
  stream(): Stream<T>;
}

CursorToken

interface CursorToken {
  lastId?: bigint;
  lastValue?: unknown;
  offset?: number;
  direction: 'forward' | 'backward';
  limit: number;
  timestamp: number;
  version: number;
}

Documentation Structure

• Implementation Guide - How to implement custom cursors
• Usage Patterns - Common usage examples
• Best Practices - Guidelines and anti-patterns
• fp-ts Integration - How cursors work with fp-ts

Implementations

Database Implementation

Location: src/lib/db/api/ (consider renaming to src/lib/db/cursor/)
Documentation: Database Cursor Patterns

Database-specific cursor implementation for PostgreSQL queries.

Storage Implementation

Location: src/lib/storage/ (if exists)
Documentation: TBD

File system or object storage cursor implementation.

Design Principles

1. Stateless Tokens

Cursors use encoded tokens that contain all state needed to resume pagination. No server-side session required.

2. Lazy Evaluation

Cursors fetch data on-demand via AsyncIterable protocol. No upfront loading.

3. Composability

Cursors integrate seamlessly with streams for transformation pipelines:

cursor.stream().map(...).filter(...).collect()

4. Type Safety

Full TypeScript support with generic types for data items.

Related Documentation

• Streams Architecture - Transform and collect cursor data
• Database Cursor Patterns - PostgreSQL implementation
• Forge Home: Cursor/Stream Separation - Implementation plan

Future Enhancements

• Bidirectional cursors (backward pagination)
• Cursor expiration and validation
• Performance monitoring and metrics
• Cursor serialization formats (JSON, protobuf)

---

Next steps: See Implementation Guide for creating custom cursors, or Usage Patterns for examples.