Streams Architecture

Purpose: Lazy transformation and collection of cursor data
Location: src/lib/stream/
Last Updated: 2024-11-26

Overview

The stream system provides functional, composable transformations for cursor data. Streams handle data transformation and collection while cursors handle pagination, creating a clean separation of concerns.

Core Principle: Streams transform data through composable generator functions with lazy evaluation.

Architecture

Separation from Cursors

Stream Responsibilities:

• Lazy data transformations (map, filter, flatMap)
• Data collection (collect, toArray, reduce)
• Functional composition via generators
• No pagination logic

Cursor Responsibilities:

• See Cursors Documentation

Source Code

File	Purpose	Source
Base stream implementation	Stream transformations	src/lib/stream/base-stream.ts
Transform composition	Functional composition	src/lib/stream/transform.ts
Collection strategies	Collectors	src/lib/stream/collectors.ts
TypeScript interfaces	Stream and transform types	src/lib/stream/types.ts
Public exports	Module exports	src/lib/stream/index.ts

Core Interfaces

For complete type definitions, see src/lib/stream/types.ts.

Stream Interface

interface Stream<T> extends AsyncIterable<T> {
  // Transformations
  map<U>(fn: (item: T) => U | Promise<U>): Stream<U>;
  filter(fn: (item: T) => boolean | Promise<boolean>): Stream<T>;
  flatMap<U>(fn: (item: T) => AsyncIterable<U>): Stream<U>;
  take(n: number): Stream<T>;
  skip(n: number): Stream<T>;
  
  // Collectors
  collect(): Promise<T[]>;
  reduce<U>(fn: (acc: U, item: T) => U, initial: U): Promise<U>;
  forEach(fn: (item: T) => void | Promise<void>): Promise<void>;
}

Transform Type

type Transform<A, B> = (source: AsyncIterable<A>) => AsyncIterable<B>;

Documentation Structure

• Usage Patterns - Common stream usage examples
• Transformations - Built-in and custom transformations
• Best Practices - Guidelines and anti-patterns
• Streams vs fp-ts - How streams and fp-ts work together

Design Principles

1. Functional Composition

Streams compose transformations without creating intermediate collections.

2. Lazy Evaluation

Transformations build a pipeline without executing until collection.

3. Generator-Based

All transformations use async generators for natural lazy evaluation and memory efficiency.

4. Type Safety

Full TypeScript generics ensure type safety through transformation chains.

Related Documentation

• Cursors Architecture - Pagination and cursor tokens
• Database Cursor Patterns - PostgreSQL implementation
• Forge Home: Cursor/Stream Separation - Implementation plan

Future Enhancements

• Parallel stream processing
• Stream backpressure control
• Stream error recovery strategies
• Performance monitoring and metrics

---

Next steps: See Usage Patterns for examples, or Transformations for available operations.