Stream Transformations

Purpose: Built-in and custom stream transformations
Last Updated: 2024-11-26

Overview

Streams provide built-in transformations and support for custom transforms using async generators.

Built-in Transformations

For implementation details, see src/lib/stream/base-stream.ts.

map()

Transform each item:

const names = await stream
  .map(user => user.name)
  .collect();

filter()

Keep items matching predicate:

const active = await stream
  .filter(user => user.isActive)
  .collect();

flatMap()

Map and flatten nested iterables:

const allTags = await stream
  .flatMap(post => post.tags)
  .collect();

take()

Limit number of items:

const first10 = await stream
  .take(10)
  .collect();

skip()

Skip first N items:

const afterFirst10 = await stream
  .skip(10)
  .collect();

Custom Transformations

Creating Custom Transforms

function deduplicate<T>(key: (item: T) => string): Transform<T, T> {
  return async function*(source: AsyncIterable<T>) {
    const seen = new Set<string>();
    
    for await (const item of source) {
      const k = key(item);
      if (!seen.has(k)) {
        seen.add(k);
        yield item;
      }
    }
  };
}

// Usage
const unique = await stream
  .transform(deduplicate(item => item.id))
  .collect();

Batch Transform

See src/lib/stream/transform.ts for the batch implementation:

function batch<T>(size: number): Transform<T, T[]> {
  return async function*(source: AsyncIterable<T>) {
    let batch: T[] = [];
    
    for await (const item of source) {
      batch.push(item);
      if (batch.length === size) {
        yield batch;
        batch = [];
      }
    }
    
    if (batch.length > 0) {
      yield batch;
    }
  };
}

Functional Composition

For composition utilities, see src/lib/stream/transform.ts.

Compose Transforms

const transform = compose(
  map(x => x * 2),
  filter(x => x > 10),
  take(5)
);

const result = await stream
  .transform(transform)
  .collect();

Related Documentation

• Usage Patterns - Common usage examples
• Best Practices - Guidelines and anti-patterns
• Source Code - Implementation details

---

Next steps: See Best Practices for guidelines on using transformations effectively.