Serendipity Engine

The Serendipity Engine prevents filter bubbles and creates delightful discoveries. It ensures users find vendors they love but would never have searched for.

Overview

Mission: Prevent filter bubbles, enable delightful discoveries

Key Insight: The best wedding moments come from unexpected inspiration. Our job is to expand horizons while maintaining relevance.

Phase: Introduced in Phase 3 (Week 9)

Core Strategies

1. Cross-Regional Discovery

Show users exceptional vendors from neighboring regions:

async getCrossRegionalDiscoveries(
  user: User,
  currentFeed: Content[]
): Promise<Content[]> {
  const userRegion = user.location.region;

  // Find nearby regions
  const nearbyRegions = this.getNearbyRegions(userRegion);

  // Get top vendors from nearby regions
  const crossRegional = [];

  for (const region of nearbyRegions) {
    const topVendors = await this.getTopVendorsInRegion(
      region,
      user.style,
      limit: 3
    );

    // Only include if quality > 0.8 and worth the travel
    const worthy = topVendors.filter(v =>
      v.qualityScore > 0.8 &&
      this.travelDistance(user.location, v.location) < 200  // km
    );

    crossRegional.push(...worthy);
  }

  return crossRegional;
}

2. Style Exploration

Introduce tasteful variations to prevent monotony:

async getStyleExplorations(
  user: User,
  currentStyle: StyleEmbedding
): Promise<Content[]> {
  // Find adjacent styles in embedding space
  const adjacentStyles = await this.findAdjacentStyles(
    currentStyle,
    radius: 0.3  // Semantic distance
  );

  const explorations = [];

  for (const style of adjacentStyles) {
    // Get exemplary content in this style
    const examples = await this.getStyleExamples(style, limit: 2);

    // Only include high-quality examples
    explorations.push(
      ...examples.filter(e => e.qualityScore > 0.85)
    );
  }

  return explorations;
}

private async findAdjacentStyles(
  currentStyle: StyleEmbedding,
  radius: number
): Promise<StyleEmbedding[]> {
  // Vector search for nearby styles
  const results = await this.qdrant.search({
    collection: 'style_embeddings',
    vector: currentStyle,
    limit: 5,
    scoreThreshold: 0.7  // Still relevant
  });

  return results
    .filter(r => r.score < (1 - radius))  // Not too similar
    .map(r => r.payload);
}

3. Emerging Vendor Boost

Surface exceptional new vendors before they're popular:

async getEmergingVendors(
  user: User,
  userStyle: StyleEmbedding
): Promise<Content[]> {
  // Find vendors with:
  // - High quality but low follower count
  // - Recent account creation (< 6 months)
  // - Growing engagement
  // - Style match to user

  const emerging = await this.db.query(`
    SELECT * FROM vendors
    WHERE quality_score > 0.85
      AND follower_count < 5000
      AND account_age_days < 180
      AND growth_rate > 0.1
      AND region = $1
    ORDER BY quality_score DESC
    LIMIT 10
  `, [user.location.region]);

  // Filter by style match
  return emerging.filter(v => {
    const match = this.cosineSimilarity(v.styleEmbedding, userStyle);
    return match > 0.7;  // Relevant but not identical
  });
}

4. Collaborative Filtering

"Users like you also loved...":

async getCollaborativeDiscoveries(
  user: User
): Promise<Content[]> {
  // Find similar users
  const similarUsers = await this.findSimilarUsers(user, limit: 20);

  // Get content they loved that our user hasn't seen
  const discoveries = new Set<Content>();

  for (const similarUser of similarUsers) {
    const theirFavorites = await this.getUserFavorites(similarUser);

    for (const content of theirFavorites) {
      // Skip if user already saw it
      if (await this.hasSeenContent(user, content)) continue;

      // Skip if too different from user's style
      const match = this.styleMatch(user, content);
      if (match < 0.6) continue;

      discoveries.add(content);
    }
  }

  return Array.from(discoveries)
    .sort((a, b) => b.qualityScore - a.qualityScore)
    .slice(0, 10);
}

Diversity Measurement

Feed Diversity Metrics

measureDiversity(feed: Content[]): DiversityScore {
  // 1. Category diversity
  const categories = new Set(feed.map(c => c.vendorType));
  const categoryDiversity = categories.size / TOTAL_CATEGORIES;

  // 2. Style diversity (embedding space coverage)
  const styleDiversity = this.computeEmbeddingCoverage(
    feed.map(c => c.styleEmbedding)
  );

  // 3. Regional diversity
  const regions = new Set(feed.map(c => c.region));
  const regionalDiversity = regions.size / NEARBY_REGIONS.length;

  // 4. Vendor diversity (avoid same vendors)
  const vendors = new Set(feed.map(c => c.vendorId));
  const vendorDiversity = vendors.size / feed.length;

  // 5. Novelty (% of content not similar to past)
  const noveltyScore = await this.computeNovelty(feed);

  return {
    overall: (
      categoryDiversity * 0.25 +
      styleDiversity * 0.3 +
      regionalDiversity * 0.15 +
      vendorDiversity * 0.15 +
      noveltyScore * 0.15
    ),
    category: categoryDiversity,
    style: styleDiversity,
    regional: regionalDiversity,
    vendor: vendorDiversity,
    novelty: noveltyScore
  };
}

Embedding Space Coverage

private computeEmbeddingCoverage(
  embeddings: number[][]
): number {
  if (embeddings.length < 2) return 0;

  // Compute pairwise distances
  const distances = [];

  for (let i = 0; i < embeddings.length; i++) {
    for (let j = i + 1; j < embeddings.length; j++) {
      const dist = this.euclideanDistance(
        embeddings[i],
        embeddings[j]
      );
      distances.push(dist);
    }
  }

  // Average distance = diversity
  // Normalize to 0-1 scale
  const avgDistance = distances.reduce((a, b) => a + b) / distances.length;
  return Math.min(avgDistance / 2.0, 1.0);
}

Injection Strategy

When to Inject Serendipity

async generateSerendipitousCandidates(
  user: User,
  recentFeed: Content[]
): Promise<Content[]> {
  // Measure current diversity
  const diversity = this.measureDiversity(recentFeed);

  // If diversity is low, inject more serendipity
  if (diversity.overall < DIVERSITY_THRESHOLD) {
    const injectionRate = 1.0 - diversity.overall;  // 0.0 to 1.0

    const serendipitous = [];

    // Cross-regional (if low regional diversity)
    if (diversity.regional < 0.5) {
      const crossRegional = await this.getCrossRegionalDiscoveries(
        user,
        recentFeed
      );
      serendipitous.push(
        ...this.sample(crossRegional, Math.ceil(injectionRate * 3))
      );
    }

    // Style exploration (if low style diversity)
    if (diversity.style < 0.6) {
      const styleExplore = await this.getStyleExplorations(
        user,
        user.styleEmbedding
      );
      serendipitous.push(
        ...this.sample(styleExplore, Math.ceil(injectionRate * 3))
      );
    }

    // Emerging vendors (always good)
    const emerging = await this.getEmergingVendors(user, user.styleEmbedding);
    serendipitous.push(
      ...this.sample(emerging, Math.ceil(injectionRate * 2))
    );

    // Collaborative discoveries
    const collaborative = await this.getCollaborativeDiscoveries(user);
    serendipitous.push(
      ...this.sample(collaborative, Math.ceil(injectionRate * 2))
    );

    return serendipitous;
  }

  return [];
}

Mixing Serendipity with Relevance

async mixSerendipityIntoFeed(
  mainFeed: Content[],
  serendipitous: Content[],
  diversityScore: number
): Promise<Content[]> {
  // Determine injection ratio based on diversity
  const injectionRatio = Math.max(0.1, 1.0 - diversityScore);

  const mixedFeed = [];
  let serendipityIndex = 0;

  for (let i = 0; i < mainFeed.length; i++) {
    // Add main content
    mixedFeed.push(mainFeed[i]);

    // Inject serendipitous content at intervals
    if (
      serendipityIndex < serendipitous.length &&
      Math.random() < injectionRatio
    ) {
      mixedFeed.push(serendipitous[serendipityIndex++]);
    }
  }

  return mixedFeed;
}

Quality Safeguards

Only Good Surprises

async validateSerendipitousContent(
  content: Content,
  user: User
): Promise<boolean> {
  // Quality threshold (only show high-quality surprises)
  if (content.qualityScore < 0.75) return false;

  // Minimum relevance (not TOO different)
  const relevance = this.cosineSimilarity(
    user.styleEmbedding,
    content.styleEmbedding
  );
  if (relevance < 0.5) return false;

  // Regional feasibility
  const distance = this.travelDistance(
    user.location,
    content.vendorLocation
  );
  if (distance > 300) return false;  // 300km max

  return true;
}

API Endpoints

Generate Serendipitous Feed

Endpoint: GET /api/serendipity/:userId

Response:

href="#__codelineno-9-1">{ "discoveries": [ { "contentId": "content-789", "type": "cross_regional", "vendor": { "name": "Garden Weddings Sydney", "region": "Sydney", "distance": 80 }, "reasoning": "Exceptional garden venue just 80km away" }, { "contentId": "content-790", "type": "style_exploration", "styleShift": "modern → modern_bohemian", "reasoning": "Tasteful bohemian elements that complement your modern style" }, { "contentId": "content-791", "type": "emerging", "vendor": { "name": "New Wave Photography", "qualityScore": 0.92, "followers": 1200 }, "reasoning": "Exceptional emerging photographer in your area" } ], "diversityImprovement": 0.35 }

Measure Diversity

Endpoint: POST /api/serendipity/diversity

Request:

{
  "feedContents": ["content-1", "content-2", ...]
}