Skip to content

Instantly share code, notes, and snippets.

@Theaxiom

Theaxiom/Building a Pull-Through Cache in Flutter with Drift, Firestore, and SharedPreferences.md

Last active April 16, 2025 09:07
Show Gist options
  • Save Theaxiom/3d85296d2993542b237e6fb425e3ddf1 to your computer and use it in GitHub Desktop.
Save Theaxiom/3d85296d2993542b237e6fb425e3ddf1 to your computer and use it in GitHub Desktop.
Download ZIP
Building a Pull-Through Cache in Flutter with Drift, Firestore, and SharedPreferences
Raw
Building a Pull-Through Cache in Flutter with Drift, Firestore, and SharedPreferences.md

Building a Pull-Through Cache in Flutter with Drift, Firestore, and SharedPreferences

Hey fellow Flutter and Dart Devs!

I wanted to share a pull-through caching strategy we implemented in our app, MyApp, to manage data synchronization between a remote backend (Firestore) and a local database (Drift). This approach helps reduce backend reads, provides basic offline capabilities, and offers flexibility in data handling.

The Goal

Create a system where the app prioritizes fetching data from a local Drift database. If the data isn't present locally or is considered stale (based on a configurable duration), it fetches from Firestore, updates the local cache, and then returns the data.

Core Components

  1. Drift: For the local SQLite database. We define tables for our data models.
  2. Firestore: As the remote source of truth.
  3. SharedPreferences: To store simple metadata, specifically the last time a full sync was performed for each table/entity type.
  4. connectivity_plus: To check for network connectivity before attempting remote fetches.

Implementation Overview

Abstract Cache Manager

We start with an abstract CacheManager class that defines the core logic and dependencies.

// Standard Dart Libraries
import 'dart:developer';

// External Libraries and Packages
import 'package:cloud_firestore/cloud_firestore.dart';
import 'package:connectivity_plus/connectivity_plus.dart';
import 'package:firebase_auth/firebase_auth.dart';
import 'package:shared_preferences/shared_preferences.dart';

// Services
import 'package:reely/services/firebase_auth_service.dart';

abstract class CacheManager<T> {
  static const Duration defaultCacheDuration = Duration(minutes: 3);

  final Duration cacheExpiryDuration;
  final FirebaseFirestore _firestore = FirebaseFirestore.instance;
  final FirebaseAuthService _authService = FirebaseAuthService(
    FirebaseAuth.instance,
  );

  static DateTime? _lastConnectivityCheck;
  static bool? _lastConnectivityResult;

  CacheManager({this.cacheExpiryDuration = defaultCacheDuration});

  FirebaseAuthService get authService => _authService;

  FirebaseFirestore get firestore => _firestore;

  String getCurrentUserId() {
    // Retrieve the current user's ID from the authentication service.
    // This is used to associate data with the logged-in user.
    final String userId = _authService.getCurrentUserId();
    return userId;
  }

  // Gets a single entity from the local Drift database.
  Future<T?> getFromLocal(String id);

  // Saves or updates a single entity in the local Drift database.
  Future<void> saveToLocal(T entity);

  // Deletes all entities from the local Drift database.
  Future<void> deleteAll();

  Future<T> fetchFromRemote(String id);

  // Maps a Firestore document to an entity.
  T mapFirestoreToEntity(Map<String, dynamic> data);

  // Maps an entity to a Firestore document.
  Map<String, dynamic> mapEntityToFirestore(T entity);

  // Prunes stale records from the local Drift database based on the lastSynced field.
  Future<void> pruneStaleRecords(DateTime threshold);

  static Future<bool> hasConnectivity() async {
    log('Checking connectivity...');
    final now = DateTime.now();
    if (_lastConnectivityCheck != null &&
        now.difference(_lastConnectivityCheck!) < Duration(seconds: 10)) {
      log('Using cached connectivity result: $_lastConnectivityResult');
      return _lastConnectivityResult!;
    }

    try {
      log('Performing actual connectivity check...');
      final connectivityResult = await Connectivity().checkConnectivity();
      _lastConnectivityResult =
          connectivityResult.contains(ConnectivityResult.mobile) ||
          connectivityResult.contains(ConnectivityResult.wifi);
      _lastConnectivityCheck = now;
      log('Connectivity result: $_lastConnectivityResult');
      return _lastConnectivityResult!;
    } catch (e) {
      log('Error checking connectivity: $e', error: e);
      throw CacheManagerException('Failed to check connectivity', e);
    }
  }

  Future<T> getById(String id) async {
    log('Fetching entity with ID: $id');
    try {
      // Attempt to retrieve the entity from the local Drift database.
      final T? localEntity = await getFromLocal(id);

      // Return the local entity if it exists and is not expired.
      if (localEntity != null && !isCacheExpired(localEntity, DateTime.now())) {
        log('Returning cached entity for ID: $id');
        return localEntity;
      }

      // Check for connectivity before attempting to fetch from remote.
      if (!await hasConnectivity()) {
        if (localEntity != null) {
          log('No connectivity, returning cached entity for ID: $id');
          return localEntity;
        }
        throw CacheManagerException(
          'No internet connection and no local entity found for ID: $id',
        );
      }

      // Fetch the entity from the remote source and update local storage.
      final T remoteEntity = await fetchFromRemote(id);
      await saveToLocal(remoteEntity);

      // Retrieve and return the saved entity from the local Drift database.
      final T? savedEntity = await getFromLocal(id);
      if (savedEntity != null) {
        log('Fetched and saved entity from remote for ID: $id');
        return savedEntity;
      } else {
        throw CacheManagerException(
          'Failed to retrieve the saved entity from local storage for ID: $id',
        );
      }
    } catch (e) {
      log('Error fetching entity with ID: $id, Error: $e', error: e);
      throw CacheManagerException('Failed to get entity by ID $id', e);
    }
  }

  Future<List<T>> fetchAllFromRemote(
    String collectionPath, {
    List<String>? ids,
    Query Function(Query)? queryModifier,
  }) async {
    try {
      log('Fetching all entities from remote collection: $collectionPath');
      // Ensure connectivity before fetching data from Firestore.
      if (!await hasConnectivity()) {
        throw CacheManagerException(
          'No internet connection (mobile or Wi-Fi).',
        );
      }

      // Build a Firestore query, optionally filtering by document IDs.
      Query query = _firestore.collection(collectionPath);
      if (ids != null && ids.isNotEmpty) {
        log('Filtering by document IDs: $ids');
        query = query.where(FieldPath.documentId, whereIn: ids);
      }

      // Apply custom query modifiers if provided.
      if (queryModifier != null) {
        log('Applying custom query modifier');
        query = queryModifier(query);
      }

      // Execute the query and map the results to entities.
      log('Executing Firestore query...');
      final querySnapshot = await query.get();
      log(
        'Query executed successfully, ${querySnapshot.docs.length} documents found',
      );
      return querySnapshot.docs.map((doc) {
        final data = doc.data() as Map<String, dynamic>;
        data['id'] = doc.id; // Ensure ID is in the map
        return mapFirestoreToEntity(data);
      }).toList();
    } catch (e) {
      log('Error fetching all entities from remote: $e', error: e);
      throw CacheManagerException(
        'Failed to fetch all entities from remote',
        e,
      );
    }
  }

  // Add a method to sync all entities if the global cache is expired.
  Future<List<T>> syncAndGetAll(
    String collectionPath, {
    Query Function(Query)? queryModifier,
  }) async {
    log('Starting sync and fetch for all entities in $collectionPath');
    try {
      if (await isCacheExpiredForAllSync()) {
        log('Cache expired for $collectionPath, fetching from remote');
        final remoteEntities = await fetchAllFromRemote(
          collectionPath,
          queryModifier: queryModifier,
        );
        for (final entity in remoteEntities) {
          await saveToLocal(entity);
        }
        await pruneStaleRecords(DateTime.now().subtract(cacheExpiryDuration));
        await setLastSyncedAll(DateTime.now());
        log('Sync completed for $collectionPath');
      }
      final localEntities = await getAllFromLocal();
      log(
        'Fetched ${localEntities.length} entities from local database for $collectionPath',
      );
      return localEntities;
    } catch (e) {
      log('Error during sync and fetch for $collectionPath: $e', error: e);
      throw CacheManagerException('Failed to sync and fetch all entities', e);
    }
  }

  Future<List<T>> syncAndGetSubset(
    String collectionPath, {
    required Query Function(Query) queryModifier,
  }) async {
    log('Starting sync and fetch for subset of entities in $collectionPath');
    try {
      final remoteEntities = await fetchAllFromRemote(
        collectionPath,
        queryModifier: queryModifier,
      );
      for (final entity in remoteEntities) {
        await saveToLocal(entity);
      }
      log('Fetched and saved subset of entities for $collectionPath');
      return remoteEntities;
    } catch (e) {
      log(
        'Error during sync and fetch for subset in $collectionPath: $e',
        error: e,
      );
      throw CacheManagerException(
        'Failed to sync and fetch subset of entities',
        e,
      );
    }
  }

  // Abstract method to retrieve all entities from the local Drift database.
  Future<List<T>> getAllFromLocal();

  bool isCacheExpired(T entity, DateTime now) {
    log('Checking if cache is expired for entity: $entity');
    if (entity is CacheableEntity) {
      if (cacheExpiryDuration == Duration.zero) {
        log('Cache expiry duration is zero, cache is always expired');
        return true;
      }
      log('Cache expiry duration: $cacheExpiryDuration');
      return entity.lastSynced.isBefore(now.subtract(cacheExpiryDuration));
    }
    throw UnimplementedError('isCacheExpired must be implemented for $T');
  }

  String get lastSyncedAllKey;

  Future<DateTime?> getLastSyncedAll() async {
    log('Retrieving last synced timestamp');
    try {
      final prefs = await SharedPreferences.getInstance();
      final lastSyncedMillis = prefs.getInt(lastSyncedAllKey);
      if (lastSyncedMillis != null) {
        final lastSynced = DateTime.fromMillisecondsSinceEpoch(
          lastSyncedMillis,
        );
        log('Last synced timestamp retrieved: $lastSynced');
        return lastSynced;
      }
      log('No last synced timestamp found');
      return null;
    } catch (e) {
      log('Error retrieving last synced timestamp: $e', error: e);
      throw CacheManagerException(
        'Failed to retrieve last synced timestamp',
        e,
      );
    }
  }

  Future<void> setLastSyncedAll(DateTime lastSynced) async {
    log('Setting last synced timestamp to $lastSynced');
    try {
      final prefs = await SharedPreferences.getInstance();
      await prefs.setInt(lastSyncedAllKey, lastSynced.millisecondsSinceEpoch);
      log('Last synced timestamp set successfully');
    } catch (e) {
      log('Error setting last synced timestamp: $e', error: e);
      throw CacheManagerException('Failed to set last synced timestamp', e);
    }
  }

  Future<void> clearLastSyncedAll() async {
    log('Clearing last synced timestamp');
    try {
      final prefs = await SharedPreferences.getInstance();
      await prefs.remove(lastSyncedAllKey);
      log('Last synced timestamp cleared successfully');
    } catch (e) {
      log('Error clearing last synced timestamp: $e', error: e);
      throw CacheManagerException('Failed to clear last synced timestamp', e);
    }
  }

  Future<bool> isCacheExpiredForAllSync() async {
    log('Checking if cache is expired for all sync');
    try {
      final lastSynced = await getLastSyncedAll();
      if (lastSynced == null) {
        log('Cache is expired: no last synced timestamp found');
        return true;
      }
      final isExpired = lastSynced.isBefore(
        DateTime.now().subtract(cacheExpiryDuration),
      );
      log('Cache expired status: $isExpired');
      return isExpired;
    } catch (e) {
      log('Error checking cache expiration for all sync: $e', error: e);
      throw CacheManagerException(
        'Failed to check cache expiration for all sync',
        e,
      );
    }
  }
}

// Define CacheableEntity interface
abstract class CacheableEntity {
  DateTime get lastSynced;
}

class CacheManagerException implements Exception {
  final String message;
  final Object? cause;

  CacheManagerException(this.message, [this.cause]);

  @override
  String toString() {
    return 'CacheManagerException: $message, Cause: $cause';
  }
}

Drift Table Definitions

Define your tables in Drift as usual. Include a lastSynced nullable DateTime column.

import 'package:drift/drift.dart';

@DataClassName('User')
class Users extends Table {
  TextColumn get uid => text().customConstraint('NOT NULL')();
  TextColumn get email => text().customConstraint('NOT NULL')();
  TextColumn get photoUrl => text()();
  DateTimeColumn get createdTime => dateTime()();
  TextColumn get username => text()();
  // ... other user fields ...
  DateTimeColumn get lastSynced => dateTime().nullable()(); // Crucial for caching!

  @override
  Set<Column> get primaryKey => {uid};
}

@DataClassName('Connection')
class Connections extends Table {
  TextColumn get id => text()(); // Firestore document ID
  TextColumn get userA => text().customConstraint('REFERENCES users(uid) NOT NULL')();
  TextColumn get userB => text().customConstraint('REFERENCES users(uid) NOT NULL')();
  TextColumn get initiator => text().customConstraint('REFERENCES users(uid) NOT NULL')();
  DateTimeColumn get createdAt => dateTime()();
  DateTimeColumn get acceptedAt => dateTime().nullable()();
  BoolColumn get isActive => boolean().withDefault(const Constant(false))();
  // Locally derived/managed fields, not directly from Firestore:
  BoolColumn get isRequester => boolean().withDefault(const Constant(false))(); 
  TextColumn get connectionStatus => text().nullable()(); // e.g., 'sent', 'received', 'connected'
  DateTimeColumn get lastSynced => dateTime().nullable()(); // Cache expiry tracking

  @override
  // Use Firestore ID as primary key locally if userA/userB isn't guaranteed unique
  // or handle composite key appropriately in upserts. Using 'id' might be simpler.
  Set<Column> get primaryKey => {id}; 
  // Or if (userA, userB) is truly unique: Set<Column> get primaryKey => {userA, userB};
}

Concrete Cache Manager Implementations

Create specific managers for each entity type, implementing the abstract methods.

UsersCacheManager Example

import 'package:myapp/database/app_database.dart'; // Your Drift DB class
import 'package:myapp/database/cache_manager.dart';

class UsersCacheManager extends CacheManager<User> {
  final AppDatabase database;

  // Key for SharedPreferences global sync tracking for users
  static const String _lastSyncedAllKey = 'last_synced_all_users'; 
  // Specific cache duration for users
  static const Duration _cacheDuration = Duration(hours: 1); 

  UsersCacheManager({required this.database}) 
      : super(cacheExpiryDuration: _cacheDuration);

  @override
  String get lastSyncedAllKey => _lastSyncedAllKey;

  @override
  Future<User?> getFromLocal(String id) async {
    return (database.select(database.users)..where((tbl) => tbl.uid.equals(id))).getSingleOrNull();
  }

  @override
  Future<void> saveToLocal(User entity) async {
    // Use insertOrReplace to handle both inserts and updates
    await database.into(database.users).insert(entity, mode: InsertMode.insertOrReplace); 
  }

  @override
  Future<User> fetchFromRemote(String id) async {
     if (!await CacheManager.hasConnectivity()) {
       throw Exception('No internet connection.');
     }
     final doc = await firestore.collection('users').doc(id).get();
     if (!doc.exists) {
       throw Exception('User $id not found in Firestore');
     }
     final data = doc.data() as Map<String, dynamic>;
     data['id'] = doc.id; // Ensure ID is in the map
     return mapFirestoreToEntity(data);
  }

  @override
  User mapFirestoreToEntity(Map<String, dynamic> data) {
    // Perform mapping from Firestore fields to Drift User object
    // Remember to set lastSynced locally!
    return User(
      uid: data['id'], // Use 'id' which we added
      email: data['email'] ?? '',
      // ... map other fields ...
      lastSynced: DateTime.now(), // Set sync time *locally*
    );
  }

  @override
  Map<String, dynamic> mapEntityToFirestore(User entity) {
    // Map Drift User entity back to Firestore structure for updates
    // *Exclude* lastSynced as it's local metadata
    return {
      'email': entity.email,
      // ... map other fields, excluding uid/id if managed by Firestore
    };
  }


  @override
  bool isCacheExpired(User entity, DateTime now) {
    if (cacheExpiryDuration == Duration.zero) return true;
    return entity.lastSynced == null || 
           entity.lastSynced!.isBefore(now.subtract(cacheExpiryDuration));
  }

  // Example: Method to sync all users if the global cache is expired
  Future<List<User>> syncAndGetAllUsers() async {
    if (await isCacheExpiredForAllSync()) {
      print('User cache expired, fetching all from remote...');
      // Add necessary filters if you don't want *all* users (e.g., based on current user)
      // final currentUser = authService.getCurrentUserId(); // Example
      // final remoteUsers = await fetchAllFromRemote('users', queryModifier: (q) => q.where(...)); 
      final remoteUsers = await fetchAllFromRemote('users'); 
      for (final user in remoteUsers) {
        await saveToLocal(user);
      }
      await setLastSyncedAll(DateTime.now());
    }
    return database.select(database.users).get();
  }
}

ConnectionsCacheManager Highlights

  • Similar structure to UsersCacheManager.
  • Data Normalization/Enhancement: In mapFirestoreToEntity, you'd calculate isRequester based on the initiator field and the current user ID. You'd also determine the connectionStatus ('connected', 'sent', 'received') based on isActive and isRequester. Ensure userA and userB are stored consistently (e.g., lexicographically). Set lastSynced locally.

Filtering WorkspaceAllFromRemote: When fetching all connections for the current user, modify the query:

// Inside ConnectionsCacheManager
Future<List<Connection>> syncAndGetAllConnectionsForCurrentUser() async {
  if (await isCacheExpiredForAllSync()) {
     final currentUserId = authService.getCurrentUserId(); 
     final userRef = firestore.collection('users').doc(currentUserId);

     // Build a Firestore filter for connections involving the current user
     final queryModifier = (Query query) {
        return query.where(Filter.or(
            Filter('user_a', isEqualTo: userRef),
            Filter('user_b', isEqualTo: userRef)
        ));
     };

     final remoteConnections = await fetchAllFromRemote('connections', queryModifier: queryModifier);
     for (final conn in remoteConnections) {
         await saveToLocal(conn); // mapFirestoreToEntity handles normalization
     }
     await setLastSyncedAll(DateTime.now());
  }
  // Query local DB for connections relevant to the user
  final currentUserId = authService.getCurrentUserId();
  return (database.select(database.connections)
            ..where((c) => c.userA.equals(currentUserId) | c.userB.equals(currentUserId)))
         .get();
}

Mapping to Firestore: In mapEntityToFirestore, convert userA, userB, initiator back to Firestore DocumentReferences. Exclude locally managed fields like isRequester, connectionStatus, and lastSynced.

Using the Cache in Your App (e.g., in Repositories/Services)

import 'package:cloud_firestore/cloud_firestore.dart'; // Required for Query type
import 'package:myapp/database/app_database.dart'; // Your Drift DB class
import 'package:myapp/database/cache_managers/users_cache_manager.dart';
import 'package:myapp/database/cache_manager.dart'; // For CacheManager base and connectivity

class UserRepository {
  final AppDatabase db;
  // The default cache manager instance for this repository
  late final UsersCacheManager cacheManager;

  UserRepository(this.db) {
    // Initialize the default manager with its standard duration
    cacheManager = UsersCacheManager(database: db);
  }

  /// Gets a single user, handling caching automatically.
  /// Optionally accepts an [overrideDuration] for this specific call.
  Future<User> getUser(String uid, {Duration? overrideDuration}) async {
    if (overrideDuration != null) {
      // Override: Use a temporary manager
      print("Using temporary UsersCacheManager with duration: $overrideDuration");
      final tempCacheManager = UsersCacheManager(
        database: db,
        cacheExpiryDuration: overrideDuration
      );
      return await tempCacheManager.getById(uid);
    } else {
      // Default: Use the standard repository manager
      print("Using default UsersCacheManager with duration: ${cacheManager.cacheExpiryDuration}");
      return await cacheManager.getById(uid);
    }
  }

  /// Gets users based on a specific Firestore query.
  ///
  /// This method directly fetches from Firestore using the provided [queryModifier],
  /// updates the local cache with the results, and returns the fetched list.
  /// It bypasses the global cache expiry check used by [getAllRelevantUsers].
  ///
  /// Example [queryModifier]: `(query) => query.where('level', isGreaterThan: 5).limit(10)`
  Future<List<User>> getUsersByQuery(
    Query Function(Query query) queryModifier,
  ) async {
    print("getUsersByQuery: Forcing remote fetch.");
    final tempCacheManager = UsersCacheManager(
      database: db,
      cacheExpiryDuration: Duration.zero // Force remote fetch for this query
    );
    // Always fetch remotely for a specific query, ensuring connectivity first.
    if (!await tempCacheManager.hasConnectivity()) {
      // Optionally, you could try returning local data matching the query here,
      // but that requires translating Firestore query logic to Drift queries.
      // For simplicity, we'll require connectivity for direct queries.
      throw Exception('No internet connection to fetch users by query.');
    }

    try {
      print("Fetching users from remote with custom query...");
        // Use the base fetchAllFromRemote, applying the modifier through tempCacheManager.
      final List<User> remoteUsers = await tempCacheManager.fetchAllFromRemote(
        'users', // Firestore collection path
        queryModifier: queryModifier, // Apply the custom query logic
      );

      // Update the local cache with the fetched users
      print("Updating local cache with ${remoteUsers.length} fetched users...");
      for (final user in remoteUsers) {
        // saveToLocal handles insert or replace
        await tempCacheManager.saveToLocal(user); 
      }

      // Return the list of users fetched from the remote source
      return remoteUsers;

    } catch (e) {
      print("Failed to fetch users by query: $e");
      // Depending on requirements, you might return an empty list,
      // or try to fetch matching data from local cache as a fallback.
      throw Exception('Failed to fetch users by query: $e');
    }
  }


  /// Gets all relevant users, syncing from remote if the global cache is expired.
  /// Returns all users currently stored in the local database after the potential sync.
  Future<List<User>> getAllRelevantUsers() async {
    print("Getting all relevant users (syncing if cache expired)...");
    // This method uses the default cacheManager instance and its sync logic.
    // The query performed during sync (if it happens) is the default one
    // inside cacheManager.syncAndGetAllUsers (likely fetches all or user-specific).
    return await cacheManager.syncAndGetAllUsers();
  }

  /// Updates user data in Firestore and refreshes the local cache for that user.
  Future<void> updateUserProfile(String uid, Map<String, dynamic> data) async {
     if (!await CacheManager.hasConnectivity()) {
        throw Exception('No internet connection to update profile.');
     }
     try {
       // Update Firestore first
       await cacheManager.firestore.collection('users').doc(uid).update(data);

       // Fetch the updated data fresh from the remote source to ensure consistency
       final updatedUser = await cacheManager.fetchFromRemote(uid);

       // Save the freshly fetched data to the local cache
       await cacheManager.saveToLocal(updatedUser);

     } catch (e) {
        print("Failed to update user profile for $uid: $e");
        throw Exception('Failed to update user profile: $e');
     }
  }
}

// --- Example Usage ---
/*
final userRepository = UserRepository(myAppDatabaseInstance);

// Get user using default caching
final user1 = await userRepository.getUser("user123");

// Get user forcing cache check against 1 minute
final user2 = await userRepository.getUser("user456", overrideDuration: Duration(minutes: 1));

// Sync all users if needed, then get all from local DB
final allUsers = await userRepository.getAllRelevantUsers();

// Get only users with level > 10 directly from Firestore, update cache, return them
final highLevelUsers = await userRepository.getUsersByQuery(
  (query) => query.where('level', isGreaterThan: 10).orderBy('xp', descending: true)
);

// Get users matching a specific email directly from Firestore
final specificUserList = await userRepository.getUsersByQuery(
  (query) => query.where('email', isEqualTo: '[email protected]').limit(1)
);

*/

Targeted Data Fetching & Solution Extensibility

In the previous examples, methods like getUser and getAllRelevantUsers primarily rely on a time-based cache expiration strategy. They check if the data (either a single record or the dataset as a whole) is stale based on a predefined duration. If it is, they fetch from the remote source (Firestore) and update the local cache (Drift).

However, sometimes you need to fetch a very specific subset of data based on criteria not related to simple cache expiry. For example, you might need "the 10 newest users in Group X" or "all users whose score is above 1000".

Trying to shoehorn this into the getAllRelevantUsers method presents problems:

  1. Conflicting Purpose: getAllRelevantUsers is meant to ensure the general local cache is reasonably up-to-date by potentially syncing a broad set of data if the overall cache timer has expired. It's not designed for specific, one-off queries.
  2. Incorrect Results: Even if you modified the sync part of getAllRelevantUsers to use a specific query, the method (as originally designed) typically returns all users from the local cache after the sync, not just the specific ones fetched by your query.

Therefore, adding a dedicated method like getUsersByQuery provides a clearer and more direct approach:

  • Direct Remote Fetch: It explicitly bypasses the general cache timer and goes directly to the remote source (Firestore).
  • Specific Query: It executes the exact query you define using the queryModifier.
  • Targeted Cache Update: It updates the local cache only with the specific records returned by your query.
  • Precise Return Value: It returns only the list of records fetched by your query.

This separates the concern of routine, time-based caching from targeted, on-demand data fetching based on specific criteria.

Extensibility and Flexibility Examples

This pull-through cache architecture, built around the abstract CacheManager and specific implementations, offers considerable flexibility. Here are some ways it can be extended or adapted:

  1. Swapping Backends:

    • How: Replace Firestore with a REST API, GraphQL endpoint, or another database. You only need to modify the implementation details within the specific CacheManager subclasses: primarily WorkspaceFromRemote, WorkspaceAllFromRemote, mapFirestoreToEntity (renamed appropriately), and mapEntityToFirestore. The core caching logic (getById, expiry checks) and the Repository layer remain largely unchanged.
    • Example: Create a UserRestCacheManager where WorkspaceFromRemote uses http.get and mapRestToEntity parses JSON instead of a Firestore DocumentSnapshot.

  2. Combining Data Sources:

    • How: An entity's data might come from multiple places. For instance, a UserProfile might have basic info from Firestore but recent activity from a separate REST endpoint.
    • Example: The UserCacheManager's WorkspaceFromRemote could first fetch from Firestore, then make a secondary call to a REST API, combining the data before calling mapCombinedDataToEntity and saving the unified User object locally.

  3. Fine-Grained Cache Durations:

    • How: Different data types have different volatility. Set distinct cacheExpiryDuration values in each specific CacheManager implementation.
    • Example: UsersCacheManager(cacheExpiryDuration: Duration(hours: 1)), PostsCacheManager(cacheExpiryDuration: Duration(minutes: 5)), AppSettingsCacheManager(cacheExpiryDuration: Duration(days: 1)).

  4. Complex Local Queries:

    • How: Once data from different sources/tables is cached locally in Drift, your repository methods can perform complex SQL queries (via Drift's Dart API) that join this data.
    • Example: In ConnectionsRepository, you could add a method findUsersConnectedToAdmins() that joins the local connections table with the local users table (where isAdmin == true), all using data previously cached from potentially different remote sources or queries.

  5. Offline Mutations (Write-Behind Cache):

    • How: Currently, updateUserProfile fails offline. Extend it by adding a local "pending updates" queue (e.g., another Drift table).
    • Example: If offline, updateUserProfile writes the change to the pending_updates table. A separate service monitors connectivity and processes this queue, sending updates to Firestore when back online, then clearing the queue entry and potentially refreshing the main local cache (saveToLocal).

  6. Real-time Data Integration:

    • How: For data needing real-time updates, integrate Firestore listeners (collection.snapshots()) or WebSockets.
    • Example: A service could listen to Firestore's user snapshots. When an update is received, it calls mapFirestoreToEntity and then UsersCacheManager.saveToLocal(updatedUser), pushing changes into the local cache proactively, complementing the pull-through mechanism.

  7. Selective Syncing/Fetching:

    • How: Instead of fetching all users/connections on expiry, fetch only specific ones needed currently.
    • Example: Add a syncSpecificUsers(List<String> userIds) method to UserRepository that calls cacheManager.fetchAllFromRemote('users', ids: userIds), ensuring only those specific users are updated in the cache.

  8. Enhanced Data Transformation:

    • How: The mapFirestoreToEntity (or equivalent) method is a powerful spot for data cleansing, normalization, or calculating derived properties.
    • Example: If Firestore stores timestamps inconsistently, mapFirestoreToEntity can standardize them. If user roles are stored as integers, it can map them to meaningful enums. It can calculate User.fullName from firstName and lastName fields in Firestore.

This layered approach (Repository -> Cache Manager -> Local DB / Remote Source) provides distinct points for modification and extension, allowing the system to evolve without requiring a complete rewrite for every new requirement or backend change.

Extensibility Implementation Examples

// --- Assume these methods exist in the base CacheManager ---
// abstract class CacheManager<T> {
//   // ... existing properties and methods ...
//   FirebaseFirestore get firestore;
//   Future<bool> hasConnectivity();
//   Future<void> saveToLocal(T entity);
//   T mapFirestoreToEntity(Map<String, dynamic> data);

//   // Fetches from remote, applies query modifier. Base version doesn't handle caching itself.
//   Future<List<T>> fetchAllFromRemote(
//     String collectionPath, {
//     List<String>? ids,
//     Query Function(Query query)? queryModifier,
//   }) async {
//      if (!await hasConnectivity()) {
//         throw Exception('No internet connection.');
//      }
//      Query query = firestore.collection(collectionPath);
//      if (ids != null && ids.isNotEmpty) {
//         // Firestore 'in' query limit is 10; needs chunking for more.
//         query = query.where(FieldPath.documentId, whereIn: ids.take(10));
//      }
//      if (queryModifier != null) {
//          query = queryModifier(query);
//      }
//      final querySnapshot = await query.get();
//      return querySnapshot.docs.map((doc) {
//         final data = doc.data() as Map<String, dynamic>;
//         data['id'] = doc.id;
//         return mapFirestoreToEntity(data);
//      }).toList();
//   }

//   // Checks individual entity expiry - used by getById
//   bool isCacheExpired(T entity, DateTime now, {Duration? overrideDuration});

//   // Checks global sync expiry - used by syncAndGetAllUsers
//   Future<bool> isCacheExpiredForAllSync();
//   Future<void> setLastSyncedAll(DateTime lastSynced);
//   String get lastSyncedAllKey;
// }


// --- Modifications/Additions to UsersCacheManager ---
class UsersCacheManager extends CacheManager<User> {
  final AppDatabase database;
  static const String _lastSyncedAllKey = 'last_synced_all_users';
  static const Duration _cacheDuration = Duration(hours: 1);

  UsersCacheManager({required this.database})
      : super(cacheExpiryDuration: _cacheDuration);

  // --- Implementations of abstract methods (as before) ---
  @override String get lastSyncedAllKey => _lastSyncedAllKey;

  @override
  Future<User?> getFromLocal(String id) async =>
      (database.select(database.users)..where((tbl) => tbl.uid.equals(id))).getSingleOrNull();

  @override
  Future<void> saveToLocal(User entity) async =>
      await database.into(database.users).insert(entity, mode: InsertMode.insertOrReplace);

  @override
  Future<User> fetchFromRemote(String id) async {
     // (Implementation as before - fetches single doc by ID)
     if (!await CacheManager.hasConnectivity()) throw Exception('No internet connection.');
     final doc = await firestore.collection('users').doc(id).get();
     if (!doc.exists) throw Exception('User $id not found in Firestore');
     final data = doc.data()! as Map<String, dynamic>;
     data['id'] = doc.id;
     return mapFirestoreToEntity(data);
  }

  @override
  User mapFirestoreToEntity(Map<String, dynamic> data) {
     // (Implementation as before - maps fields, sets lastSynced=now())
     return User(
          uid: data['id'],
          email: data['email'] ?? '',
          // ... other fields ...
          isAdmin: data['isAdmin'] ?? false, // Added for local query example
          lastSynced: DateTime.now(),
        );
  }

   @override
   Map<String, dynamic> mapEntityToFirestore(User entity) {
      // (Implementation as before - maps for writing, excludes lastSynced)
      return {
          'email': entity.email,
          'isAdmin': entity.isAdmin,
          // ... other fields ...
        };
   }


  @override
  bool isCacheExpired(User entity, DateTime now, {Duration? overrideDuration}) {
     // (Implementation as before - uses overrideDuration or instance default)
     final durationToUse = overrideDuration ?? cacheExpiryDuration;
     if (durationToUse == Duration.zero) return true; // Explicitly handle zero duration
     return entity.lastSynced == null ||
            entity.lastSynced!.isBefore(now.subtract(durationToUse));
  }

  // --- Method for the standard "sync all if expired" pattern ---
  Future<List<User>> syncAndGetAllUsers() async {
    if (await isCacheExpiredForAllSync()) {
      print('User cache expired, fetching all from remote...');
      // Base fetchAllFromRemote gets all (or applies default filter if any)
      final remoteUsers = await fetchAllFromRemote('users');
      for (final user in remoteUsers) {
        await saveToLocal(user);
      }
      await setLastSyncedAll(DateTime.now());
    }
    // Returns all locally after sync attempt
    return database.select(database.users).get();
  }

  /// **NEW/REFINED METHOD:** Fetches users based on a query, updates cache,
  /// and allows forcing the remote fetch via cache duration override.
  ///
  /// If [cacheDuration] is `Duration.zero`, it forces a remote fetch.
  /// Otherwise, it could potentially check local cache first (though this impl focuses on remote).
  Future<List<User>> fetchAndCacheByQuery({
    required Query Function(Query query) queryModifier,
    Duration? cacheDuration, // Pass Duration.zero to force remote fetch
  }) async {

    // Determine effective duration for any potential pre-check (optional)
    final effectiveDuration = cacheDuration ?? cacheExpiryDuration;

    // --- Force Remote Fetch Logic ---
    // If Duration.zero is passed, we *must* fetch from remote.
    // Even if not zero, this specific method's primary goal is often a targeted remote fetch.
    // A more complex version could try to satisfy the query locally first if cache is valid.
    bool forceRemote = (effectiveDuration == Duration.zero);

    // TODO (Optional Advanced): Implement local cache check first.
    // If !forceRemote, try querying local Drift based on a translation of the
    // Firestore queryModifier. If results are found and NOT expired based on
    // effectiveDuration, return them. This is complex to implement generally.

    // --- Proceed with Remote Fetch ---
    print("Fetching users from remote with custom query (Forced: $forceRemote)...");
    if (!await CacheManager.hasConnectivity()) {
      throw Exception('No internet connection to fetch users by query.');
    }

    try {
      // Use the base fetchAllFromRemote, applying the modifier.
      final List<User> remoteUsers = await fetchAllFromRemote(
        'users', // Firestore collection path
        queryModifier: queryModifier, // Apply the custom query logic
      );

      // Update the local cache with the fetched users
      print("Updating local cache with ${remoteUsers.length} fetched users...");
      for (final user in remoteUsers) {
        await saveToLocal(user); // saveToLocal handles insert or replace
      }

      // Return the list of users fetched from the remote source
      return remoteUsers;

    } catch (e) {
      print("Failed to fetch users by query: $e");
      throw Exception('Failed to fetch users by query: $e');
    }
  }
}


// --- 2. Updated UserRepository ---
// Imports remain the same (cloud_firestore, drift, http, shared_preferences, local files)
// Placeholder models (UserWithActivity, PendingUpdate) remain the same.

class UserRepository {
  final AppDatabase db;
  late final UsersCacheManager cacheManager;
  final String _activityApiBaseUrl = "https://api.myapp.example/activity";
  final String _offlineQueueKey = 'user_offline_updates_queue';

  UserRepository(this.db) {
    cacheManager = UsersCacheManager(database: db);
    // _initRealtimeListeners(); // Placeholder
  }

  // --- Core Caching Methods ---

  /// Gets a single user. Uses local cache unless expired or [overrideDuration] forces refresh.
  /// Pass `Duration.zero` to force a remote fetch for this user.
  Future<User> getUser(String uid, {Duration? overrideDuration}) async {
    // No change needed here - it correctly uses the temporary manager
    // whose `isCacheExpired` handles Duration.zero.
     if (overrideDuration != null) {
      print("getUser: Using temporary UsersCacheManager with duration: $overrideDuration");
      final tempCacheManager = UsersCacheManager(db: db, cacheExpiryDuration: overrideDuration);
      // getById internally calls isCacheExpired which respects Duration.zero
      return await tempCacheManager.getById(uid);
    } else {
       print("getUser: Using default UsersCacheManager with duration: ${cacheManager.cacheExpiryDuration}");
      return await cacheManager.getById(uid);
    }
  }

  /// Gets users based on a specific Firestore query.
  /// **Always fetches directly from Firestore**, bypassing local cache checks for this query.
  /// Updates the local cache with the results.
  Future<List<User>> getUsersByQuery(
    Query Function(Query query) queryModifier,
  ) async {
    // **MODIFIED:** Call the new manager method, explicitly passing Duration.zero
    // to signal that we want to force a remote fetch for this query,
    // regardless of any potential local cache state for these items.
    print("getUsersByQuery: Forcing remote fetch.");
    return await cacheManager.fetchAndCacheByQuery(
      queryModifier: queryModifier,
      cacheDuration: Duration.zero, // <-- Force remote fetch for this query
    );
  }

  /// Gets all relevant users, syncing from remote *if the global cache is expired*.
  /// Returns all users currently stored in the local database after the potential sync.
  Future<List<User>> getAllRelevantUsers() async {
    // No change - uses standard manager sync logic based on global expiry
    print("Getting all relevant users (syncing if global cache expired)...");
    return await cacheManager.syncAndGetAllUsers();
  }

  // --- Extensibility Demonstrations (Remain Functionally the Same) ---

  /// 1. COMBINING DATA SOURCES: Get user data and combine with recent activity from a REST API.
  Future<UserWithActivity> getUserWithRecentActivity(String uid, {Duration? overrideDuration}) async {
    // Implementation unchanged - relies on getUser which handles caching/overrides
    final user = await getUser(uid, overrideDuration: overrideDuration);
    List<String> activity = [];
    try {
      activity = await _fetchUserActivityFromApi(uid); // Placeholder API call
    } catch (e) { /* Handle error */ }
    return UserWithActivity(user: user, recentActivity: activity);
  }

   // Placeholder for API call logic (unchanged)
  Future<List<String>> _fetchUserActivityFromApi(String uid) async {
     await Future.delayed(Duration(milliseconds: 300)); // Simulate delay
     return ["Activity 1", "Activity 2"]; // Placeholder data
  }


  /// 2. OFFLINE MUTATIONS: Attempt to update profile, queue if offline.
  Future<void> updateUserProfile(String uid, Map<String, dynamic> updatedData) async {
     // Implementation unchanged - handles online/offline logic
      if (await CacheManager.hasConnectivity()) {
         print("Online: Updating profile for $uid in Firestore and cache.");
         await cacheManager.firestore.collection('users').doc(uid).update(updatedData);
         final freshUser = await cacheManager.fetchFromRemote(uid);
         await cacheManager.saveToLocal(freshUser);
      } else {
        print("Offline: Queueing profile update for $uid.");
        final pendingUpdate = PendingUpdate(
            entityId: uid, collection: 'users', data: updatedData, type: 'update', queuedAt: DateTime.now());
        await _queueOfflineUpdate(pendingUpdate); // Placeholder queueing
      }
  }

   // Placeholder for queueing logic (unchanged)
   Future<void> _queueOfflineUpdate(PendingUpdate update) async {
       final prefs = await SharedPreferences.getInstance();
       final queueJson = prefs.getStringList(_offlineQueueKey) ?? [];
       queueJson.add(jsonEncode(update.toJson()));
       await prefs.setStringList(_offlineQueueKey, queueJson);
   }

  /// 2b. OFFLINE MUTATIONS: Process the queue when connectivity returns.
  Future<void> syncOfflineUpdates() async {
    // Implementation unchanged - processes the simulated queue
     if (!await CacheManager.hasConnectivity()) return;
     print("Attempting to sync offline updates...");
     final prefs = await SharedPreferences.getInstance();
     List<String> queueJson = prefs.getStringList(_offlineQueueKey) ?? [];
     if (queueJson.isEmpty) return;
     // ... (rest of processing logic remains the same) ...
     List<String> remainingQueue = List.from(queueJson);
     // Loop, try update, remove from remainingQueue on success
     await prefs.setStringList(_offlineQueueKey, remainingQueue);
  }


  /// 3. REAL-TIME INTEGRATION: Placeholder where listener updates would be processed.
   void _handleRealtimeUserUpdate(Map<String, dynamic> firestoreData, String docId) {
     // Implementation unchanged - maps data and saves to local cache
     print("Received real-time update for user $docId");
      try {
         firestoreData['id'] = docId;
         final updatedUser = cacheManager.mapFirestoreToEntity(firestoreData);
         cacheManager.saveToLocal(updatedUser); // Fire and forget or await/handle
      } catch (e) {/* Handle mapping error */}
   }
   // Placeholder for listener setup remains commented out


  /// 4. SELECTIVE SYNCING: Fetch and cache only specific users from remote.
  Future<List<User>> syncSpecificUsers(List<String> userIds) async {
     // Implementation unchanged - uses fetchAllFromRemote with 'ids'
     if (userIds.isEmpty) return [];
     if (!await CacheManager.hasConnectivity()) {
       print("No connection to sync specific users.");
       return (db.select(db.users)..where((u) => u.uid.isIn(userIds))).get();
     }
     print("Syncing specific users: ${userIds.join(', ')}");
     final List<User> remoteUsers = await cacheManager.fetchAllFromRemote('users', ids: userIds);
     for (final user in remoteUsers) await cacheManager.saveToLocal(user);
     return remoteUsers;
  }

  /// 5. COMPLEX LOCAL QUERY: Find users marked as admin directly from local cache.
  Future<List<User>> findAdminUsersLocally() async {
    // Implementation unchanged - queries local Drift DB
     print("Querying local cache for admin users...");
     try {
        // Ensure Users table has 'isAdmin' column defined in Drift schema
       final query = db.select(db.users)..where((tbl) => tbl.isAdmin.equals(true));
       return await query.get();
     } catch (e) {
        print("Error querying local admin users: $e");
        return [];
     }
  }
}

Strengths

  • Reduced Backend Load: Significantly cuts down on Firestore reads for frequently accessed, relatively stable data.
  • Improved Perceived Performance: Reading from the local SQLite DB is much faster than network requests.
  • Basic Offline Support: Users can view previously cached data even when offline (though it might be stale).
  • Backend Flexibility: The CacheManager abstraction makes it easier to swap Firestore for another backend (or use multiple sources) by just changing the implementation of WorkspaceFromRemote, mapFirestoreToEntity, etc.
  • Data Normalization: Allows cleaning/enhancing data (like Connections) before it's used in the UI.

Weaknesses

  • Complexity: It's a non-trivial amount of boilerplate code to set up for each entity type.
  • Potential Stale Data: Data can be stale between cache expiry intervals. Not suitable for real-time data unless combined with streams/listeners (which adds more complexity).
  • Manual Cache Invalidation: While expiry helps, explicit invalidation might be needed after mutations (e.g., updating a user profile requires refetching that user). The example updateUserProfile shows one way to handle this.
  • Storage Usage: Storing data locally consumes device storage.
  • Initial Sync: The first time a user opens the app or after a long time, syncing all data might take a moment.

Simplicity/Complexity

  • Moderate to High. While the concept is straightforward, the implementation requires careful handling of asynchronous operations, database interactions, state management, error handling (especially offline scenarios), and mapping between different data models (Firestore Maps vs. Drift Typed Objects).

Potential Improvements

  • Real-time Updates: Integrate Firestore listeners (snapshots()) to push real-time updates to the local cache instead of relying purely on time-based expiry. This would involve merging snapshot data into the Drift DB.
  • Smarter Syncing: Implement delta-syncing (fetching only changed records since the last sync) if your backend supports it, instead of fetching all records on expiry.
  • Background Sync: Use background processing (like workmanager) to update the cache periodically even when the app isn't active.
  • Refined Error Handling: More granular error handling for offline vs. actual server errors.
  • Query Caching: Cache results of specific queries, not just individual entities or full table syncs.
  • Code Generation: Potentially use build_runner or other code-gen tools to reduce the boilerplate for CacheManager implementations.

Applying the Cache: Handling Complexity in an Asynchronous Video App (Reely)

Having detailed this caching and data synchronization strategy, the natural question is: what kind of application demands such an approach? This system was designed specifically to meet the challenges of Reely, a mobile-first platform built for asynchronous video conversations. On the surface, the concept is intuitive – users reply to friends with short videos presented in horizontally scrolling, feed-style conversations, much like a conversational TikTok. Users record, add creative effects, and swipe through replies. Simple, right?

However, the underlying requirements to make this feel effortless and instantaneous, bridging the gap between asynchronous messaging and real-time responsiveness, introduce significant complexity. Managing individual read-progress pointers across numerous conversations, ensuring immediate access to video metadata for smooth playback, and keeping the UI snappy requires more than simple data fetching. Delivering that smooth, "always flowing" experience necessitates the robust caching and background orchestration previously discussed, aiming to make it feel like the core backend logic is running right on the user's device.

Consider these challenges inherent to Reely's design:

  1. Asynchronous State Management: Unlike a simple chat app, Reely needs to meticulously track each user's position (the chronological pointer) within potentially dozens or hundreds of ongoing video conversations. Syncing this state reliably across devices and sessions is critical so users can always pick up exactly where they left off.
  2. Seamless Media Handling: Users expect instant playback when they swipe to a new video reply and automatic advancement. This demands pre-fetching or aggressive caching of video segments and metadata, anticipating user navigation within the horizontal feed.
  3. Responsiveness Expectations: Even though the conversations are asynchronous, the UI interactions -- opening conversations, navigating replies, accessing profiles, applying creative tools -- need to feel instantaneous, matching the responsiveness users expect from top-tier social apps. Network latency cannot be a constant blocker.
  4. Data Volume: Video conversations generate significantly more data than text chats. Efficiently managing the storage, retrieval, and caching of video metadata, user profiles, connection statuses, conversation participants, and read-state pointers is crucial for performance and scalability.

This is where the caching system becomes fundamental, not just an optimization. The pull-through cache we discussed plays a vital role:

  • Reduces Latency: Fetching user profiles, connection data, conversation metadata, and pointer positions from the local Drift cache is orders of magnitude faster than hitting Firestore every time.
  • Enhances Responsiveness: UI elements can populate instantly with cached data while fresh data is potentially fetched in the background based on cache expiry rules or specific triggers (like receiving a notification).
  • Manages State: The local database acts as a reliable source for the user's current state within conversations, even if temporarily offline.
  • Offline Capability: Users can still browse previously viewed parts of conversations when offline, thanks to the local cache.
  • Reduces Backend Load: Significantly cuts down on Firestore read operations, saving costs and ensuring the backend remains responsive under load.

Creating that "Reely You," raw, real-time feeling in an asynchronous environment requires this constant, non-trivial background dance of checking connectivity, validating cache freshness, fetching from remote sources, updating the local database, and ensuring the UI always has the data it needs before the user explicitly asks for it. It's this hidden complexity that allows users to just focus on the vibe and the conversation.

Reely is currently in beta! If you're interested in seeing this system in action and trying out a new way to have video convos, you can join the beta at https://getreely.app. We'd love feedback, especially from fellow developers!

Conclusion

This pull-through cache pattern provides a solid balance between performance, offline capability, and backend efficiency for many common app scenarios. It requires some setup effort but offers significant benefits, especially for data that doesn't need to be strictly real-time. Remember to adapt the cache durations and sync strategies based on how frequently your data changes and how critical freshness is for each entity type.

Hope this helps someone looking into similar data management challenges! Let me know your thoughts or alternative approaches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment