Complete Data Import Service: A Comprehensive Guide

by Alex Johnson 52 views

As a Tempo user, the ability to import a complete Tempo export file is crucial for data restoration, migration to new instances, and recovery from data loss. This article delves into the intricacies of implementing a complete data import service, ensuring that users can seamlessly restore their data exactly as it was exported. This includes workouts, media files, settings, shoes, and calculated data. Let's explore the essential aspects of this feature, from acceptance criteria to technical requirements.

Overview of the Data Import Service

The data import service is designed to provide Tempo users with a robust mechanism for restoring their data from backup files, facilitating seamless migration to new instances, and ensuring quick recovery from potential data loss scenarios. This feature is critical for maintaining data integrity and ensuring business continuity. The service will support the import of a complete Tempo export ZIP file, which contains a comprehensive snapshot of the user's data.

The primary goal is to restore the user's data precisely as it was at the time of export. This includes a wide range of data types, such as workout logs, media files (photos and videos), user settings (like heart rate zones and unit preferences), shoe information, and various calculated metrics. By providing a comprehensive import capability, users can avoid data inconsistencies and ensure a smooth transition when restoring or migrating their data.

To achieve this, the data import service must address several key challenges. First, it needs to handle large volumes of data efficiently. Tempo users often accumulate significant amounts of data over time, including numerous workout sessions and media files. The import process must be optimized to minimize the time required for data restoration. Second, the service must maintain data integrity throughout the import process. This means ensuring that all relationships between different data entities (e.g., shoes associated with workouts, media files linked to specific sessions) are correctly preserved. Finally, the service must provide clear and informative feedback to the user, including progress updates, success notifications, and error reports.

The data import service will be a valuable addition to Tempo, enhancing its appeal to users who require reliable data management capabilities. Whether users are backing up their data for security, migrating to a new Tempo instance, or recovering from data loss, this feature will provide the necessary tools to ensure a seamless and efficient process. By prioritizing data integrity, performance, and user feedback, the service will significantly improve the overall user experience and reinforce Tempo's commitment to data reliability.

Acceptance Criteria for the Data Import Service

For a data import service to be considered successful, it must meet a comprehensive set of acceptance criteria. These criteria ensure that the service functions reliably, efficiently, and provides a seamless user experience. The acceptance criteria cover various aspects of the import process, from file handling and data restoration to error management and user feedback.

Firstly, the service must allow users to easily upload an export ZIP file directly from the Settings page. This integration within the user interface ensures that the import functionality is readily accessible. Once the file is uploaded, the service should validate its format and version. This validation step is crucial for ensuring compatibility and preventing issues caused by corrupted or outdated files. The system should verify that the uploaded file adheres to the expected structure and format, including the presence of necessary metadata files.

Data restoration is a core function, and the service must accurately restore all user settings, including heart rate zones, unit preferences, and default shoe selections. These settings are essential for personalized tracking and analysis, and their preservation ensures continuity for the user. Similarly, all shoe records, along with their associated relationships to workouts, must be restored correctly. This includes details such as shoe brand, model, and mileage tracking, which are important for monitoring equipment usage and performance.

The import process must also restore all workout data, including complete session details, routes (maps), splits, and time series data. Workout data forms the core of Tempo's functionality, and its accurate restoration is vital for maintaining historical records and generating meaningful insights. This includes ensuring that workout routes and maps are correctly imported, allowing users to review their past activities on a geographical basis. Workout splits, which provide detailed performance metrics for specific segments of a workout, and time series data, which capture continuous measurements like heart rate and pace, must also be accurately restored.

In addition to workout data, the service must handle media files, including photos and videos, which users often associate with their workouts. These files should be imported and linked to the correct workout sessions, allowing users to relive their experiences. Furthermore, the service should restore raw workout files in various formats, such as GPX, FIT, and CSV, which contain detailed sensor data. Best efforts, representing peak performance metrics for various activities, must also be restored to maintain a complete record of user achievements.

Data integrity is a paramount concern, and the import service must preserve all relationships between different data entities. For example, the service must ensure that shoes are correctly linked to the workouts in which they were used, and that media files are properly associated with their corresponding workouts. Handling duplicate data is another critical aspect. The service should implement a mechanism for detecting duplicates and provide options for either skipping them or updating existing records, ensuring that data is consistent and accurate.

Large imports can be time-consuming, and the service should provide progress updates to keep users informed about the status of the import process. This feedback is essential for managing user expectations and preventing frustration. Detailed feedback on what was imported, including statistics and summaries, should also be provided at the end of the import process. Finally, the service must handle errors gracefully. It should be able to resume or rollback imports in case of failures, preventing data corruption and ensuring a smooth recovery process. Data integrity should be validated before committing any changes, adding an extra layer of protection against data loss.

Import Process Flow: A Step-by-Step Guide

The import process flow is a structured sequence of steps designed to ensure the seamless and accurate restoration of user data. This process involves several critical stages, from the initial upload and validation of the export file to the final verification of imported data. Each step plays a vital role in maintaining data integrity and providing a smooth user experience. Let's break down each stage of the import process in detail.

The first stage, Upload & Validation, is where the user initiates the data import by uploading a ZIP file containing their exported data. The service begins by extracting the contents of the ZIP file and validating a manifest file, typically named manifest.json. This file contains essential metadata about the export, such as the export version, date, and user information. The service checks the export version to ensure compatibility with the current import functionality, preventing issues caused by outdated or incompatible export formats. The ZIP file's structure is also validated to ensure that it conforms to the expected directory structure and contains all necessary files.

Next is the Data Validation stage, which focuses on ensuring the integrity and consistency of the data to be imported. The service validates that all JSON files within the export are well-formed and adhere to the JSON syntax. Required fields within these files are checked to ensure that no critical information is missing. The service also validates relationships between different data entities. For example, it verifies that shoes exist before workouts reference them, ensuring that there are no orphaned references. File references, such as media files and raw workout files, are checked to confirm that they exist within the ZIP archive, preventing broken links and missing data.

The Import Execution stage is where the actual data restoration takes place. This stage is often performed within a transaction, if possible, to ensure that the import process is atomic—either all data is imported successfully, or the entire operation is rolled back in case of an error. This transactional approach is crucial for maintaining data integrity. The import process follows a specific order to maintain referential integrity. User settings are imported first, as they have no dependencies. Shoes are imported next, followed by workouts, workout routes, splits, time series data, and best efforts. Media files are then imported and written to the filesystem, while raw files are stored in the database. This order ensures that all relationships between data entities are correctly preserved.

Finally, the Verification stage confirms that all data has been imported correctly. The service checks the integrity of the imported data and generates an import report summarizing the results. This report includes statistics on the number of items imported, skipped, or encountered errors. By verifying the data integrity and providing a detailed report, the service ensures that the user can trust the restored data and identify any potential issues.

Data Import Order: Maintaining Referential Integrity

The data import order is a critical aspect of the import process, essential for maintaining referential integrity within the system. Referential integrity ensures that relationships between different data entities are correctly preserved during the import. Importing data in the wrong order can lead to inconsistencies, broken links, and data corruption. Therefore, a specific import sequence must be followed to guarantee data accuracy and reliability.

The import process begins with User Settings. These settings can be imported first because they have no dependencies on other data entities. User settings include preferences such as heart rate zones, unit preferences, and default shoe selections. Importing these settings at the outset ensures that the user's personalized configurations are in place before other data is imported.

Next, Shoes must be imported. Shoes have a critical relationship with workouts, as each workout may be associated with a specific pair of shoes. Importing shoes before workouts ensures that when workouts are imported, the system can correctly link them to the corresponding shoe records. This prevents orphaned workout records and maintains accurate equipment usage data.

The third step involves importing Workouts, which form the core of the user's activity data. Workouts contain essential information about each training session, such as start time, duration, distance, and performance metrics. Because workouts may reference shoes, it is crucial that shoes are imported first. Once workouts are imported, related data such as workout routes, splits, and time series data can be imported in subsequent steps.

Following workouts, Workout Routes are imported. Workout routes represent the geographical path taken during a workout and are dependent on the workout records. Importing routes after workouts ensures that each route can be correctly associated with its corresponding workout session. Similarly, Workout Splits, which provide detailed performance metrics for different segments of a workout, are imported after workouts. This ensures that each split is correctly linked to its workout.

Workout Time Series data, which includes continuous measurements like heart rate and pace, is also dependent on workout records and must be imported after workouts. Time series data provides a granular view of performance throughout the workout, and its correct association with workout records is essential for accurate analysis. Best Efforts, representing peak performance metrics for various activities, are also imported after workouts. Best efforts are derived from workout data, so workouts must be imported first to ensure that these metrics can be correctly calculated and associated.

Finally, Media Files and Raw Files are imported. Media files, such as photos and videos, are often associated with specific workouts. These files are typically written to the filesystem and linked to workout records. Raw files, such as GPX, FIT, and CSV files, contain detailed sensor data from workouts. These files are often stored in the database as part of the workout record. Importing media and raw files last ensures that all workout data is in place, allowing for correct association and storage of these files.

By adhering to this specific import order, the service ensures that all relationships between data entities are correctly maintained, preserving the integrity and accuracy of the user's data. This systematic approach minimizes the risk of data inconsistencies and ensures a reliable data restoration process.

Duplicate Handling: Strategies and Detection Methods

Duplicate handling is a crucial aspect of the data import service, ensuring data integrity and preventing redundancy. When importing data, there is a potential for duplicate records to exist, either within the imported data itself or between the imported data and existing data in the system. Implementing effective duplicate handling strategies and detection methods is essential for maintaining a clean and accurate dataset. This section explores various strategies for handling duplicates and the methods used to detect them.

Strategy Options

  1. Skip Duplicates (Default) This strategy involves comparing imported records with existing records and skipping the import of any duplicates. It is the default strategy because it prioritizes preserving existing data and avoiding accidental overwrites. The comparison criteria for determining duplicates vary depending on the data type. For workouts, the comparison typically involves StartedAt, DistanceM, and DurationS values, with a small tolerance to account for minor discrepancies. If a duplicate is found, the item is skipped, and a log entry is created to record the skipped item. This approach ensures that the existing data remains unchanged while providing a record of the duplicate entries.

  2. Update Existing This strategy involves identifying duplicates and updating the existing records with the data from the imported records. If a duplicate is found, the existing workout record is updated with the imported data, preserving the existing GUID (Globally Unique Identifier). This approach is useful for synchronizing data and ensuring that the most current information is reflected in the system. Updating related data, such as routes, splits, and time series data, is also crucial to maintain consistency. This strategy requires careful implementation to avoid data loss or corruption, but it can be beneficial in scenarios where data synchronization is essential.

  3. User Choice This strategy provides the user with the flexibility to choose how duplicates are handled. When a duplicate is detected, the user is prompted to select one of the following options: Skip, Update, or Import as New. This approach allows users to make informed decisions based on their specific needs and preferences. The chosen action is then applied consistently across all duplicates encountered during the import process. This strategy is the most user-centric, but it requires a more complex implementation to manage user input and apply the chosen action across multiple duplicates.

Duplicate Detection

Duplicate detection is the process of identifying records that are considered duplicates based on specific criteria. The criteria vary depending on the data type.

  • Workouts: Duplicates are typically detected by comparing the StartedAt, DistanceM, and DurationS values. A small tolerance is applied to these values to account for minor variations. If two workouts have similar start times, distances, and durations, they are considered duplicates.
  • Shoes: Duplicates are detected by comparing the Brand and Model fields. An exact match is required for both fields to consider two shoe records as duplicates. This ensures that only shoes with the same brand and model are treated as duplicates.
  • Settings: Since only one settings record exists per user, any imported settings record will replace the existing record. There is no need for duplicate detection in this case.
  • Best Efforts: Duplicates are detected based on the Distance field. Only one best effort record per distance is allowed, so if an imported record has the same distance as an existing record, it is considered a duplicate.

By implementing these duplicate handling strategies and detection methods, the data import service can maintain data integrity, prevent redundancy, and provide a reliable data restoration process.

Import Format Validation: Ensuring Data Integrity

Import format validation is a critical step in the data import process, ensuring that the data being imported is in the correct format and meets the required standards for integrity. This process involves a series of checks and validations to prevent corrupted, incomplete, or invalid data from being imported into the system. Proper validation ensures that the data import service functions reliably and maintains the integrity of the existing data. This section details the various aspects of import format validation.

The validation process begins with the Manifest File. The manifest file, typically a JSON file named manifest.json, contains metadata about the export, such as the export version, date, and user information. The import service must check for the presence of required fields within the manifest file. These fields are essential for identifying and processing the export. The service also verifies version compatibility to ensure that the export format is supported by the current import functionality. Incompatible versions may require specific migration steps or may not be supported at all. The export date and user information are validated to provide context for the import and ensure that the data belongs to the correct user.

Next, the ZIP Structure is validated. The import service checks for the existence of required directories within the ZIP archive. These directories typically contain different types of data, such as workout data, media files, and settings. The presence of these directories ensures that the export has a consistent structure and contains all the necessary components. The service also verifies that the required JSON files exist within the appropriate directories. These JSON files contain the actual data to be imported, and their presence is crucial for a successful import. The file paths within the ZIP archive are also validated to ensure that they are correct and consistent with the expected structure.

Data Integrity validation is a critical step in ensuring the quality and consistency of the imported data. The service validates that all JSON files are valid JSON, adhering to the JSON syntax and structure. Invalid JSON files can cause import failures and data corruption. The presence of required fields within the JSON files is also checked. These fields contain essential data elements, and their absence can lead to incomplete or inaccurate imports. The data types of the fields are validated to ensure that they match the expected types. For example, numeric fields should contain numbers, and date fields should contain valid date values. GUIDs (Globally Unique Identifiers) are validated to ensure that they are in the correct format and are unique. Invalid GUIDs can indicate data corruption or inconsistencies. Relationships between different data entities are validated to ensure that referenced entities exist. For example, if a workout references a shoe, the shoe must exist in the imported data. This validation step helps maintain referential integrity and prevents broken links.

Finally, File References are validated. The service verifies that media files referenced in the metadata exist within the ZIP archive. This ensures that media files, such as photos and videos, are correctly associated with their corresponding workouts. Raw files referenced in workouts are also checked to ensure that they exist in the ZIP archive. Raw files contain detailed sensor data and are essential for complete workout records.

By performing these import format validation steps, the service ensures that only valid and consistent data is imported, maintaining the integrity and reliability of the system.

Technical Requirements for a Robust Data Import Service

Implementing a robust data import service requires careful consideration of several technical requirements. These requirements span various aspects of the system, including API endpoints, service implementation, data handling, and error management. Meeting these requirements ensures that the import service is efficient, reliable, and secure. This section outlines the key technical requirements for a comprehensive data import service.

API Endpoint

The service must expose an API endpoint that allows users to initiate the data import process. The recommended specifications for this endpoint are:

  • Endpoint: POST /workouts/import/export
  • Authentication: Required (user must be logged in)
  • Request: Multipart form data
    • File: ZIP file (up to 500MB, same as bulk import)
  • Response: JSON with import results
    • Statistics (imported, skipped, errors)
    • Detailed report

This API endpoint uses the HTTP POST method to handle the file upload. Authentication is required to ensure that only authorized users can initiate the import process. The request is formatted as multipart form data, which is suitable for file uploads. The file should be a ZIP archive, with a maximum size of 500MB, consistent with bulk import limits. The response is a JSON object containing import results, including statistics on the number of items imported, skipped, and encountered errors, as well as a detailed report for further analysis.

Service Implementation

The core logic of the import service is implemented in the ImportService class, typically located in the api/Services/ImportService.cs file. The responsibilities of this service include:

  • Orchestrating the import process
  • Validating the export format
  • Extracting the ZIP file
  • Parsing JSON files
  • Importing data in the correct order
  • Handling errors and rollback

The ImportService acts as the central coordinator for the import process. It manages the various steps involved, from initial validation to final data import. The service must ensure that the data is imported in the correct order to maintain referential integrity. Error handling and rollback mechanisms are crucial to prevent data corruption in case of failures.

Data Deserialization

Data deserialization is the process of converting JSON data into objects that can be used by the system. The following considerations apply:

  • Use System.Text.Json for JSON deserialization
  • Handle GUIDs and relationships
  • Validate data types
  • Handle missing/null values

System.Text.Json is the recommended library for JSON deserialization due to its performance and security benefits. The deserialization process must handle GUIDs and relationships between data entities. Data types should be validated to ensure that they match the expected types. Missing or null values should be handled gracefully to prevent errors and maintain data integrity.

File Handling

File handling is a critical aspect of the import service, particularly for media files and raw workout data. The following requirements apply:

  • Extract media files from ZIP
  • Write media files to filesystem (media/{workoutId}/)
  • Store raw files in RawFileData byte array
  • Validate file sizes and types
  • Handle file system errors

Media files are extracted from the ZIP archive and written to the filesystem, typically in a directory structure based on the workout ID (media/{workoutId}/). Raw files, such as GPX, FIT, and CSV files, are stored in the RawFileData byte array within the database. File sizes and types should be validated to prevent security vulnerabilities and ensure data integrity. File system errors should be handled gracefully to prevent import failures.

Database Operations

Database operations are at the core of the data import service, ensuring that data is stored correctly and efficiently. The following requirements apply:

  • Use transactions where possible
  • Batch inserts for performance
  • Handle foreign key constraints
  • Preserve GUIDs from export
  • Update timestamps appropriately

Transactions should be used where possible to ensure atomicity—either all database operations succeed, or none do. Batch inserts can improve performance when importing large volumes of data. Foreign key constraints should be handled to maintain referential integrity. GUIDs from the export should be preserved to maintain relationships between data entities. Timestamps should be updated appropriately to reflect the import time or preserve the original timestamps, depending on the requirements.

Error Handling

Robust error handling is essential for a reliable data import service. The following requirements apply:

  • Validate before importing
  • Use transactions for atomicity
  • Rollback on critical errors
  • Continue with non-critical errors (log and report)
  • Provide detailed error messages

Data should be validated before importing to prevent errors. Transactions should be used to ensure atomicity. Critical errors should trigger a rollback to prevent data corruption. Non-critical errors should be logged and reported, allowing the import process to continue while documenting the issues. Detailed error messages should be provided to help users and administrators diagnose and resolve issues.

Progress Reporting

Progress reporting is crucial for providing feedback to users during the import process. The following requirements apply:

  • Stream progress updates if possible
  • Report statistics as import progresses
  • Final summary report

Progress updates should be streamed to the user interface if possible, providing real-time feedback on the import progress. Statistics should be reported as the import progresses, including the number of items imported, skipped, and encountered errors. A final summary report should be generated at the end of the import process, providing a comprehensive overview of the import results.

Conclusion

In conclusion, implementing a complete data import service for Tempo is a complex but essential undertaking. By carefully considering the acceptance criteria, import process flow, data import order, duplicate handling strategies, import format validation, and technical requirements, a robust and reliable service can be developed. This service will empower Tempo users to seamlessly restore their data, migrate to new instances, and recover from data loss, ensuring a consistent and reliable user experience.

For further reading on best practices for data import and export, consider exploring resources available on platforms like AWS Data Migration Service.