Pipeline.md

IFC to GLB Conversion Pipeline

Architecture Diagram

graph TB
  subgraph Client
    A[User Upload IFC File]
  end
  
  subgraph "Node.js/Express Backend"
    B[API Server]
    D[Status Endpoint]
    E[Documentation Endpoint]
  end
  
  subgraph "AWS Services"
    F[S3 Bucket]
    G[SQS Queue]
    H[RDS MySQL]
    
    subgraph "Processing Worker"
      I[SQS Poller]
      J[Puppeteer]
      K[Online3DViewer]
      L[IFC to GLB Converter]
    end
  end
  
  A -->|Upload IFC| B
  B -->|Store File| F
  B -->|Create Job| H
  B -->|Send Message| G
  G -->|Poll Messages| I
  I -->|Process Job| J
  J -->|Launch Browser| K
  K -->|Convert Format| L
  L -->|Upload Result| F
  L -->|Update Status| H
  L -->|Cleanup Temp Files| L
  Client -->|Check Status| D
  D -->|Query| H
  Client -->|View Docs| E
  
  classDef aws fill:#FF9900,stroke:#232F3E,stroke-width:2px,color:white;
  classDef nodejs fill:#68A063,stroke:#2e2e2e,stroke-width:2px,color:white;
  classDef client fill:#4285F4,stroke:#2e2e2e,stroke-width:2px,color:white;
  
  class F,G,H aws;
  class B,D,E,I,J,K,L nodejs;
  class A client;

Loading

Implementation Flow

1. Client uploads IFC file through the Node.js/Express API
2. Backend processes the upload:
   • Stores the original file in S3 (optimized for larger files)
   • Creates a job record in MySQL database
   • Sends a message to SQS queue with job details
3. Processing Worker:
   • Continuously polls SQS for new messages
   • When a message is received, starts the conversion process
   • Uses Puppeteer to launch headless browser
   • Leverages existing Online3DViewer code to convert IFC to GLB
   • Updates job status in MySQL database
   • Uploads resulting GLB file to S3
   • Cleans up temporary files
4. Status Endpoint allows clients to check conversion progress
5. API Documentation provides usage information

Technical Components

Backend (Node.js/Express)

• Handles file uploads and API requests
• Manages S3 interactions (HTTPS-enabled)
• Implements comprehensive error handling and logging
• Provides status checking endpoint
• Serves API documentation

AWS Services

• S3: Stores both input IFC and output GLB files
• SQS: Manages job queue for asynchronous processing
• RDS (MySQL): Tracks job status and metadata

Processing Worker

• Utilizes Puppeteer for headless browsing
• Integrates with existing Online3DViewer code
• Performs the IFC to GLB conversion
• Updates job status in database
• Handles error cases and retries

Database Schema

jobs
├── id (primary key)
├── status (pending, processing, completed, failed)
├── input_path (S3 path to IFC file)
├── output_path (S3 path to GLB file)
├── created_at
├── updated_at
└── error_message (null unless failed)