Evaluation Protocol

Simulation Fidelity Can Be Measured

SFR is evaluated through a repeatable protocol based on motion accuracy, timing, and system coherence.

A simulation system must be evaluated under controlled conditions to determine whether it preserves correct motion, timing, and sensory alignment. The SFR framework defines how this evaluation is performed.

Definition Architecture Measurement Classification Consequences Impact Evaluation Determination

Purpose of Evaluation


The purpose of the evaluation protocol is to determine whether a simulation system produces motion and sensory input that align with real-world vehicle dynamics and human perception.

Evaluation is required because appearance does not indicate correctness.

Standard Test Environment


All evaluation must occur under controlled and consistent conditions. Variation in test conditions invalidates comparison between systems.

All systems must be evaluated under identical conditions to ensure comparability.

Measurement Categories


SFR is composed of four measurement categories. Each must be assessed individually. The overall SFR score reflects performance across all four.

Category 1

Degrees of Freedom (DOF)

Measures whether rotational and translational axes exist independently and align with the vehicle's center of mass.

Category 2

Vestibular Load

Measures whether motion provides correct physical cues for rotation and acceleration that the vestibular system can interpret.

Category 3

Synchronization (Sync)

Measures alignment between motion, visuals, and control input timing across all hardware and software layers.

Category 4

Unified System Behavior

Measures whether all systems operate as a single coherent environment rather than a collection of disconnected outputs.

Each category contributes to the overall SFR score.

Extended Metrics


These metrics are applied only when validated data is available. They extend the baseline SFR evaluation but are not required for standard classification.

G — Force Output

Measured physical force output relative to vehicle events, available only when instrumented data exists.

NA — Neurological Accuracy

Measured alignment between simulator response and real-world human response, available only when independently validated.

Total Fidelity = SFR × G × (1 + NA)

Extended metrics enhance evaluation but are not required for baseline classification.

Data Collection


Standard Inputs
  • Motion system telemetry
  • Vehicle physics output
  • Visual system timing
  • Control input timing
  • Sensor data (if available)
Optional Advanced Inputs
  • Eye tracking
  • Reaction timing analysis
  • Biometric feedback

Advanced inputs enable extended metric calculations when validated.

All data must be time-aligned for accurate analysis.

Test Procedures


1

Initialize system under controlled conditions

2

Run predefined motion scenarios

3

Capture synchronized system data

4

Compare motion output to physics reference

5

Evaluate timing alignment across systems

6

Score each SFR category

7

Compile total SFR score

The same procedure must be applied to all systems being evaluated.

Scoring


Each category is scored on a normalized scale. The total SFR score represents overall system fidelity and is used to assign the system to an interpretation band.

High Alignment

Consistent with in-the-loop systems

Moderate Alignment

Partial fidelity, limited transfer

Low Alignment

Surface-level or non-transferable

Scores must be interpreted alongside system classification.

From Score to Classification


SFR scores inform system classification. Score bands correspond directly to classification categories.

View System Classification

Repeatability


A valid evaluation must produce consistent results across repeated tests. Any system whose performance varies substantially between identical test runs cannot be reliably classified.

Repeatability is required for credibility.

Limitations of Evaluation


Evaluation is dependent on available measurement tools and data quality. Results must be interpreted within these constraints.

Absence of data does not confirm accuracy.

Determination


A simulation system must be evaluated through a structured protocol to determine its fidelity and classification.

If it cannot be measured, it cannot be validated.

Continue Through the Framework


Application Layer

Request Evaluation

Apply the framework to a real system, environment, or use case through a structured review pathway.

For teams, facilities, researchers, and organizations seeking structured classification or review.

Request Evaluation