Timedemo and Framerate Test

= About the X-Plane Framerate Test =

X-Plane contains a built-in framerate test mode. In this mode the sim ignores preferences and loads a fully known and controllable configuration, runs some framerate tests (with user interaction disabled, e.g. no dialog boxes will block operation) and quits. Information is output to the Log.txt file.

The framerate test is aimed at two audiences:
 * It allows Laminar Research to gather performance information from beta users' machines without the risk of configuration error.
 * It allows driver writers to run X-Plane as part of an automated regression system.

This document is written for programmers who want to use X-Plane in a production environment, like an automated regression harness.

= About the X-Plane Time Demo =

The X-Plane time demo is a 3-platform bzipped tarball of X-Plane 9 RC3 with some optional content removed to reduce download size.

If you are looking for the X-Plane web demo, do not download the time-demo. Go to http://www.x-plane.com/demo.html and use the web demo installer to get the full demo. The time demo has been stripped of most airplanes to reduce its size. The web demo is hosted by significantly more servers and should be quicker to download. The web demo is also a more recent version of X-Plane!

You should use the timedemo if you are creating an automated test because:
 * It is pre-packaged in a format that can be extracted without running a tool with user interface.
 * It is packed with at least one test flight to vary tests.
 * Because it is a snapshot of RC3, the test parameters will not change when patches come out.

= How to Download the Time Demo =

Use these shell commands to get and install the time demo:

curl -O "http://update.x-plane.com/X-Plane_900r3_timedemo.tar.bz2" tar -xjf X-Plane_900r3_timedemo.tar.bz2 rm X-Plane_900_r3_timedemo.tar.bz2 cd X-Plane_900r3_timedemo

The x-Plane 945 time demo test can be found at this URL: http://update.x-plane.com/xplane_945_timedemo.tar.bz2 and can be unpacked with similar command-line instructions.

= Running Simple Tests With the Time Demo =

Here are some example tests:

./X-Plane-i686 --load_smo=Output/movies/test.smo --fps_test=1 grep FRAMERATE Log.txt

= FPS Test Options And Commands In Detail =

The framerate test is controlled by command-line options. Please note that options and preferences discussed here are accurate for RC3 but may be subject to change in the future.

Running a FPS Test
To run a framerate test, you use the option:

--fps_test=

Where N is a number indicating the FPS test to be run. The FPS tests are pre-built settings. The basic framerate test will run for 90 seconds:
 * 30 seconds of a forward view with the panel.
 * 30 seconds of a forward view with no panel.
 * 30 seconds with a forward view, no panel, simulation paused.

This structure was originally designed to provide a differential cost of the 2-d panel and flight model.

The performance of each of the tests is output to the log file like this:

FPS Test Codes
Currently the framerate test numbers are built from three digits (leading zeros can be omitted):


 * Hundreds digits - Viewpoint. 0 = default, 1 = view from above, 2 = night.
 * Tens digits - Weather settings (0 is default) - a variety of different weather conditions, 1-6).
 * Ones digits - controls rendering settings: 1 = low, 2 = medium, 3 = high. This controls a range of rendering settings; use individual preferences for more control (see below).  Note that 1 gives you "high" tex res (4x reduction in all dimensions, 2 gives you "very high" (2x reduction) and 3 gives you "extreme" (no reduction).  This is useful because the _tex_res_ pref seems to not be command-line controllable.

Note that the fly-over interferes with some views - see prefs below.

Minimum FPS Mode
X-Plane can also be run in minimum-fps mode. In this mode, the sim will run one 30 second test (with flight model and panel on) and then return 1 on success or 0 on failure. Framerate is not logged. Typically this would be used to fail an automated regression test, e.g.

./X-Plane-i686 --fps_test=1 --require_fps=30 || echo X-Plane is running slower than 30 fps.

Note that you must use --require_fps with --fps_test.

Playing Replay Movies
Normally the fps test will simply leave the plane on the runway; however you can also program X-Plane to load and play a "replay movie" (.smo file) - this is a binary file containing a replay of an X-Plane flight. Example:

./X-Plane-i686 --fps_test=1 --load_smo=Output/movies/test.smo

The smo file is a relative path from the root of the X-Plane folder. The timedemo test contains one movie, test.smo.

The fps test will run for 90 seconds (or 30 seconds for --require_fps) regardless of movie length; you should set your movie to about 90 seconds. Note that the .smo file format is fixed-size; you will not save disk space with shorter movies. (But shorter movies will zip more efficiently.)

One use of replay movies is to vary the viewpoint (by flying the plane) to get a more representative rendering load.

Controlling Individual Settings
You can override individual settings using the --pref command; the syntax is:

--pref: =

For example:

./X-Plane-i686 --fps_test=1 --pref:_x_res_req_ALL=1440 --pref:_y_res_req_ALL=900

will run the sim with fps test 1 but at 1440x900 res.

Preference names are strings; you can find them by viewing the contents of the Resources/preferences folder (after running the sim normally to init preferences).
 * Res X-System.prf contains a few resolution settings, global to all apps.
 * Set X-Plane.prf contains most non-binary X-Plane prefs that can be set.

Here are some settings that are useful for testing X-Plane performance. Note that I have not tested all of these; this comes from dumping a prefs file to the terminal.

WARNING: if you use a fps test number, it will override settings set by individual preferences. When the fps test is used, you can only control resolution-related preferences.

Note: in X-Plane patches before 941, _alias_req will change FSAA but will not correctly show the results in the UI when --fps_test is in use.

= Changes in Settings For 945 =

Window Management Changes
To use 945 in Windowed mode. (Note: window must be no larger than the size of the primary screen) --pref:_x_res_wind_ALL= --pref:_y_res_wind_ALL= To use 940 in full screen mode at current monitor settings: --pref:_is_full_ALL=1 --pref:_x_res_full_ALL=0 --pref:_y_res_full_ALL=0 To use 940 in full screen mode at a specific res: --pref:_is_full_ALL=1 --pref:_x_res_full_ALL=1024 --pref:_y_res_full_ALL=768 --pref:_bpp_full_ALL=32

Note: X-Plane will not change the monitor res unless it can determine that the selected res matches a published resolution from the OS/driver. Usually this means supplying a bits per pixel (bpp) of 32.

However, X-Plane does _not_ actually put the device into full screen mode, so there is no advantage to running "full screen" vs. running in a window with the window size set to the monitor size.

Replay Movies
In X-Plane 945, replays are called .rep files and live in Output/replays/. Unlike the 900 time demo, the replay will play from start to end and the time demo will end with one framerate count output.

Custom Log Paths
The command-line option --log_path= will redirect the output file Log.txt to another directory. The very beginning of the file will end up in the x-plane dir (if writable) but most of the file, including all of the interesting parts) will be redirected. The path must be an absolute path including file name, e.g.

./X-Plane-i686 --log_path=/my_dir/log.txt

= Changes for Version 10 =

Version 10 keeps the version 945 screen res format. Besides all v9 features, v10 allows the settings from "Resources/settings.txt" to be overridden using the --ren:XXXX=YYY syntax. An example:

X-Plane_NODEV_OPT.app/Contents/MacOS/X-Plane_NODEV_OPT --fps_test=2 --load_smo=Output/replays/test_flight.rep \ --pref:_is_full_ALL=1 --pref:_x_res_full_ALL=1024 --pref:_y_res_full_ALL=768 --pref:_bpp_full_ALL=32 \ --ren:draw_HDR=1

X-Plane 10 contains a config file in the Resources folder called "settings.txt". The line SETTING category type name max defines the setting parameter,s e.g SETTING EFFECTS SLIDER shadow_quality 9 means that the shadow_quality setting will show as a slider, with a min of 0 and a max of 9. You can change these settings from the cmd line, e.g. --ren:shadow_quality=5 overriding the defaults from fps tests. You can use the command line to rapidly find all settings like this: grep "^SETTING " resources.settings.txt

The time demo is also data driven - you can view the default settings for each of the rendering settings as follows: grep DEMO Resources/settings.txt The TIMEDEMO lines define each time demo (1-X) and the DEMO_SETTING shows the deafults used.)

Note that the time demos affect the rendering settings in settings.txt, which are mostly engine related, but _do not_ set raw windowing factors like resolution and full screen anti-aliasing. Thus you should expect to use --pref to customize FSAA and use monitor controls to get the res you want.

As a general rule:
 * EFFECTS settings tend to push the GPU harder, e.g. consume more shading power.
 * DRAWING settings tend to push bus bandwidth and the CPU harder, e.g. they result in more geometry.
 * EXPERT settings should be left at their default, under all conditions.

= QA Scripts and Visual Output =

X-Plane 945 features a simple script file engine to produce conformance tests. To run a test file, use the command-line --qa_script=. The X-Plane 945 time demo comes with a sample script (conformance.txt) that can be used for baseline driver testing.

The script commands consist of a series of single-line commands, with white space and lines starting with # as comments.

QA script commands
WAIT

causes the script to pause, letting the sim run, for a number of seconds.

NEXT_FRAME

causes the sim to render the next frame. When settings are changed, you will need to either wait or go to the next frame to see the changes.

CMND

executes one of X-Plane's built in commands - any command that can be tied to a joystick or key binding can be executed by script. To find the commands, look in Resources/plugins/commands.txt

DREF

sets a sim dataref to a given value. Look in Resources/plugins/datarefs.txt for a list of datarefs.

LOOK   

causes the sim to change the camera to be placed at x,y,z and look in a given heading. You can use the data outut screen to determine the current camera position for writing scripts.

RELOAD

causes scenery to reload - use this after changing settings to change scenery.

TIMEER_START TIMER_STOP

measures an interval at a given framerate. Use this to measure fps for a given camera angle and framerate. The tag cannot have whitespace.

PUSH_APP_PATH POP_APP_PATH

Changes the current directory for the script to a new path (path should be a global path starting with /) for the purpose of taking pictures. Popping the path sets it back to the previous path.