X-Plane Scenery Tools Development

= Preface =

I started working on the Scenery Tools as the result of submitting a bug report for X-Plane. The conclusion is, don't report bugs, it's dangerous, it can involve you in very time consuming work. Seriously, Ben was asking me for starting the port of the Scenery Tools to the Linux operating system, focussing first on ObjView and World Editor, and I thought why not. But I felt uncomfortable with the state of the current source code distribution. Luckily Ben gave me CVS access, so I was at least able to do development on an actual tree. After finishing initial work on the build environment for Linux, the commandline tools and ObjView, I had the idea to ask several Linux distributions for including the Scenery Tools in their official package repositories. This however, in the case where the package maintainers aren't the developers, would require a public accessible source code repository, so that the maintainers can get and review the code for the package they want to build. Beside that other developers might benefit from the fact, that they are able to create patches on base of a more recent state of the tree now.

= Getting the Build Tools You Need =

The X-Plane scenery tools code (XPTools) can be compiled for Mac, Windows, or Linux (port still in progress). Before you can work on the tools, you may need to get/update your development environment.

Mac OS X
At this point we support the following minimum requirements for compiling the scenery tools: To get X-Code, go to http://developer.apple.com/, register as an ADC develop if needed (it's free as in beer), and download the X-Code DMG. It's big. The DMG runs an installer that gets you everything you need.
 * OS X 10.5 or newer.
 * X-Code 3.1.2 or newer.
 * An x86 Macintosh.

You will also need the command-line tool CMake, which can be found here: http://www.cmake.org/

Important: if you install CMake using the disk image (.dmg) be sure to say yes when asked if you want to isntall the symbolic links to the command-line tools. If you do not install symbolic links, the libraries will not build.

The currently tested configuration is OS X 10.5 with X-Code 3.1.2.

OS X 10.4 and PPC
At this point we have dropped official support for OS X 10.4 and PPC compilation. The reason for this is two-foold: Therefore in order to make everything compile properly, we need 10.5 (for GCC 4.2) and i386 architecture (that is, 32-bit x86) for consistency.
 * The scenery tools require GCC 4.2 or newer.
 * Some of the libraries we use do not handle multiple architectures gracefully.

The following were legacy instructions for compiling tools on 10.4: If you want to build the scenery tools on OS X 10.4, you will need to get GCC 4.2. * Download this build: http://r.research.att.com/tools/gcc-4.2-5566-darwin8-all.tar.gz * From the command line, run this to install: sudo tar fvxz gcc-4.2-5566-darwin8-all.tar.gz -C / * More info here. http://r.research.att.com/tools/ (Note: I have not tested this process since I no longer have a functioning 10.4 machine with full dev tools.)

Windows
WARNING: make sure there are no spaces in the paths up to the code on Windows! The Makefile does not handle this case yet. You may want to consider this when installing the tools.

MinGW is the supported compiler - you will need to set up a full mingw environment with GCC 4.2.x or newer. The simplest way to do that is with the XPTools Environment Installer, which can be downloaded from this page: http://scenery.x-plane.com/code.php

The installer is a net-installer - that is, it will download the latest components to the build environment. The build environment is entirely self-contained - simply delete it when done - it does not install any registry keys, COM components, or other global "stuff". To start the environment, run startenv.bat from the root of the installation directory. It will put you automatically in the Scenery Tools Source directory if you choosed to download the sources during the installation, or into your home directory if you skipped that step.

To update the buildenvironment itself (i.e. when we added a new gcc release for example) just start the environment and execute following commands:

cd / git pull origin master:master

This however might fail when we update bash or git in the environment because Windows locks files which are currently in use. In this case just re-run the Buildenvironment Installer over an existing installation to update the environment (you will need an installer version >= 1.1.1 for this to work).

Similary you can update the Scenery Tools source tree with:

cd /xptools git pull origin master:master

Note that you aren't on the master branch after a fresh installation, but a specific commit because the Scenery Tools tree is registered as a git submodule in the build environment. Therefore you need to make sure that you switch to the master branch before building the tools after updating the tree (you need to do this only once after a fresh installation):

cd /xptools git checkout master

Linux
You will need GCC 4.2 or newer, and a few packages:


 * gnu make
 * gnu coreutils
 * binutils, binutils-devel if applicable for libbfd.a
 * libmpfr-devel
 * qt4-devel
 * mesa-libGL-devel, mesa-libGLU-devel or the proprietary variants from ATI or NVidia

= Get the source code =

In the past the source code of the X-Plane Scenery Tools was only available as a snapshot tarball, updated every now and then. The main disadvantage of this approach is, that it is considerably hard to apply patches to the current tree which were created on base of these tarballs. Their main characteristic of being outdated very quickly makes them almost useless for developers. To improve interaction with people who are willing to participate in Scenery Tools development, we have setup an infrastructure which most open source developers should be familar with (ok, currently we don't have a bugtracker, please report bugs on x-plane-dev@yahoogroups.com).

Browse the tree online
For those who want to keep track of the most recent changes, or only want to steal shamelessly code via copy&paste, have a look at:

http://dev.x-plane.com/cgit/cgit.cgi/xptools.git/

Get most recent snapshot tarballs
Those who want to work on the tree, but don't want to use any version control software, or only want to review the code locally, can download a snapshot of the most recent development state. Unlike the old tarballs, these are created on the fly from the repository when you download them and reflect the current state of the tree.

xptools-master.tar.bz2 (~79 MiB) xptools-master.zip (~243 MiB)

Clone the Git repository
Those who want to contribute patches and do serious development on the tree, can clone the repository with git, using either the faster git-protocol or, if you have troubles caused by a firewall, the considerably slower http-protocol as transport:

= Compiling the Program =

Building on Linux and Windows (Command line)
I tried to make this process as simple as possible. First of all make sure that you have following prerequisites installed:


 * development packages of libbfd (mostly included in binutils), libmpfr, libz and boost
 * gcc (i think :-))
 * gnu make

then just do a

make

in the Scenery Tools root directory. Take a coffee and a walk in your favorite park :-). After serveral aeons the output can be found under

/build/ /

The platform is determined automatically, when building on Linux it is Linux of course. The configuration defaults to debug_opt. you can specify the configuration when building the tools this way:

make conf=

where can be one of the following:


 * release
 * release_opt
 * debug
 * debug_opt

the _opt variants are built with -O2 set, the others with -O0. release variants are built with -DDEV=0 set, debug variants with -DDEV=1. To clean the tree you can do:


 * make clean, this just deletes the /build directory
 * make distclean, this deletes the /build directory and the built 3rd-party libraries located in /libs/local

You can also build a single tool, e.g.

conf=release_opt make ObjView

For Windows we provide prebuilt 3rd-party libraries to speed up the build process. If you want to use them the typical build process might look like this:

make distclean make gitlibs=1 libs make

This will cause 'make' to download the prebuild libraries (see ./libs/local) instead of building them from scratch. Since this is a git repository as well, you can update the 3rd-party libraries with

cd /xptools/libs/local git pull origin master:master

without the need to do a distclean again.

Building on OS X (X-Code)
The X-Code project for almost all of the scenery tools is called SceneryTools.xcodeproj and is in the root of the tree. There is one target for each of the scenery tools - simply pick a configuration, target, and build.

Important: before building on the Mac for the first time, you must build the libraries - the procedure to build libraries on OS X is exactly the same as for Linux and Windows: do a "make" in the "libs" directory. X-Code will not do this for you.

The AC3D plugin has its own xcode project, XPExporter.xcodeproj, also in the root of the tree.

A few notes on X-Code settings:
 * The X-Code scenery tools project defines 4 build types. Besides debug and release (the normal configurations), there are two more:
 * DebugOpt is a debug build with the optimizer on. For work with CGAL-based tools (like MeshTool), this means inlining is enabled (which makes the tools a lot faster), but the debug assertion checks are on, to catch program errors.
 * Phone is a debug build with PHONE defined to 1 - this turns on a few special hooks and options that LR uses internally for deploying content to the iphone.
 * You probably won't be able to make a release build due to the universal binary problem - see X-Plane Scenery Tools on OS X, below.

= Contribute Patches =

Working like in the old days
If you do not have the time or motivation to work on the Scenery Tools on a regular basis and just want to submit a minor fix or feature enhancement, you'll probably decide to download a snapshot tarball and code against that tree. In this case just create a patch with a diff tool and submit it via email. When using GNU diff, you would do something like this:

Using git
If you decide to use the git repository for development, you'll have to be aware of some issues which might arise due to the underlying infrastructure. Due to historical reasons the main repository is managed by CVS. And this won't change, as long as Ben isn't frustrated enough to stop using it :-). Seriously though, he was kind enough to setup git on the development server, allowing us geeks to do some serious work on the tree.



I won't describe basic tasks with git here, since git's documentation evolved to a very professional state over the time. Have a look at
 * https://we.riseup.net/debian/git-development-howto for introductory informations
 * http://www.kernel.org/pub/software/scm/git/docs/everyday.html for a quick reference of common commands

The above mentioned repository does have a master branch which reflects the current state of the CVS' head. Starting a new development branch based on a commit, usually the most recent, in master is probably a good idea to avoid merge conflicts when pulling from upstream with git pull. I encourage the use of git rebase after pulling new objects and updating the master branch to have your local development branch up-to-date, except you have people pulling from your repository. In this case merging the master branch back to your development branch is a better choice because rebasing causes the creation of new commits with new sha1-sums which might distort the workflow of users pulling from your repository. Once you've finished your work and you think it's time to submit your changes, you can create a series of patches from your development branch with git format-patch and send them via email either manually or automatically using git send-email. Or you can, preferably, file a pull-request via email, naming your repository URL and the branch name you want me to pull from. This will only work if you can publish your repository somewhere on the internet of course.

Sothis is taking all my credit
Once you've submitted a series of patches, and when they get accepted, they will appear sooner or later in the master branch (yes they do! :-)). You will notice that the committer and author will be the user 'sothis', i.e. me. This is, of course, because i want to get all the credit. Seriously, it is, like mentioned earlier, because of the underlying infrastructure. As long as the main repository is managed by CVS, there is no way around this (except someone else can point me to a solution, please let me know). I have to export patches manually to CVS with git cvsexportcommit, while the git repository gets updated automatically by git cvsimport. Therefore all accepted patches will appear like as if they are written by me. I will add a note in the commit message from whom the patch originally came of course.

= X-Plane Scenery Tools on Linux =

Current State of Development
Note: at this point, due to the low total number of developers working on scenery tools, Ben often has commits on the master tree that break compilation of some of the command line tools and other targets. These get fixed as we approach a tools beta release. There are a few ways to work around this:
 * Grab a build based on a mile stone tag or tool version tag. (This may not help much for Linux tools, which have not been through a full release cycle.)
 * Grab an older build, retry.
 * Yell at Ben to fix his broken code!
 * Commandline Tools
 * DDSTool, DSFTool, ObjConverter and MeshTool should work out of the box. Since there was no heavy testing with Linux binaries yet (just because we haven't released a beta or even an alpha yet), there will be most likely critical bugs, especially on 64-bit platforms where pointer issues and integer width inconsistencies might break handling of various file formats and produce corrupt file data.


 * ObjView
 * works :-)


 * World Editor
 * This is the part which made me 50 years older while working on it. This is what we have at the moment:
 * basic windowing and opengl support
 * creating and loading scenery packages
 * basic support for editing packages
 * And now the much longer list of things which need to be implemented:
 * Menu System. The lack of a menu causes automatically the lack of almost any functionality, including saving the scenery package, using the terraserver layer, undoing tasks and so on. The idea is to implement an OpenGL drawn menu. The advantage is, that this one can be used on MacOS and Windows as well and we would get rid of platform specific menu code. Sadly Ben is the only one who understands his framework for drawing stuff and since he's busy with X-Plane of course, this one will need some time until it's implemented.
 * Clipboard functionality and keyboard shortcuts. Adding clipboard code will be next thing on the list, as it's only useful with working keyboard shortcuts it depends indirectly on:
 * Unicode support. This one is on the list for MacOS and Windows builds as well. Current keyboard code and text data handling is ASCII only which is pretty much too restrictive. Since X-Plane supports Unicode already, we plan to revamp WED as well
 * Drag&Drop. While it is working for ObjView already, the implementation for the X11 variant of WED is a bit more complex. Because drag&drop support is not mandatory to work with WED (as far as i reckon), it's currently the last item on my TODO list
 * Timer. WED makes use of timers to execute asynchronous tasks like scrollbar animation, background loading of data like the terraserver images and so on. While Windows can generate WM_TIMER messages, X11 lacks of such a functionality. Currently we use a threadbased timer which generates similar events but might cause locking issues. This one needs a revamp too.
 * Compositing Window Manager compatibility. WED and ObjView currently suffer a bug which drives them useless when using them with a compositing Window Manager and OpenGL effects enabled.
 * Packagemanagment. For being able to include the Scenery Tools in several Linux distributions, we need to create package management specific files like rpm spec's and deb dsc's.
 * Messageboxes and File-Chooser Dialogs. Currently we're using the ones provided by the GTK toolkit. GTK opens an own displayconnection and we need to flush this connection manually. This is a really dirty hack and can cause many problems. Preferably we want to implement own dialogs drawn by OpenGL in the far future.

= X-Plane Scenery Tools on OS X =

The Mac now uses the standard lib system - you should be able to build for any OS/architecture from any machine. Note that I have only tested this using X-Code 3.1.2.

= X-Plane Scenery Tools on OS Windows =

MinGW is the new official build environment.

= Mailing List/Contact =

We are using the x-plane-dev yahoo group/mailing list to discuss scenery tools coding; to subscribe, send mail to

x-plane-dev-subscribe@yahoogroups.com