Move build-script procedures to top-level

1 files changed, 449 insertions, 0 deletions

diff --git a/BUILD.md b/BUILD.md
new file mode 100644
index 000000000..2f04c7869
--- /dev/null
+++ b/BUILD.md
@@ -0,0 +1,449 @@
+# Build Script
+
+## Introduction
+
+This script builds `dosbox-staging` with your choice of compiler, release
+type, and additional options. It runs on MacOS, Linux, Windows, and possibly
+other operating systems.
+
+If this is your first time building dosbox-staging, then you will need to
+install its development tools and dependencies, which is covered in the
+notes below.
+
+- [Build Script](#build-script)
+  - [Introduction](#introduction)
+  - [Requirements](#requirements)
+  - [Windows Procedures](#windows-procedures)
+    - [Install MSYS2](#install-msys2)
+    - [Clone and Build a Windows Binary](#clone-and-build-a-windows-binary)
+  - [macOS Procedures](#macos-procedures)
+    - [Install Dependencies under macOS](#install-dependencies-under-macos)
+      - [Xcode Installation](#xcode-installation)
+      - [Brew Package Manager Installation](#brew-package-manager-installation)
+      - [MacPorts Package Manager Installation](#macports-package-manager-installation)
+    - [Clone and Build a macOS Binary](#clone-and-build-a-macos-binary)
+  - [Linux Procedures](#linux-procedures)
+    - [Install Dependencies under Linux](#install-dependencies-under-linux)
+    - [Build a Linux Binary](#build-a-linux-binary)
+  - [Haiku Procedures](#haiku-procedures)
+    - [Install Dependencies under Haiku](#install-dependencies-under-haiku)
+    - [Build a Haiku Binary](#build-a-haiku-binary)
+  - [Additional Tips](#additional-tips)
+    - [Compiler variations](#compiler-variations)
+    - [Release types](#release-types)
+    - [Build Results, Rebuilding, and Cleaning](#build-results-rebuilding-and-cleaning)
+    - [CCache](#ccache)
+    - [Optimization Modifiers](#optimization-modifiers)
+  - [AutoFDO Procedures](#autofdo-procedures)
+    - [Prerequisites for AutoFDO](#prerequisites-for-autofdo)
+    - [Record Data for AutoFDO Builds](#record-data-for-autofdo-builds)
+    - [Build Using AutoFDO Data](#build-using-autofdo-data)
+
+## Requirements
+
+- **Windows newer than XP**
+  - **NTFS-based C:**, because msys2 doesn't work on FAT filesystems
+- **MacOS** 10.x
+- **Haiku** up-to-date
+- **Ubuntu** 16.04 or newer
+- **Fedora** up-to-date
+- **RedHat or CentOS** 7 or newer
+- **Arch-based distribution** up-to-date
+- **OpenSUSE Leap** 15 or newer
+
+## Windows Procedures
+
+### Install MSYS2
+
+1. Download and install Chocolatey: <https://chocolatey.org/install>
+1. Open a console and run Cholocatey's command line interface (CLI)
+   to install msys2 and git:
+
+   `choco install msys2 git --no-progress`
+
+   ``` text
+   Chocolatey v0.10.15
+   Installing the following packages:
+   msys2 git
+   By installing you accept licenses for the packages.
+
+   msys2 v20180531.0.0 [Approved]
+   msys2 package files install completed. Performing other installation steps.
+   Installing to: C:\tools\msys64
+   Extracting 64-bit C:\ProgramData\chocolatey\lib\msys2\tools\msys2-base-x86_64.tar.xz
+   Extracting C:\tools\msys64\msys2-base-x86_64-20180531.tar to C:\tools\msys64...
+   Starting initialization via msys2_shell.cmd
+   PATH environment variable does not have C:\tools\msys64 in it. Adding...
+   ```
+
+1. Launch a new MSYS2 terminal (not a CMD prompt or Powershell window):
+   1. Start Menu > Run ... `c:\tools\msys64\msys2_shell.cmd`
+   1. Run all subsequent steps within this terminal.
+
+### Clone and Build a Windows Binary
+
+1. Clone and enter the repository's directory:
+   1. `git clone https://github.com/dreamer/dosbox-staging.git`
+   1. `cd dosbox-staging`
+
+   Be sure to run all subsequent steps below while inside the repo's directory.
+
+2. (🏁 first-time-only) Install the build tools and runtime dependencies:
+
+   `./scripts/list-build-dependencies.sh -m msys2 -c clang | xargs pacman -S
+   --noconfirm`
+
+3. Build an optimized binary with either compiler:
+
+- GCC: `./scripts/build.sh -c gcc -t release --bin-path /mingw64/bin`
+- Clang: `./scripts/build.sh -c clang -t release --bin-path /mingw64/bin`
+
+1. To build a debug binary, use `-t debug` in place of `-t release`.
+
+## macOS Procedures
+
+Builds on macOS can be performed with Clang or GCC. For general use, we
+recommend building with `Clang` as it supports linking with Apple's CoreMidi
+SDK. Developers interested in testing wider compiler coverage might also be
+interested in building with GCC. The following sections describe how to install
+and build with both compilers.
+
+### Install Dependencies under macOS
+
+Before installing either Brew or MacPorts, the Apple's Xcode tools need to be
+installed and the license agreed to:
+
+#### Xcode Installation
+
+1. Install the command line tools: `xcode-select --install`
+1. Accept the license agreement: `sudo xcodebuild -license`
+
+#### Install the Brew Package Manager and Dependencies
+
+1. Download and install brew per the instructions here: <https://brew.sh>
+1. Update it with: `brew update`
+1. Install git with: `brew install git`
+1. Clone the repository: `git clone
+   https://github.com/dreamer/dosbox-staging.git`
+1. Change directories into the repo: `cd dosbox-staging`
+1. Install dependencies:
+  
+   `brew install $(./scripts/list-build-dependencies.sh -m brew -c gcc)`
+
+#### Install the MacPorts Package Manager and Dependencies
+
+1. Build and install MacPorts along with dosbox-staging dependencies with the
+   following sequence:
+
+   ``` shell
+   git clone --quiet --depth=1 https://github.com/macports/macports-base.git
+   cd macports-base
+   ./configure
+   make -j"$(sysctl -n hw.physicalcpu)"
+   sudo make install
+   PREFIX="/opt/local"
+   PATH="${PREFIX}/sbin:${PREFIX}/bin:${PATH}"
+   sudo port -q selfupdate
+   ```
+1. Clone the repository: `git clone
+   https://github.com/dreamer/dosbox-staging.git`
+1. Change directories into the repo: `cd dosbox-staging`
+1. Install depedencies:
+
+   `sudo port -q install $(./scripts/list-build-dependencies.sh -m macports -c gcc)`
+
+### Build a macOS Binary (common for Brew and MacPorts)
+
+1. Build an optimized binary with various compilers:
+
+- Clang: `./scripts/build.sh -c clang -t release -p /usr/local/bin`
+- GCC (brew): `./scripts/build.sh -c gcc -v 9 -t release -p /usr/local/bin`
+- GCC (macports): `./scripts/build.sh -c gcc -v mp-9 -t release -p
+  /opt/local/bin`
+
+1. To build a debug binary, use `-t debug` in place of `-t release`.
+
+## Linux Procedures
+
+### Install Dependencies under Linux
+
+1. Install git: `sudo apt install -y git`
+1. Clone the repository: `git clone
+   https://github.com/dreamer/dosbox-staging.git`
+1. Change directories into the repo: `cd dosbox-staging`
+1. (🏁 first-time-only) Install dependencies based on your package manager.
+   In this example, we use Ubuntu 20.04:
+
+   `sudo apt install -y $(./scripts/list-build-dependencies.sh -m apt -c clang)`
+
+   For other supported package managers, run:
+   `./scripts/list-build-dependencies.sh`
+
+### Build a Linux Binary
+
+1. Build an optimized binary using various compilers:
+
+- Clang: `./scripts/build.sh -c clang -t release -m lto`
+- GCC (default version): `./scripts/build.sh -c gcc -t release -m lto`
+- GCC (specific version, ie: 10): `./scripts/build.sh -c gcc -v 10 -t release -m
+  lto`
+
+1. To build a debug binary, use `-t debug` in place of `-t release -m lto`.
+
+## Haiku Procedures
+
+### Install Dependencies under Haiku
+
+1. Clone the repository: `git clone
+   https://github.com/dreamer/dosbox-staging.git`
+1. Change directories into the repo: `cd dosbox-staging`
+1. (🏁 first-time-only) Install dependencies:
+
+   `pkgman install -y $(./scripts/list-build-dependencies.sh -m haikuports -c
+   clang -v 9)`
+
+### Build a Haiku Binary
+
+1. Build an optimized binary using various compilers:
+
+- Clang: `./scripts/build.sh --compiler clang -t release -m lto
+  --prefix=$HOME/config/non-packaged`
+- GCC: `./scripts/build.sh -c gcc -t release --prefix=$HOME/config/non-packaged`
+
+1. To build a debug binary, use `-t debug` in place of `-t release`.
+1. Install the binary: `make install`
+1. Edit your configuration file by running: `dosbox -editconf` and make the
+   following suggested changes (leave all other settings as-is):
+ 
+   ``` ini
+   [sdl]
+   windowresolution = 800x600
+   output = texturenb
+   texture_renderer = software
+
+   [renderer]
+   scaler = none
+   glshader = none
+
+   [cpu]
+   core = normal
+   ```
+
+   The state of Haiku's GPU Hardware-acceleration is being discussed here:
+   https://discuss.haiku-os.org/t/state-of-accelerated-opengl/4163
+
+1. You may now run `dosbox` inside any directory in your Terminal.
+
+## Additional Tips
+
+### Compiler variations
+
+The compiler, version, and bit-depth can be selected by passing the following
+common options to the **list-build-dependencies.sh** and **build.sh** scripts:
+
+- `--compiler clang` or `-c clang` to use CLang instead of GCC
+- `--compiler-version 8` or `-v <version>` to specify a particular version of
+  compiler (if available in your package manager)
+- `--bit-depth 32`, to build a 32-bit binary instead of 64-bit
+
+### Release types
+
+Build release types includes:
+
+- **release**, optimizes the binary and disables some checks, such as
+  assertions.
+- **debug**, adds debug symbols and disables optimizations for ideal debugging.
+  - You can run the resulting binary in the GNU debugger: `gdb /path/to/
+    dosbox`, followed by `run mygame.bat`
+- **pgotrain** adds Profile Guided Optimization (PGO) tracking instrumentation
+  to the compiled binary.
+  
+  This allows the recording of profile statistics that can be used to compile a
+  PGO-optimized binary. Note that PGO optimization is different from
+  Automatic Feedback Directed Optimization (AutoFDO) mentioned below.
+
+  After compiling your PGO binary, the build script presents instructions
+  describing how to generate and use the profiling data.
+
+- **warnmore**, displays additional helpful C and C++ warnings for developers.
+- **fdotrain**, add tracing symbols used to generate AutoFDO sampling data.
+- **$SANITIZER TYPE**, builds a binary instrumented with code to catch issues at
+  runtime that relate to the type of sanitizer being used. For example: memory
+  leaks, threading issues, and so on. This is for Linux and macOS only.
+  
+  - see `./scripts/build.sh --help` for a list of sanitizer-types that are
+    available.
+  - Run your binary like normal and it will generate output describing
+    problematic behavior
+  - Some sanitizers accept runtime options via an environment variables,
+    such as `ASAN_OPTIONS`, described here:
+    <https://github.com/google/sanitizers/wiki/AddressSanitizerFlags>
+
+### Build Results, Rebuilding, and Cleaning
+
+After building, your `dosbox` or `dosbox.exe` binary will reside inside
+`./dosbox-staging/src/`.
+
+The build script records the prior build type and will clean if needed between
+builds.  To manually remove all intermediate object files and ephemeral
+auto-tools outputs, run `make distclean`.
+
+To additionally remove all files except for the repository files, use `git
+clean -fdx`.
+
+### CCache
+
+The build script will make use of ccache, which saves compiled objects for
+potential re-use in future builds (hence the name, "cache") to speed up build
+times. If you performed the one-time installation step above, then you will
+already have ccache installed.
+
+Simply having `ccache` in your path is sufficient to use it; you do not
+need to invasively symlink `/usr/bin/gcc` -> `ccache`.
+
+The build script enables ccache's object compression, which significantly
+reduces the size of the cache. It will also display cache statistics after each
+build. To see more details, run `ccache -s`.
+
+To learn more about ccache run `ccache -h`, and read
+<https://ccache.dev/manual/latest.html>
+
+### Optimization Modifiers
+
+The following modifier flags can be added when building a **release** type:
+
+- `-m lto`, optimize the entire object space instead of per-file (Only
+  available on Mac and Linux)
+
+- `-m fdo`, performs feedback directed optimizations (FDO) using an AutoFDO
+  data set. Export the `FDO_FILE` variable with the full path to your merged
+  FDO dataset. For example:
+
+  - GCC: `export FDO_FILE=/full/path/to/current.afdo` and then build with:
+  
+    `./scripts/builds.sh -c gcc -t release -m fdo -m lto`
+
+  - Clang: `export FDO_FILE=/full/path/to/current.profraw`, and then build
+    with:
+
+    `./scripts/builds.sh -c clang -t release -m fdo -m lto`
+
+  The section below describes how to collect an AutoFDO dataset for GCC and
+  Clang.
+
+## AutoFDO Procedures
+
+Feedback Directed Optimization (FDO) involves recording performance data from
+the Linux kernel and using it to direct the compiler's optimizer.
+
+### Prerequisites for AutoFDO
+
+- An **Intel processor** that supports the last branch record (LBR) instruction.
+- A Linux **kernel** built with Branch Profiling tracers enabled:
+
+  ``` shell
+  CONFIG_PM_TRACE=y
+  CONFIG_TRACE_BRANCH_PROFILING=y
+  CONFIG_BRANCH_TRACER=y
+  ```
+
+  These can be enable directly in your kernel's `.config` file or using `make
+  menuconfig` via the following menu options:
+  1. `Kernel hacking  --->`
+  1. `[*] Tracers  --->`
+  1. `Branch Profiling (Trace likely/unlikely profiler)`
+  1. `(X) Trace likely/unlikely profiler`
+
+- The **AutoFDO** software package. It may be available via your package
+  manager or built from sources <https://github.com/google/autofdo>.
+
+  - **Note about compiler versions** the autofdo binaries need to be compiled
+    with the exact version of the compiler that will later be used to compile
+    our final optimized version of dosbox-staging.
+  
+    So for example, if you install autofdo via package-manager, then it will be
+    valid for the default version of gcc and clang also installed by your
+    package manager.  Where as if you plan to build with  `gcc-<latest>`, then
+    you will need to compile autofdo from sources using `gcc-<latest>` by
+    pointing the `CC` and `CXX` environment variables to the newer gcc
+    binaries.
+
+  - **Note about clang** If you plan to compile with a version of clang newer
+    than your package manager's default version, then you will need to compile
+    autofdo from source and configure it with the corresponding version of
+    `llvm-config`.  For example, if I want to build with clang-10, then I would
+    configure autofdo with: `./configure --with-llvm=/usr/bin/llvm-config-10`.
+
+  - The included `.github/scripts/build-autofdo.sh` script can be used to build
+    and install autofdo, for example:
+
+    - default GCC:
+
+      `sudo .github/scripts/build-autofdo.sh`
+    - newer GCC:
+
+      ``` shell
+      export CC=/usr/bin/gcc-9
+      export CXX=/usr/bin/g++-9
+      sudo .github/scripts/build-autofdo.sh
+      ```
+
+    - Clang version 10:
+  
+      `sudo .github/scripts/build-autofdo.sh`
+
+- The **pmu-tools** software package, which can be downloaded from
+  <https://github.com/andikleen/pmu-tools.> This is a collection of python
+  scripts used to assist in capturing sampling data.
+
+### Record Data for AutoFDO Builds
+
+1. Ensure the custom Linux Kernel supporting LBR tracing is running.
+
+2. Build `dosbox-staging` from source using the `fdotrain` target:
+   `./scripts/build.h -c gcc -t fdotrain`
+
+3. Record kernel sample profiles while running dosbox-staging:
+
+    `/path/to/pmu-tools/ocperf.py record -F max -o "samples-1.prof" -b -e
+    br_inst_retired.near_taken:pp -- /path/to/fdo-trained/dosbox ARGS`
+
+   Where `samples-1.prof` is the file that will be filled with samples.
+
+   Repeat this for multiple training runs, each time saving the output to a new
+   `-o samples-N.prof` file.  Ideally you want to exercise all code paths in
+   dosbox-staging (core types, video cards, video modes, sound cards, and audio
+   codecs).
+
+4. Convert your sample profiles into compiler-specific records using tools
+   provided in the `autofdo` package:
+
+   For GCC, run:
+   - `create_gcov --binary=/path/to/fdo-trained/dosbox
+     --profile=samples-1.prof -gcov=samples-1.afdo -gcov_version=1`
+
+     ... for each `.prof` file, creating a corresponding `.afdo` file.
+
+   - At this point, you now have an `.afdo` file for each `.prof` file. Merge
+     the `.afdo`s into a single `curren.afdo` file with:
+
+     `profile_merger -gcov_version=1 -output_file=current.afdo *.afdo`
+
+   For Clang, run:
+
+   - `create_llvm_prof --binary=/path/to/fdo-trained/dosbox
+     --profile=samples-1.prof --out=samples-1.profraw`
+
+     ... for each `*.prof` file, creating a corresponding `.profraw` file.
+
+   - At this point, you now have a `.profraw` file for each `.prof` file. Merge
+     them into a single `current.profraw` file with:
+  
+     `llvm-profdata-<version> merge -sample -output=current.profraw *.profraw`
+
+### Build Using AutoFDO Data
+
+You can now use your merged `.afdo` or `.profraw` file to build with the `-m
+fdo` modifier by  placing your `current.afdo/.profraw` file in the repo's root
+directory, or point to it using the FDO_FILE environment variable, and launch
+the build with `./scripts/build.sh -c <gcc or clang> -t release -m lto -m fdo`.