tde-packaging/debian/_buildscripts/local

-----
Index
-----
A) Environment preparation
B) Notes about scripts
C) How to use the scripts


--------------------------
A) Environment preparation
--------------------------
0) Initial notes
   - this guide has been prepared based on a clean debian testing netinstall system without any other DE.
     It should work for other debian/ubuntu distros as well, but eventually you may run into small differences
     here and there.
   - you can change folder names below, as long as you update the "_config.sh" file accordingly.
   - you need to choose whether to use standard git repository clones or use git worktrees.
     Different instructions will be provided where necessary, based on the choice made.
   - you need to choose whether to use standard git repository clones or use git worktrees.
     Different instructions will be provided where necessary, based on the choice made.
   - you need to choose whether to use pre built extra dependency packages or build them yourself.
     Different instructions will be provided where necessary, based on the choice made.
     More on this at point 9).

1) Install following packages: bc, cdbs, git, pbuilder, rsync and required dependencies.
   NOTE: sudo should already be installed. If not, install sudo as well.

2) Create a base folder for TDE, hereafter referred to as TDE_DIR (for example $HOME/tde_src)

3) Create the following folders [ see NOTE at point 0) about folder names ]
   - in TDE_DIR: 0_logs : contains log files for repo update and global builds (more on this later)
                 1_git  : contains the git repo clones and build hook scripts
                 2_build: folder used for build preparation and for local builds
                 3_repo : local repo for package installation (to be configured in /etc/apt/sources.list)
                 buildscripts: contains a local copy of the build scripts, which can be modified as required
   - in TDE_DIR/1_git:
                 edeps  : contains extra dependency modules necessary to build TDE (this folder is not necessary
                          if pre-built extra dependencies are used - see point 9 for more info)
                 hooks  : contains build hook scripts to execute ad-hoc code before and after the building process.
                          Mainly used to apply patches automatically during the building process

4) Clone TDE git repositories
   A) without using git worktrees
      A.1) Main repo:
        cd "$TDE_DIR/1_git"
        git clone https://mirror.git.trinitydesktop.org/gitea/TDE/tde.git
        git clone https://mirror.git.trinitydesktop.org/gitea/TDE/scripts.git tde/scripts
        cd tde
        ./scripts/switch_all_submodules_to_head_and_clean anonymous

      A.2) If you are *not* using pre-built extra dependencies:
        cd "$TDE_DIR/1_git"
        git clone https://mirror.git.trinitydesktop.org/gitea/TDE/extra-dependencies.git edeps

   B) using git worktrees
      B.1) Main repo
        cd "$TDE_DIR/1_git"
        git clone --bare --config "remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*" https://mirror.git.trinitydesktop.org/gitea/TDE/tde.git repos/tde.git

      B.2) If you are *not* using pre-built extra dependencies:
        cd "$TDE_DIR/1_git"
        git clone --bare --config "remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*" https://mirror.git.trinitydesktop.org/gitea/TDE/extra-dependencies.git repos/edeps.git

      B.3) run the script "update_repositories.sh" once you have completed the setup as described later in this document.

5) Add your user to the sudo group (not required if you are root). Mind the stand alone "-"!
     su - -c "adduser <username> sudo"
   Logout and login again to make sure the new settings are applied. If you forget this step, you will have build
   errors when you first try to build a module.
   * Optionally, consider extending your sudo timeout interval to avoid having to type your password too often
     (this could be quite painful especially during long builds, which would not be able to complete if unattended).
     To do this, type "sudo visudo" and then add "Defaults timestamp_timeout=<new timeout>", where the value is in
     minutes. Use a value of -1 to set an infinite timeout.

6) Copy the files from "TDE_DIR/1_git/tde/packaging/debian/_buildscripts/local/additional_files" folder
   to the respectivily named folders.

7) Create pbuilder base package with the command:
     sudo pbuilder create
   If you wish to build for a different distro or architecture, use the command:
     sudo DISTRO_NAME=<target distro> ARCHITECTURE=<target architecture> pbuilder create
   If all goes well there should be no errors reported and a file named "base-<distro name>@<architecture>.tgz"
   should be found at location "/var/cache/pbuilder/".

8) Setup the build scripts locally:
  - copy the contents of "TDE_DIR/1_git/tde/packaging/debian/_buildscripts/local" to "TDE_DIR/buildscripts"
    and make sure all shell scripts are executable
  - cd "TDE_DIR/buildscripts"
  - cp ./internals/_config_template.sh _config.sh
  - edit "_config.sh" to set your preferences and check that the various folder names match the structure on your disk.
    Make sure to set the following variables:
     * TDE_DIR to the correct path.
     * DISTRO, DISTRO_NAME and ARCHITECTURE to match the distro and architecture you want to build for if different
       from your current setup. Otherwise the variables can be left empty and auto detection will be performed.
     * UPDATE_BRANCHES to the branches you want to keep updated from the main repositories.
     * DEFAULT_REPO_BRANCH to the branch to check out at the end of the repositories update process.

9) Some additional packages (referred to as extra dependencies) are required to build and install TDE in debian/devuan/ubuntu.
   These modules can be built locally or alternatively pre-built binaries from the TDE archive can be used.

   9.1) Using pre-built extra dependencies
     - open "_config.sh" and set the variable USE_PREBUILD_EXTRA_DEPS to "y" (this is the default initial setting)
     - create the file /etc/apt/sources.list.d/tde.list add the following lines to it. This will setup the
       correct repository in apt for installing the required extra dependency packages.
        * For R14.2.x series (repository branch "master")
          # --- TDE EXTRA DEPENDENCIES REPOSITORY ---
          deb http://mirror.ppa.trinitydesktop.org/trinity-testing <YOUR DISTRO> deps
        * For R14.1.x series (repository branch "r14.1.x")
          # --- TDE EXTRA DEPENDENCIES REPOSITORY ---
          deb http://mirror.ppa.trinitydesktop.org/trinity-sb <YOUR DISTRO> deps-r14
       For example:
         deb http://mirror.ppa.trinitydesktop.org/trinity-sb buster deps-r14

     - install package dirmngr if required
     - import the TDE archive signing key into your apt keyring
         sudo wget http://mirror.ppa.trinitydesktop.org/trinity/deb/trinity-keyring.deb
         sudo dpkg -i trinity-keyring.deb

   9.2) Building extra dependencies locally (recommended option)
     - open "_config.sh" and set the variables USE_PREBUILD_EXTRA_DEPS to "n" and
       CFG_EXTRA_DEPS_DIR to "edeps"
     - build the extra dependency as any other normal module

10) If needed, create the file /etc/apt/sources.list.d/tde.list. Then add the following lines to it.
    This will setup your local repository in apt.
      # --- LOCAL TDE REPOSITORY ---
      deb [trusted=yes] file:$TDE_DIR/3_repo <YOUR DISTRO> main
    For example:
      deb [trusted=yes] file:/home/tde_src/3_repo buster main



----------------------
B) Notes about scripts
----------------------
1) * Notes *
Modules are built using the build_module.sh script. After the build is completed, the installation .deb files are located in TDE_DIR/2_build/debs/<MODULE NAME>/ and the source code and build reports in TDE_DIR/2_build/debs/<MODULE NAME>/src/

The source code can be either the git repo or a local copy in TDE_DIR/2_build/build/<MODULE NAME>.
A module can be built in a clean chroot environment using pbuilder (default option) or locally using dpkg-buildpackage (useful for quick debugging/developing).
When using pbuilder, a hook can be used to invoke a shell when the build fails.

Build logs are automatically stored to files, but can also be displayed during the build process.
The default location of a module build log is TDE_DIR/2_build/debs/<MODULE NAME>/src/__build__.log

When building sets of modules or the whole TDE, a global build summary is automatically stored to TDE_DIR/0_logs/build_result.log to quickly check what built and what failed. It is recommended to delete that file before starting a new TDE build (if not, build results will be appended at the end of the file).


2) * Scripts description *
- scripts in 'internals' folder
  Scripts used intenally by other scripts. No need for invoking these directly.

- update_repositories.sh:
  Script used to update the local clone of the git repositories.
  It is possible to update multiple branches as specified by the variable UPDATE_BRANCHES in the configuration
  file. After the update is completed, the local repositories will be switched to the branch specified by the
  DEFAULT_REPO_BRANCH variable.
  The script can also be used to switch the local repositories to a particular branch, without performing any update.
  Usage:
    update_repositories.sh [options]
  Options:
    -i  (Incremental) : in case the previous update was interrupted, continue from the last known updated module.
                        (useful on slow/unstable internet connections). If the previous update had completed,
                        this option is ignored.
    -v  (Verbose)     : display and log more output. Useful for troubleshooting.
    -ub "<branches>" (Update Branches) : allows to specify the branches to update. This override the variable
                        UPDATE_BRANCHES in the configuration file. If a single branch is specified and the
                        '-db'/'-sb' paraemters are not used, '-ub' can be used to update and switch a single branch.
    -db <branch> (Default Branch) : allows to specify the default branch. This override the variable
                        DEFAULT_REPO_BRANCH in the configuration file.
                        This option is only used if the argument '-so' is not specified.
    -so <branch> (Switch Only) : switch the local repositories to the specified branch, without doing any update.
                        If '-ub' is used, the '-so' branch name must be one of those specified in the 'ub' branches.

- build_module.sh
  Build a single module. The name of the module can be specified in two ways:
  1) by providing the full "folder/module name" location
     Examples:
       tdebase
       applications/abakus
       dependencies/libr
       edeps/debian/imlib
  2) by providing only the module name and let the script look up for a unique match among the known modules.
     Examples:
       tdebase
       abakus
       libr
       imlib
  3) for extra dependencies only, by providing the repository folder and the module name without the "debian"
     subfolder
       edeps/imlib
  Usage:
    build_module.sh [options] [module_name]
  Options:
    -g  (Git)         : build from git repo sources. If missing, build from the local copy in build folder.
    -l  (Local)       : build the module locally. If missing, build in a clean chroot environment
		-p <folder> (Path): build from the specified folder. This option cannot be used together with -g.
		                    The path must be a module in the git repository or in the local build folder.
												This option is mostly intended to be used when using branches with git worktrees.
    -sl (Show Log)    : output the building logs to terminal while the build is ongoing
    -lr (Log Result)  : log (append) build result (OK, FAILED) to TDE_DIR/0_logs/build_result.log file
    -sh (Shell Hook)  : use a shell hook for failing builds, only valid if building using pbuilder (clean chroot environment)
    -po (Prepare Only): only prepare the source folder but do not build the module. Useful to prepare the source code before
                        doing local changes/development. The module can then be built from the modified local folder
    -d  (Debug)       : enable debug symbols if possible (debian/rules file must contain "RelWithDebInfo" for this to work)
		module_name       : the module to build. If '-p <folder>' is used, this parameter must not be specified since the module
		                    is derived from the <folder> parameter.

- <dd>_<set_name>.sh
  A number of scripts used to build sets of modules. Each script builds an individual set.
  Modules are built from the git sources and the build result is automatically appended to
  TDE_DIR/0_logs/build_result.log.
  Usage:
    <dd>_<set_name>.sh [options]
  Options:
    -s N: if specified, skip first N modules from the set of modules

  The sets are logically grouped as "base system", "applications" and "meta packages".
  00_extradeps.sh                     : extra dependencies modules
  01_base_01.sh     - 04_base_04.sh   : base system
  05_apps_system.sh - 16_apps_misc.sh : application modules
  17_meta_packages.sh                 : meta package modules
  91_build_base.sh                    : whole TDE base system in 00_extradeps.sh - 04_base_04.sh
  92_build_applications.sh            : all TDE applications in 05_apps_system.sh - 16_apps_misc.sh

- 99_build_TDE.sh
  Script used to build the complete TDE at once.
  Usage:
    99_build_TDE.sh

- create_repo.sh
  Creates a local repository from the .deb files currently stored in TDE_DIR/2_build/debs.
  Usage:
    [sudo] create_repo.sh [options]
  Options:
    -b (Backup) : create a backup of the existing repository in TDE_DIR/CFG_REPO_DIR.backup


3) * Building hooks *
Hooks are available to execute ad-hoc code before and after the build process. For example this is very useful to automatically apply patches.
There are two type of hooks:
- pre_build : applied before switching the module to quilt format and build
- post_build: applied after the build (dpkg-buildpackage or pbuilder) has terminated
To use a hook, create an executable script (pre_build.sh and/or post_build.sh) in the TDE_DIR/1_git/hooks/<MODULE NAME> folder. The scripts are executed in the build_module.sh environment, so have access to all the variables defined in that file. See the files in the "hook examples" folder for real usage samples.



-------------------------
C) How to use the scripts
-------------------------
1) Follow the steps in section "A) Environment preparation" (only required the first time).
2) cd "TDE_DIR/buildscripts"
3) Update to latest git repository using
     ./update_repositories.sh
4) Run "sudo pbuilder update" at the beginning of the day to update the base package to the latest version. This will speed up
   the process when building several modules in sequence.
5) Build modules as per your needs. You don't need to use "sudo" directly since the scripts will do that automatically if
   required. Just type your sudo password when prompted to do so.
6) Create a local repository from the packages you have just built, to be used as installation repository.
     [sudo] ./create_repo.sh
7) Install TDE as you usually do. For example as follow:
     - sudo apt-get update
     - sudo aptitude install tdebase-trinity   (for a minimalistic TDE environment)
         or
       sudo aptitude install tde-trinity       (for a standard TDE environment)



Examples of real usage:

1) build a single module
   - ./build_module.sh -g -sl "dependencies/libr"       -> build libr package. This is a good test to check
                                                           whether everything is working fine
   - ./build_module.sh -g "tdelibs"       -> build "tdelibs" from git sources in a clean chroot environment
   - ./build_module.sh -g -l -sl "applications/amarok"  -> build "amarok" locally from git sources and
                                                           display building logs during building
   - ./build_module.sh -sh -lr "tdebase"  -> build "tdebase" from the local sources (in TDE_DIR/2_build/build/tdebase)
                                             in a clean chroot environment and launch a shell in case of building failure.
                                             Append the build result (OK, FAILED) to TDE_DIR/scripts/logs/build_result.log
   - ./build_module.sh -g -po "tdelibs"   -> prepare "tdelibs" for building from git sources. Source code will be available
                                             in TDE_DIR/2_build/build/tdelibs. After you have made changes to the source and
                                             want to build the modified package, run './build_module.sh "tdelibs"'

2) build a single set
  (optional) delete the TDE_DIR/0_logs/build_result.log file
  ./01_base_01.sh           -> build this set.
  ./03_base_03.sh -s 3      -> build this set but skip the first 3 modules of the set.

3) build all TDE
  ./99_build_TDE.sh         -> build all TDE

4) [sudo] ./create_repo.sh  -> build or update your local TDE repository