Binary and Tool Outputs

Managing Build Tools

An important aspect of creating a reproducible build process is properly managing the toolchain of compilers, linters, and other executables needed for your builds. Grog provides two mechanisms to help with this:

Binary Outputs: Targets that produce executable files that can be used as tools to build other targets
- Advantage: Enables caching and better reproducibility between hosts
- Disadvantage: Not all tools can be packaged as single portable binaries
Output Checks: Allows checking if some program is already installed on the host by running user-defined commands
- Advantage: Easy to set up using standard installation scripts
- Disadvantage: Less reproducible since Grog can’t verify the exact version or build of the tool

Binary Outputs

A binary output is an executable file produced by a build target that can be used by other targets in your build graph. This approach is ideal for tools that can be compiled or packaged as a single executable.

How to Define a Binary Output

Each target can specify a single bin_output field that points to a file output. When the build completes, Grog will:

Mark this file as executable.
Make it available to dependent target commands via the $(bin :target_name) syntax (see script functions for details).

Example: Creating and Using a Binary Tool

targets:
  # Target that builds a tool
  - name: protoc_compiler
    command: |
      curl -L https://github.com/protocolbuffers/protobuf/releases/download/v3.19.4/protoc-3.19.4-linux-x86_64.zip -o protoc.zip
      unzip protoc.zip -d protoc
      chmod +x protoc/bin/protoc
    outputs:
      - dir::protoc/include/google/protobuf/
    bin_output: protoc/bin/protoc

  # Target that uses the tool
  - name: generate_protos
    dependencies:
      - :protoc_compiler
    inputs:
      - proto/*.proto
    command: |
      # Use the binary tool with $(bin :target_name)
      $(bin :protoc_compiler) --cpp_out=generated proto/*.proto
    outputs:
      - dir::generated

Running Binary Tools Directly

You can also run binary tools directly using the grog run command:

# Run the tool with arguments
grog run //package:protoc_compiler -- --help

# This is equivalent to building the tool and then running it
grog build //package:protoc_compiler
./path/to/protoc/bin/protoc --help

You can also run multiple binaries at the same time by listing each target before the -- separator that introduces the arguments passed to the binaries:

grog run //tools:api_server //tools:web_dev -- some-arg

Defining Build Scripts

Not all binary tools have a build step. Say, for instance, that you have a shell or python script that you want to use as a grog build tool. To do so all you need to do is omit the command field and add the no-cache attribute to ensure that grog does not store the script file in its cache. Here is what that could look like:

targets:
  # Assuming that this script exists in
  # the same directory as the build filr
  - name: protoc_compiler_script
    inputs:
      - compile.sh
    bin_output: compile.sh
    tags:
      - no-cache

  - name: generate_protos
    dependencies:
      - :protoc_compiler_script
    inputs:
      - proto/*.proto
    command: |
      # Use the binary tool with $(bin :target_name)
      $(bin :protoc_compiler_script) --cpp_out=generated proto/*.proto
    outputs:
      - dir::generated

This makes it easy to share scripts across your workspace without having to invoke them by their relative file path.

Output Checks

For tools that can’t easily be packaged as a single binary or that require complex installation procedures, Grog provides output checks. With output checks you can tell grog to assert that a certain command is available in the path at a certain version.

This is the life-cycle of a target’s execution with output checks:

Always run the output checks to see if the condition is already satisfied.
If it isn’t, execute the target and run the checks again to see if the target successfully installed the tool.

It is therefore recommended to keep output checks as light and fast as possible.

How to Define Output Checks

targets:
  - name: ensure_nvm
    command: |
      # Download and install nvm:
      curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash
    output_checks:
      # The target will fail if this command
      # returns a non-zero exit code
      - command: "nvm --version"

  - name: ensure_node_22
    command: |
      nvm install 22
    output_checks:
      # If specified, grog will run this command everytime the tool
      # is used to check that it has the correct version
      - command: "node --version"
        expected_output: "v20.11.1"
      - command: "npm --version"
        expected_output: "10.2.4"
    dependencies:
      - :ensure_nvm

  - name: build_js
    dependencies:
      - :ensure_node_22
    command: |
      # Node is now guaranteed to be available
      node build.js
    inputs:
      - src/**/*.js
      - package.json
    outputs:
      - dist/bundle.js

In this example, the ensure_node target verifies that node is available in the PATH. If it’s not, the command installs it.

Advantages of Output Checks

Works with tools that have complex installation procedures
Allows using system-provided tools without rebuilding them
Simplifies integration with existing toolchains

Best Practices

Prefer binary outputs for critical build tools to ensure reproducibility
Use output checks for tools with complex dependencies or installation requirements
Version your tools explicitly in your build definitions
Consider using Docker for tools with complex system dependencies (see Docker Outputs)
Cache tool installations to speed up builds, especially in CI environments

By properly managing your build tools with Grog, you can create more reproducible and reliable build processes across different environments.