Releases: cloudbase/garm
garm v0.2.0-beta1
GARM v0.2.0-beta1
This is the first beta release from the v0.2 branch, representing a major evolution of GARM. The headline features are GitHub Actions Runner Scale Sets, Gitea forge support, Agent Mode with remote shell access, and a brand new SPA Web UI.
New Features
Scale Sets
GARM now supports GitHub Actions Runner Scale Sets as a first-class alternative to webhook-driven pools. Scale sets use GitHub's native HTTP long-polling message queue API instead of webhooks, providing:
- No webhooks required -- jobs are received via long-poll message sessions, eliminating the need for publicly-reachable webhook endpoints and the risk of lost webhook deliveries.
- GitHub-managed scheduling -- GitHub controls which runner executes which job, improving scheduling efficiency within runner groups.
- Full lifecycle management -- scale sets automatically create JIT (Just-In-Time) runners, provision instances through providers, and clean up after job completion.
- Automatic scaling -- min idle runners and max runners limits are enforced with configurable bootstrap timeouts.
- Runner group assignment -- scale sets can target specific GitHub runner groups at the organization and enterprise levels.
- Label support -- scale sets support custom labels/tags for workflow
runs-ontargeting. - New CLI commands:
garm-cli scaleset list|show|add|update|deleteandgarm-cli scaleset runner list. - New API endpoints:
GET/POST/PUT/DELETE /scalesets,GET /scalesets/{id}/instances. - Full Web UI integration with create, update, and delete support.
Gitea Forge Support (Multi-Forge Architecture)
GARM is no longer GitHub-only. A major refactor introduces a multi-forge architecture with Gitea as the first additional forge:
- Gitea endpoints -- create and manage Gitea server endpoints with
api_base_url,base_url, CA cert bundles, and custom tools metadata URLs. - Gitea credentials -- PAT-based authentication for Gitea instances.
- Repository and organization support -- manage Gitea repos and orgs the same way as GitHub entities.
- Webhook management -- automatic webhook installation/uninstallation for Gitea repos and orgs.
- Gitea runner tools -- automatic fetching of
act_runnerreleases from Gitea's public repository, with support for custom tools metadata URLs and internal tools hosting. - Forge type namespacing -- credentials and endpoints are namespaced by forge type, preventing collisions.
--forge-typeflag -- optional onrepo createandorg create(auto-detected from credentials when omitted).- Docker compose examples -- ready-to-use examples for setting up GARM with Gitea.
Agent Mode
A new deployment model where GARM installs and manages a persistent garm-agent on runner instances:
- Per-entity toggle -- enable agent mode on individual repos, orgs, or enterprises via
--agent-modeflag. - Object store -- new built-in file storage system for managing garm-agent binaries, with upload/download/delete/list/search via API, CLI, and Web UI.
- Automatic tool sync -- GARM periodically fetches the latest garm-agent releases from an upstream GitHub repository and stores them locally. Manual uploads take priority over auto-synced tools.
- WebSocket agent connectivity -- agents connect back to the controller via persistent WebSocket, enabling bidirectional communication.
- Remote shell access -- administrators can open interactive shell sessions on running agents directly from the Web UI (using xterm.js) or API, with per-pool/scale-set enable/disable control.
- Agent heartbeats -- live agent health monitoring with heartbeat staleness detection in the UI.
- Metadata URL -- new
metadata_urlcontroller setting.AgentURLis auto-derived fromMetadataURLorCallbackURLif not explicitly set. - New API endpoints for tool management:
GET/POST/DELETE /tools/garm-agent,GET /tools/garm-agent/{id}/download,POST /tools/garm-agent/sync. - New CLI commands:
garm-cli tools list|upload|delete|force-sync.
Web UI (SPA)
A brand new Single Page Application built with SvelteKit, TypeScript, and Tailwind CSS, embedded directly into the GARM binary:
- Dashboard -- overview of all entities, pools, scale sets, and system status with a setup wizard for first-time configuration.
- Entity management -- full CRUD for repositories, organizations, and enterprises with pool and scale set management inline.
- Pool and scale set management -- create, update, delete pools and scale sets with tag editing, runner group assignment, and template selection.
- Credentials management -- create, update, delete credentials with a dedicated details page showing associated entities.
- Endpoint management -- create, update, delete GitHub and Gitea endpoints with CA cert upload.
- Runner/Instance management -- list all runners with status badges, detail pages with live event logs, and delete actions.
- Runner install templates -- manage custom runner installation templates.
- Object storage browser -- upload, download, search, and manage files in the object store.
- Real-time updates -- WebSocket integration for live entity and instance status updates with connection status indicator.
- Shell terminal -- integrated xterm.js terminal for agent-mode runners with multi-tab support, maximize/minimize, and drag-to-resize.
- Webhook management -- install/uninstall webhooks from entity detail pages.
- Dark mode -- full dark/light theme support.
- Responsive design -- mobile-first layout with DataTable components that switch between card (mobile) and table (desktop) views.
- Embeddable -- built assets are compiled into the Go binary via
go:embed. No external file serving needed. - Reverse proxy support -- can be served behind nginx or similar, with documented config for WebSocket upgrades.
Enable the Web UI in your config:
[apiserver.webui]
enable=trueThen navigate to http://<your-garm-host>:<port>/ui/.
Metrics & Observability
- Job metrics -- new Prometheus metrics for workflow jobs including owner, workflow name, scale set job ID, and job URL.
- Rate limit metrics -- GitHub/Gitea API rate limit consumption is now exposed as Prometheus metrics with per-credentials tracking.
- GitHub operation metrics -- operation counts and failure counts by operation type and entity scope.
Runner Install Templates
- Template management -- create, list, get, update, and delete custom runner installation templates via API, CLI (
garm-cli template list|show|add|update|delete|restore), and Web UI. - Template restore -- restore built-in default templates if customized templates are broken.
- Per-pool/scale-set template assignment -- pools and scale sets can reference a specific template by ID.
- Forge-aware templates -- separate templates for GitHub and Gitea, for both Linux and Windows.
- No more
jqrequirement -- runner install scripts no longer requirejqto be installed at runtime.
Generation Tracking and Runner Rotation
- Generation tracking -- pools and scale sets now track a configuration generation counter. When configuration changes (image, flavor, extra specs, etc.), the generation increments, allowing identification of instances running with outdated configuration.
- Runner rotation -- new
garm-cli pool runner rotateandgarm-cli scaleset runner rotatecommands allow rotating runners, with--outdatedfiltering to target only instances from a previous generation, and--dry-runmode to preview changes.
CA Certificate Bundle for Controller
- CA bundle in controller settings -- a configurable CA certificate bundle can be set at the controller level. It is automatically injected into runner userdata, allowing runners to trust custom/internal CAs when communicating with the GARM controller. Windows is also supported.
Improvements
CLI
garm-cli top-- new interactive terminal dashboard for monitoring live GARM metrics. Displays a multi-panel TUI with summaries of entities, pools, scale sets, instances, and jobs. Refreshes in real-time via WebSocket. Navigate panels withTaband scroll with arrow keys.- Friendly names everywhere -- most commands now accept entity names in addition to UUIDs.
--endpointflag -- added to all relevant commands for filtering by endpoint.--filteroption -- rudimentary filter support when listing entities.- Improved
garm-cli debug-log-- streams GARM server logs in real-time via WebSocket with rich formatting:- Color-coded output -- log levels are color-coded (red for ERROR, yellow for WARN, blue for INFO, magenta for DEBUG) with auto-detection of terminal color support. Override with
--enable-color,NO_COLOR=1, orFORCE_COLOR=1. - Log level filtering (
--log-level) -- filter by minimum level (DEBUG, INFO, WARN, ERROR). - Attribute filtering (
--filter key=value) -- filter log entries by attribute values or message content (msg=text). Can be specified multiple times. - Filter mode (
--filter-mode any|all) -- choose between OR (any filter matches) or AND (all filters must match) logic. - Highlighting (
--highlight key) -- highlight specific attributes or message content in the output.
- Color-coded output -- log levels are color-coded (red for ERROR, yellow for WARN, blue for INFO, magenta for DEBUG) with auto-detection of terminal color support. Override with
- Runner referencing by ID -- runners can now be referenced by their numeric ID in addition to name.
garm-cli top
garm-cli debug-log
Performance & Caching
- **Entity loading in paralle...
garm v0.1.8
Welcome to GARM v0.1.8
This is a bugfix release.
What's Changed
- Bump provider versions by @gabriel-samfira in #464
- Fix typo by @gabriel-samfira in #465
- Check for pool max runners in CreateInstance tx by @gabriel-samfira in #566
- Update dependencies by @gabriel-samfira in #575
- Make runner names lowercase by @gabriel-samfira in #576
- Update dependencies by @gabriel-samfira in #583
- Remove references to Equinix by @gabriel-samfira in #584
- Update dockerfile by @gabriel-samfira in #585
- Fix leaky jit config by @gabriel-samfira in #621
- Update all dependencies by @gabriel-samfira in #682
Full Changelog: v0.1.6...v0.1.8
garm v0.1.7
Welcome to GARM v0.1.7!
This is a bugfix release. No new features were added.
What's Changed
- Bump provider versions by @gabriel-samfira in #464
- Fix typo by @gabriel-samfira in #465
- Check for pool max runners in CreateInstance tx by @gabriel-samfira in #566
- Update dependencies by @gabriel-samfira in #575
- Make runner names lowercase by @gabriel-samfira in #576
- Update dependencies by @gabriel-samfira in #583
- Remove references to Equinix by @gabriel-samfira in #584
Full Changelog: v0.1.6...v0.1.7
garm v0.1.6
Welcome to GARM v0.1.6!
Starting with this release, GARM will now have a release/v0.1 and a release/v0.2 (which will receive updates from main) channel. The v0.1.X releases will contain no new major features, just bug fixes. The goal is to offer a stable release that will work with any existing integrations like the k8s operator, and still allow to break compatibility and add new features.
The main branch will now diverge from release/v0.1 and checkpoints from main will be merged into release/v0.2.
That being said, this release has a number of stability fixes and bug fixes that should help especially for environments that are under heavy load.
New Linode provider
Thanks to @tormath1 we can now create runners on Linode. This new provider is included in the v0.1.6 container image.
What's Changed
- Replace ngrok by @gabriel-samfira in #286
- Switch to self hosted runner by @gabriel-samfira in #287
- Allow using LXD image mirror by @gabriel-samfira in #288
- Add provider interface versioning by @fabi200123 in #278
- Do a case insensitive search for tags by @gabriel-samfira in #296
- Add omitempty to all struct fields by @gabriel-samfira in #297
- Add --format command line option by @gabriel-samfira in #298
- Add a default value to the new --format option by @gabriel-samfira in #300
- Copy the CA certificates to final image by @gabriel-samfira in #301
- Use errors.Wrap() in repositories.go by @gabriel-samfira in #299
- Add Using Cached Runner documentation by @fabi200123 in #302
- Remove conflicting short hand option by @gabriel-samfira in #306
- Improve error messages in garm log by @maigl in #314
- fix: remove duplication in Makefile by @maigl in #315
- Fix hardcoded user value in template by @gabriel-samfira in #316
- Fix nil pointer dereference when rendering message by @gabriel-samfira in #318
- Update dependencies by @gabriel-samfira in #319
- Create dependabot.yml by @gabriel-samfira in #320
- Bump github.com/jedib0t/go-pretty/v6 from 6.6.4 to 6.6.5 by @dependabot[bot] in #321
- Update dependencies by @gabriel-samfira in #322
- Bump golang.org/x/oauth2 from 0.24.0 to 0.25.0 by @dependabot[bot] in #325
- Bump github.com/bradleyfalzon/ghinstallation/v2 from 2.12.0 to 2.13.0 by @dependabot[bot] in #324
- Update Performance Considerations - Cached runners section by @fabi200123 in #327
- Update integration-tests.yml by @gabriel-samfira in #335
- Bump golang.org/x/oauth2 from 0.25.0 to 0.26.0 by @dependabot[bot] in #341
- Bump golang.org/x/sync from 0.10.0 to 0.11.0 by @dependabot[bot] in #340
- Bump golang.org/x/crypto from 0.31.0 to 0.32.0 by @dependabot[bot] in #326
- Relax URLs validation by @gabriel-samfira in #343
- Relax description validation by @gabriel-samfira in #342
- Bump golang.org/x/crypto from 0.32.0 to 0.33.0 by @dependabot[bot] in #346
- Add additional info when listing resources by @gabriel-samfira in #347
- Update event logging by @gabriel-samfira in #348
- Bump github.com/jedib0t/go-pretty/v6 from 6.6.5 to 6.6.6 by @dependabot[bot] in #349
- Add knob to tweak _busy_timeout by @gabriel-samfira in #328
- Add a backoff mechanism when deleting runners by @gabriel-samfira in #329
- Update dependencies by @gabriel-samfira in #355
- Bump golang.org/x/crypto from 0.34.0 to 0.35.0 by @dependabot[bot] in #356
- Update dependencies by @gabriel-samfira in #363
- Bump github.com/go-openapi/swag from 0.23.0 to 0.23.1 by @dependabot[bot] in #364
- Bump github.com/go-openapi/errors from 0.22.0 to 0.22.1 by @dependabot[bot] in #365
- Bump github.com/BurntSushi/toml from 1.4.0 to 1.5.0 by @dependabot[bot] in #366
- Bump github.com/golang-jwt/jwt/v5 from 5.2.1 to 5.2.2 in the go_modules group by @dependabot[bot] in #367
- Bump github.com/golang-jwt/jwt/v4 from 4.5.1 to 4.5.2 in the go_modules group by @dependabot[bot] in #368
- Bump golang.org/x/sync from 0.12.0 to 0.13.0 by @dependabot[bot] in #369
- Bump golang.org/x/crypto from 0.36.0 to 0.37.0 by @dependabot[bot] in #371
- Bump golang.org/x/oauth2 from 0.28.0 to 0.29.0 by @dependabot[bot] in #370
- Bump github.com/prometheus/client_golang from 1.21.1 to 1.22.0 by @dependabot[bot] in #372
- Update all dependencies by @gabriel-samfira in #378
- set base URL in ghinstallation transport by @gabriel-samfira in #382
- Fix deadline check logic by @gabriel-samfira in #409
- Make the default github.com endpoint mutable by @gabriel-samfira in #410
- Relax update endpoint validation by @gabriel-samfira in #414
- Trigger tests on release branches by @gabriel-samfira in #420
- Add write lock for sqlite3 by @gabriel-samfira in #415
- Update dependencies. by @gabriel-samfira in #434
- Fix show webhooks. by @gabriel-samfira in #435
- Set http transport config by @gabriel-samfira in #439
- Update dependencies by @gabriel-samfira in #460
- Update docs to reflect new release by @gabriel-samfira in #461
Full Changelog: v0.1.5...v0.1.6
garm v0.1.5
Welcome to GARM v0.1.5!
Before we dive into the highlights, as always a few notes.
Database upgrade ⚠️
This version will execute a database migration. While we try not to break anything when making database migrations, it's always a good idea to create a backup of your DB as well as any configuration files. If anything goes wrong, you can revert the changes. Database downgrades are not supported. If you need to go back to a previous version, please also restore the DB from backup.
To safely create a backup of you DB, you can use the sqlite3 CLI:
# Change the path to your DB to the one configured in your config.toml
sqlite3 /etc/garm/garm.db
.backup /path/to/backup/location/garm-backup.dbBreaking changes ⚠️
This version has moved a number of options from the config file, to the database. During the upgrade process from v0.1.4 to v0.1.5 please do not make any changes to your current config. GARM will read the config and migrate any of the settings present there, to the DB, automatically. After you start GARM with the new version and the migration has finished (it should be just a matter of seconds), you can then edit your config and remove the relevant fields.
Things that have been moved from the config to the database:
- The
metadata_url,callback_urlandwebhook_urlsettings have now been moved to the database in the controller table. These settings can now be updated using thegarm-clicommand line tool. - Github credentials - credentials are now created via the
garm-clicommand line tool. Details bellow.
Note
If you receive an error regarding missing URLs, you must update your controller settings using garm-cli controller update. Make sure all your URLs are up to date, including the webhooks URL.
Highlights
This version is packed with features and changes. Although GARM can handle a large amount of runners with ease, we are working towards enabling it to eventually scale out. As a result, we've started moving some of the settings that were in the config file, to the database. The goal is to eventually have a single source of truth for those settings and enable GARM to react to any changes immediately without the need to reload the app. This is just one motivation behind the changes in this release, but as you'll see, there are many other changes that enable you to do more with GARM.
GitHub endpoints
With the migration of credentials to the database, GARM now has the notion of "github endpoints". A github endpoint represents a deployment of GHES or GitHub itself. It is a way to tell GARM to which API endpoints it needs to connect in order to use the supplied credentials or to execute operations against a particular entity (repo, org or enterprise).
All github credentials and all entities are now bound to a github endpoint. This ensures consistency and correctness when dealing with these types of resources.
See the github endpoint section of the documentation.
Github App Support
We now have the ability to configure GARM to use GitHub Apps to access the github API. This gives us a much higher rate limit for API calls (15.000/hour compared to 5000/hour as is the case of PATs).
Check out the github credentials documentation section for more info on how to configure app credentials, as well as the required credentials for GitHub Apps.
All sensitive info is encrypted at rest and is never returned by the API.
Runner default labels
Starting with version v2.305.0 of the GitHub actions runner, the default lables ($ARCH, $OS and self-hosted) are no longer added if the --no-default-labels flag is used. GARM now takes advantage of this feature to create runners without default labels. The default labels can still be added explicitly if you require them.
Pool balancing strategies
You can now specify two modes of balancing for jobs that get handled by a configured entity. The balancing mode can be set on the entity itself, and this will influence how the pool manager for that entity will behave. The two modes of operation are:
roundrobin- During the consolidation loop, if we have multiplequeuedjobs, theroundrobinstrategy will balance each job to a different matching pool. This is useful if you want to spread the workload evenly across multiple clouds or regions. This is the default.pack- The pack strategy will try to fill up the first pool before moving on to the next, This is useful if you have cheap runners you want to use first before moving on to pools of more expensive runners.
To set the balancing stratedy for a repo, org or enterprise you can use the following command:
garm-cli repo update --pool-balancer-type pack <repo_id>Additionally, pools now have a priority field. This priority field is used when sorting pools that match jobs which need to be handled. You can set the priority when you create the pool or you can update it later. The higher the number, the more likely the pool is to be at the beginning of the list.
A new events websocket endpoint
We now have a new websocket endpoint that can be used by projects which integrate with GARM. This websocket endpoint streams database events as they happen in GARM itself, allowing applications to not need to poll for updates.
Here is a demo of an earlier implementation, in action:
For details regarding the current implementation and how to use it, check out the documentation.
There are many other fixes and features added to this release, a lot of which are internal to GARM but they do have a positive impact on stability and reliability. I invite you to check out the full list of changes, bellow.
Thanks to everyone that has contributed to this release, be it via pull requests, raising issues or helping us debug weird behavior!
What's Changed
- Dockerfile: Build actually static binaries by @jepio in #215
- chore: refactor metrics endpoint by @bavarianbidi in #216
- extend metrics for github and provider executions by @bavarianbidi in #217
- Introduce golangci config and
make helpby @bavarianbidi in #218 - Allow integration tests to run locally by @gabriel-samfira in #219
- Install dependencies and set RUN_USER by @gabriel-samfira in #220
- Switch to apg for password gen by @gabriel-samfira in #221
- Add script and service template by @gabriel-samfira in #222
- Use user service by @gabriel-samfira in #223
- simplify integration test job by @gabriel-samfira in #224
- Fix variable name by @gabriel-samfira in #225
- Add GitHub App support by @gabriel-samfira in #228
- Add ToC to using_garm.md by @gabriel-samfira in #229
- Allow bypassing Unauthorized error when deleting runner by @gabriel-samfira in #231
- Add job info in runner list by @gabriel-samfira in #232
- Add pool balancing strategy by @gabriel-samfira in #233
- Add OCI to provider list by @gabriel-samfira in #234
- Refactor internal gh client by @gabriel-samfira in #235
- Fix JIT config with empty runner group name by @gabriel-samfira in #237
- Check for nil pointer before dereferencing by @gabriel-samfira in #238
- Fix potential nil pointer dereference in cli by @gabriel-samfira in #239
- Deduplicate db code by @gabriel-samfira in #240
- Slightly simplify code by @gabriel-samfira in #241
- Remove some code, move some code around by @gabriel-samfira in #242
- Update dependencies and tests by @gabriel-samfira in #244
- Move github credentials to the database by @gabriel-samfira in #243
- Add TOC by @gabriel-samfira in #247
- Remove check for duplicate pool by @gabriel-samfira in #246
- New runners without default labels by @bavarianbidi in #249
- Update go-swagger and run generate by @gabriel-samfira in #251
- Add note about stable release documentation by @gabriel-samfira in #252
- fix: use the american english type of cancelled by @bavarianbidi in #254
- fix: remove unnecessary github api call by @bavarianbidi in #255
- Move URLs from default section of config to DB by @gabriel-samfira in #256
- Fix typos by @gabriel-samfira in #259
- Refactor Integration tests by @fabi200123 in https://g...
garm v0.1.4
Welcome to GARM version v0.1.4!
Documentation relevant to this version is available at: https://github.com/cloudbase/garm/tree/v0.1.4. The main branch contains code that is still under development, and the documentation is relevant to that code. When deploying stable versions, please view the docs available under the relevant tag.
Before we dive into the highlights, there are a couple of things we need to mention.
Breaking change warning ⚠️
This version removes the LXD internal provider in favor of two new external providers:
This was done mainly due to the licensing change of LXD. If you're using LXD and are upgrading from previous versions, you must download the LXD external provider and configure it.
Another breaking (sort of) change is in the garm-cli. The --force flag in the runner remove command, is no longer mandatory when removing a runner. Moreover, the --force flag now ignores provider errors. In previous versions, if a provider erred when removing a runner, GARM would keep trying until it succeeded. The problem was that if a provider was misconfigured, we would be stuck with a runner in error state until re manually removed it from the database. The --force flag allows us to remove a runner from GARM even if the provider returns an error.
Database upgrade ⚠️
This version will execute a database migration. While we try not to break anything when making database migrations, it's always a good idea to create a backup of your DB as well as any configuration files. If anything goes wrong, you can revert the changes.
To safely create a backup of you DB, you can use the sqlite3 CLI:
# Change the path to your DB to the one configured in your config.toml
sqlite3 /etc/garm/garm.db
.backup /path/to/backup/location/garm-backup.dbHighlights
This release is packed with optimizations, changes and a couple of nice features.
JIT runners
The main highlight of this release is the ability to use just-in-time self-hosted runners. To accommodate this change, GARM now has more robust metadata endpoints that can serve the needed JIT files. This will most likely be expanded in the future to include setup scripts for clouds where we have userdata size constraints.
JIT runners allows us to avoid sending a runner registration token over the wire. Registration tokens can be used to register multiple runners and have a validity of one hour. A bad actor may intercept this token and use it to register their own runners. With JIT runners, only one runner may use the credentials at any given point in time. The credentials are fetched from GARM by the runner when it spins up.
This feature can be disabled in the provider config by specifying:
[[provider]]
disable_jit_config = trueeasily access controller info
We now have the ability to gain more insight into how GARM is configured. We've added a new command which can show us some info about the controller:
ubuntu@garm:~$ garm-cli controller-info show
+------------------------+----------------------------------------------------------------------------+
| FIELD | VALUE |
+------------------------+----------------------------------------------------------------------------+
| Controller ID | a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3 |
| Hostname | garm |
| Metadata URL | https://garm.example.com/api/v1/metadata |
| Callback URL | https://garm.example.com/api/v1/callbacks |
| Webhook Base URL | https://garm.example.com/webhooks |
| Controller Webhook URL | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3 |
+------------------------+----------------------------------------------------------------------------+Easy webhook installation
Installing webhooks for entities (repos, orgs, enterprises) can be a chore. In this version, we've added the ability to let GARM install the needed webhook, with the correct settings by running a simple command. To make use of this feature, the PAT you're using must have access to admin:org_hook:
ubuntu@garm:~$ garm-cli org add \
--credentials org_token \
--name exampleOrg \
--install-webhook \
--random-webhook-secret
+----------------------+--------------------------------------+
| FIELD | VALUE |
+----------------------+--------------------------------------+
| ID | b90911e1-8727-4bb7-a1eb-96855d73a27b |
| Name | exampleOrg |
| Credentials | org_token |
| Pool manager running | true |
+----------------------+--------------------------------------+This command will add the org exampleOrg to your GARM controller and install a webhook for it with a random webhook secret. You can also install a webhook for an existing organization or repository:
ubuntu@garm:~$ garm-cli repo webhook install b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD | VALUE |
+--------------+----------------------------------------------------------------------------+
| ID | 449734752 |
| URL | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3 |
| Events | [workflow_job] |
| Active | true |
| Insecure SSL | false |
+--------------+----------------------------------------------------------------------------+To view the status of a webhook:
ubuntu@garm:~$ garm-cli repo webhook show b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD | VALUE |
+--------------+----------------------------------------------------------------------------+
| ID | 449734752 |
| URL | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3 |
| Events | [workflow_job] |
| Active | true |
| Insecure SSL | false |
+--------------+----------------------------------------------------------------------------+You can also uninstall a webhook:
ubuntu@garm:~$ garm-cli repo webhook uninstall b90911e1-8727-4bb7-a1eb-96855d73a27bWebhooks installed by the above commands are always namespaced to the controller ID you get when you run garm-cli controller-info show. This way, when we remove a webhook, we don't accidentally remove someone else's hook. You can also manually install we webhook just like before. You don't need to namespace it to the controller ID. GARM will see the webhook and let you know if the hook is already installed.
Removing a repo or org will also clean up the webhook if it was namespaced to our controller. You can opt to keep the hook by passing the --keep-webhook flag.
Webhook installation is not available for enterprises.
Structured logging
GARM has now switched to the slog standard package for structured logging. As part of this change, we now have a dedicated [logging] config section where you can set the log level, log format, enable the log streamer, etc. Check out the sample config for more info.
Other updates
There are a lot of bug fixes and stability updates that should make this version behave better in relation to the GitHub API.
What's Changed
- Update to latest version of garm-provider-common and update docs by @gabriel-samfira in #144
- Add GitHub workflow for integration tests by @mihaelabalutoiu in #145
- Add upload artifacts and log instance details on timeout exceeded by @mihaelabalutoiu in #146
- Add workflow dispatch trigger by @gabriel-samfira in #147
- Set pipefail to the script and log org/repo details on timeout exceeded by @mihaelabalutoiu in #148
- Cleaning up leftover runners for
org/repoby @mihaelabalutoiu in #149 - Add controller info by @gabriel-samfira in #150
- Log
orgPool/repoPooldetails on timeout exceeded by @mihaelabalutoiu in https://github...
garm v0.1.4-rc1
Welcome to GARM version v0.1.4-rc1!
This is a pre-release of GARM and it is packed with changes.
Before we dive into the highlights, there are a couple of things we need to mention.
Breaking change warning ⚠️
This version removes the LXD internal provider in favour of two new external providers:
This was done mainly due to the licensing change of LXD. If you're using LXD and are upgrading from previous versions, you must download the LXD external provider and configure it.
Database upgrade ⚠️
This version will execute a database migration. While we try not to break anything when making database migrations, it's always a good idea to create a backup of your DB as well as any configuration files. If anything goes wrong, you can revert the changes.
To safely create a backup of you DB, you can use the sqlite3 CLI:
# Change the path to your DB to the one configured in your config.toml
sqlite3 /etc/garm/garm.db
.backup /path/to/backup/location/garm-backup.dbHighlights
This release is packed with optimizations, changes and a couple of nice features.
JIT runners
The main highlight of this release is the ability to use just-in-time self-hosted runners. To accommodate this change, GARM now has more robust metadata endpoints that can serve the needed JIT files. This will most likely be expanded in the future to include setup scripts for clouds where we have userdata size constraints.
JIT runners allows us to avoid sending a runner registration token over the wire. Registration tokens can be used to register multiple runners and have a validity of one hour. A bad actor may intercept this token and use it to register their own runners. With JIT runners, only one runner may use the credentials at any given point in time. The credentials are fetched from GARM by the runner when it spins up.
This feature can be disabled in the provider config by specifying:
[[provider]]
disable_jit_config = trueeasily access controller info
We now have the ability to gain more insight into how GARM is configured. We've added a new command which can show us some info about the controller:
ubuntu@garm:~$ garm-cli controller-info show
+------------------------+----------------------------------------------------------------------------+
| FIELD | VALUE |
+------------------------+----------------------------------------------------------------------------+
| Controller ID | a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3 |
| Hostname | garm |
| Metadata URL | https://garm.example.com/api/v1/metadata |
| Callback URL | https://garm.example.com/api/v1/callbacks |
| Webhook Base URL | https://garm.example.com/webhooks |
| Controller Webhook URL | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3 |
+------------------------+----------------------------------------------------------------------------+Easy webhook installation
Installing webhooks for entities (repos, orgs, enterprises) can be a chore. In this version, we've added the ability to let GARM install the needed webhook, with the correct settings by running a simple command. To make use of this feature, the PAT you're using must have access to admin:org_hook:
ubuntu@garm:~$ garm-cli org add \
--credentials org_token \
--name exampleOrg \
--install-webhook \
--random-webhook-secret
+----------------------+--------------------------------------+
| FIELD | VALUE |
+----------------------+--------------------------------------+
| ID | b90911e1-8727-4bb7-a1eb-96855d73a27b |
| Name | exampleOrg |
| Credentials | org_token |
| Pool manager running | true |
+----------------------+--------------------------------------+This command will add the org exampleOrg to your GARM controller and install a webhook for it with a random webhook secret. You can also install a webhook for an existing organization or repository:
ubuntu@garm:~$ garm-cli repo webhook install b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD | VALUE |
+--------------+----------------------------------------------------------------------------+
| ID | 449734752 |
| URL | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3 |
| Events | [workflow_job] |
| Active | true |
| Insecure SSL | false |
+--------------+----------------------------------------------------------------------------+To view the status of a webhook:
ubuntu@garm:~$ garm-cli repo webhook show b90911e1-8727-4bb7-a1eb-96855d73a27b
+--------------+----------------------------------------------------------------------------+
| FIELD | VALUE |
+--------------+----------------------------------------------------------------------------+
| ID | 449734752 |
| URL | https://garm.example.com/webhooks/a4dd5f41-8e1e-42a7-af53-c0ba5ff6b0b3 |
| Events | [workflow_job] |
| Active | true |
| Insecure SSL | false |
+--------------+----------------------------------------------------------------------------+You can also uninstall a webhook:
ubuntu@garm:~$ garm-cli repo webhook uninstall b90911e1-8727-4bb7-a1eb-96855d73a27bWebhooks installed by the above commands are always namespaced to the controller ID you get when you run garm-cli controller-info show. This way, when we remove a webhook, we don't accidentally remove someone else's hook. You can also manually install we webhook just like before. You don't need to namespace it to the controller ID. GARM will see the webhook and let you know if the hook is already installed.
Removing a repo or org will also clean up the webhook if it was namespaced to our controller. You can opt to keep the hook by passing the --keep-webhook flag.
Webhook installation is not available for enterprises.
Other updates
There are a lot of bug fixes and stability updates that should make this version behave better in relation to the GitHub API. I'm pretty sure I'm forgetting something, but you can consult the list bellow for a full list of changes.
In any case, take it for a spin! If you run into any issue, feel free to open a new issue.
What's Changed
- Update to latest version of garm-provider-common and update docs by @gabriel-samfira in #144
- Add GitHub workflow for integration tests by @mihaelabalutoiu in #145
- Add upload artifacts and log instance details on timeout exceeded by @mihaelabalutoiu in #146
- Add workflow dispatch trigger by @gabriel-samfira in #147
- Set pipefail to the script and log org/repo details on timeout exceeded by @mihaelabalutoiu in #148
- Cleaning up leftover runners for
org/repoby @mihaelabalutoiu in #149 - Add controller info by @gabriel-samfira in #150
- Log
orgPool/repoPooldetails on timeout exceeded by @mihaelabalutoiu in #151 - Fix timeout logic by @mihaelabalutoiu in #152
- Small fixes for the docs by @mihaelabalutoiu in #153
- Enable Windows builds by @gabriel-samfira in #156
- Add webhook management for repositories and organizations by @gabriel-samfira in #154
- Fix e2e secrets generation by @ionutbalutoiu in #162
- Add webhooks integration tests for
organizationandrepositoryby @mihaelabalutoiu in #161 - Use apg to generate passwords by @gabriel-samfira in #164
- Fix nil pointer dereference by @gabriel-samfira in #165
- Fix TLS client bug by @gabriel-samfira in #166
- Refactor integration E2E tests by @ionutbalutoiu in https://github.com/clo...
garm v0.1.3
Welcome to GARM version v0.1.3!
This release brings a number of really nice improvements.
Highlights
Thanks to @mihaelabalutoiu, we now have proper OpenAPI annotations, as well as a full client generated from them. The GARM CLI has switched to this new client, which is located in the root of the project. If you plan to integrate with GARM in any way, I encourage you to use this client instead of the old, hand-written one we previously had.
Another interesting change in this release is the fact that we split some functionality from GARM itself, into an external package. This new package will hold functionality that is common to GARM itself and external providers.
As part of this migration we also added the ability to:
- Completely override the runner installation script template. This can be done with any provider that leverages the new external package and calls the helper functions that generate the install script. This includes the current LXD, OpenStack and Azure providers.
- Send additional template context (if overriding the install template)
- For Linux only (for now):
- Add pre-install scripts. These scripts are executed in alphabetical order before the runner install script is run.
This can all be done via extra-specs.
Upgrade notes
This update does not come with database schema changes, however, we do recommend you update any external providers to their latest versions if you update GARM.
What's Changed
- Fix
apiserver/routers/routers.gotypo by @mihaelabalutoiu in #131 - Add dockerfile and workflow by @gabriel-samfira in #132
- Properly set garm-cli version by @gabriel-samfira in #133
- Implement more features for the swagger client API by @mihaelabalutoiu in #134
- Add more functionality to swagger client library by @mihaelabalutoiu in #136
- Small fixes to the GARM api server by @ionutbalutoiu in #138
- Fix swagger annotations by @mihaelabalutoiu in #139
- Complete the swagger client library with full functionality by @mihaelabalutoiu in #140
- Update some docs by @gabriel-samfira in #129
- Remove unfinished doc by @gabriel-samfira in #141
- Move code to external package by @gabriel-samfira in #143
- Update
garm-clitool to use the new swagger generated client library by @mihaelabalutoiu in #142
Full Changelog: v0.1.2...v0.1.3
garm v0.1.2
Welcome to garm version v0.1.2!
This release brings a a lot of performance improvements and bug fixes.
Highlights
The most notable change in this release is in the way garm reacts to jobs sent by GitHub. Previously, garm reacted to new web hooks, but never recorded them in any way. When a queued hook came in, garm would attempt to create a new runner for it. The outcome of that action would depend on a few factors like if we had room for a new runner (max-runners), or if we had a pool that matched the requested tags.
One other shortcoming of the old approach was the fact that if we had multiple hierarchy levels configured (repo, org, enterprise), they would all receive the web hook with the queued job and would each create one runner for the same job. This had the potential to spin up resources for no reason.
This release introduces a new job tracking feature. From now on, garm will record all jobs that GitHub sends and the entity that received them (repo, org, enterprise). It will then attempt to periodically (roughly every 5 seconds) consume the jobs found in queued state and spin up runners for them. If we have multiple hierarchy levels configured, they will contend to spin up a runner for a job, but in the end only one of them will spin up a runner. This reduces the number of runners that get spun up for the same job. Job tracking also means that if a pool reaches max-runners we no longer lose jobs. We will eventually spin up a runner if the job is still in queued state. Which also means that it should be safe to use pools with min-idle-runners set to zero.
Other highlights
- WAL is now enabled for the DB store. This was an oversight that caused load on large scale deployments.
- Aditional indexes have been created for further speedups
- Pool manager loops have been refactored. Each operation now runs in their own loop.
- If multiple pools match a certain label, garm will now attempt to round-robin runners in matching pools
Upgrade notes
This release brings a number of changes to the database. Please back up your sqlite database before updating. The procedure is simple:
systemctl stop garm
cp /etc/garm/garm.db /etc/garm/garm.db.backup
# update garm then start
systemctl start garmWhat's Changed
- Add more
runner/pools.gounit tests by @mihaelabalutoiu in #110 - Fix
runner/pools.gotypo by @mihaelabalutoiu in #111 - Refactor pool manager loop by @gabriel-samfira in #112
- Use su to install the runner by @gabriel-samfira in #115
- Rotate log file on SIGHUP by @ionutbalutoiu in #116
- Add logging and fix backoff loop by @gabriel-samfira in #118
- Generate client library via swagger by @mihaelabalutoiu in #119
- Add doc about performance considerations by @SystemKeeper in #120
- Replace wait implementation with errgroup by @gabriel-samfira in #121
- Add job tracking by @gabriel-samfira in #86
- Update comment on function by @gabriel-samfira in #122
- Add entity update subcommand by @gabriel-samfira in #125
- Fix entity update by @gabriel-samfira in #124
- Add more swagger client API implementation by @ionutbalutoiu in #126
- Add swagger client auth info by @ionutbalutoiu in #127
- Set on delete for jobs by @gabriel-samfira in #128
New Contributors
- @SystemKeeper made their first contribution in #120
Full Changelog: v0.1.1...v0.1.2
garm v0.1.1
Welcome to garm version v0.1.1!
This release adds some resiliency fixes to garm.
Highlights
- Instances that fail to set up their GitHub agent and transition to
failedare now reaped. - The cloud-init userdata now retries setting up the github agent for up to 5 times before giving up
- You can now specify
extra_packagesextra specs with the LXD provider, which allows you to install additional packages on runner nodes. (#103)
What's Changed
- Validate provider responses by @gabriel-samfira in #102
- Allow installing additional packages in lxd container by @HippocampusGirl in #103
- Add more
users.gounit tests by @mihaelabalutoiu in #105 - Add test cases for the
runner/pools.goby @mihaelabalutoiu in #107 - Add more test cases for the
/database/sql/users.goby @mihaelabalutoiu in #108 - Reap failed agent by @gabriel-samfira in #106
- Fix
runner/pools.gotypo by @mihaelabalutoiu in #109
New Contributors
- @HippocampusGirl made their first contribution in #103
Full Changelog: v0.1.0...v0.1.1

