[major] Implement Redis-Based Distributed Locking for GitOps Operations by Hardik-Prajapati-10 · Pull Request #2189 · ibm-mas/cli

Hardik-Prajapati-10 · 2026-04-16T09:28:12Z

Issue

https://jsw.ibm.com/browse/MASCORE-13109

Description

This PR implements a comprehensive Redis-based distributed locking mechanism to replace the legacy Git branch-based locking system for GitOps operations. The change addresses critical issues with concurrent pipeline executions that previously caused race conditions, merge conflicts, and branch pollution.

Key Changes

1. Redis Locking Infrastructure (`image/cli/mascli/functions/gitops_utils`)

Added Redis connection management with TLS support
Implemented atomic lock acquisition using Redis SET NX operations
Added lock release with ownership verification using Lua scripts
Included automatic lock expiry (TTL) to prevent orphaned locks
Added comprehensive retry logic with configurable parameters

2. Container Image Updates

Added redis-cli installation script (image/cli/install/install-redis-cli.sh)
Updated Dockerfile to include redis-cli in the CLI container image (v19.6.1+)
Supports multiple package managers (microdnf, dnf, yum, apt-get, apk)

3. Migrated GitOps Functions (10 functions)

All functions now use gitops_lock_and_modify with Redis locking:

gitops_cp4d_service
gitops_suite_app_config
gitops_deprovision_app_config
gitops_mas_config
gitops_suite_workspace
gitops_deprovision_suite_workspace
gitops_deprovision_cp4d_service
gitops_db2u_database
gitops_rds_db2_database
gitops_deprovision_db2u_database

4. Documentation

Comprehensive setup guide (docs/redis-locking-setup.md)
IBM Toolchain configuration script (docs/ibm-toolchain-redis-setup.sh)
Migration examples and troubleshooting guide

Impact

Reliability Improvements:

Eliminates race conditions through atomic Redis operations
Prevents merge conflicts in concurrent updates
Removes temporary Git branch pollution
Automatic lock expiry prevents orphaned locks
Fail-fast behavior when Redis is unavailable

Breaking Change:

Redis is now required for migrated GitOps flows
No fallback to Git branch locking
Operations fail immediately if Redis is unavailable or misconfigured
GITOPS_USE_REDIS_LOCKING=true must be set

Test Results

Manual Testing

✅ Redis connection validation with TLS
✅ Lock acquisition and release operations
✅ Concurrent pipeline execution without conflicts
✅ Lock TTL expiry behavior
✅ Retry logic under contention
✅ All 10 migrated GitOps functions tested

Integration Testing

✅ IBM Cloud Redis instance connectivity
✅ IBM Toolchain pipeline execution
✅ Multi-pipeline concurrent operations
✅ Error handling for Redis unavailability

Configuration Testing

✅ TLS certificate handling
✅ Authentication (username/password)
✅ Environment variable configuration
✅ Setup script validation (ibm-toolchain-redis-setup.sh)

Backporting

Related Pull Requests

https://github.ibm.com/maximoappsuite/saas-deploy-py/pull/262

⚠️ Notes for Reviewers

Ensure you have understood the guidelines before proceeding with a review.
Ensure all sections in the PR template are appropriately completed.
Critical: This is a breaking change - Redis is now mandatory for migrated GitOps operations
Review the migration strategy in docs/redis-locking-setup.md
Verify the setup script docs/ibm-toolchain-redis-setup.sh for IBM Cloud integration
Check that all 10 migrated functions follow the new locking pattern consistently
Validate error handling for Redis unavailability scenarios
Review security aspects: TLS configuration, credential handling, and password masking

image/cli/mascli/functions/gitops_suite_app_config - Migrated image/cli/mascli/functions/gitops_mas_config - Migrated image/cli/mascli/functions/gitops_deprovision_app_config - Migrated

…ging

whitfiea · 2026-04-17T07:41:22Z

    --mount=type=secret,id=GITHUB_REF_TYPE,env=GITHUB_REF_TYPE \
    umask 0002 && \
    ls /tmp/install && \
+    bash /tmp/install/install-redis-cli.sh && \


normally the third party cli packages are installed as part of the cli-base image at https://github.com/ibm-mas/cli-base so we should move this redis-cli install to that

whitfiea · 2026-04-17T08:07:00Z

+### Required Runtime Requirement
+
+For migrated GitOps flows:
+- `GITOPS_USE_REDIS_LOCKING` must be `true`


It seems that GITOPS_USE_REDIS_LOCKING has to be true, so i am not sure what the value is of having an env var anyway?

whitfiea · 2026-04-17T08:09:11Z

+### Step 2: Configure IBM Toolchain Environment Variables
+
+Add these environment variables to your IBM Toolchain pipeline.
+
+#### In IBM Toolchain → Pipeline → Environment Properties:
+
+| Variable Name | Type | Value | Description |
+|--------------|------|-------|-------------|
+| `REDIS_USERNAME` | Secure | `ibm_cloud_user` | Redis username from credentials |
+| `REDIS_HOST` | Text | `c-abc123.databases.appdomain.cloud` | Redis hostname |
+| `REDIS_PORT` | Text | `31234` | Redis port |
+| `REDIS_PASSWORD` | Secure | `your-redis-password` | Redis password |
+| `REDIS_TLS_CA_CERT_B64` | Secure | `LS0tLS1CRUdJTi...` | Base64-encoded TLS certificate |
+| `GITOPS_USE_REDIS_LOCKING` | Text | `true` | Required for migrated locking flows |
+| `REDIS_TLS` | Text | `true` | Enable TLS (required for IBM Cloud) |
+| `REDIS_DB` | Text | `0` | Redis database number |
+
+#### Optional Tuning Parameters
+
+| Variable Name | Default | Description |
+|--------------|---------|-------------|
+| `GITOPS_LOCK_TTL` | `300` | Lock expires after N seconds |
+| `GITOPS_LOCK_RETRY_MAX` | `100` | Maximum retry attempts |
+| `GITOPS_LOCK_RETRY_DELAY` | `20` | Seconds between retry attempts |


Using the IBM toolchain to call the functions is optional. We should update the docs in https://ibm-mas.github.io/gitops/main/ so state these env vars should be set when executing the functions, and also add that to this file.

whitfiea · 2026-04-17T08:10:51Z

+### Adding redis-cli to Custom Images
+
+If you're building a custom CLI image, add redis-cli installation.
+
+**1. Create installation script** (`image/cli/install/install-redis-cli.sh`):
+```bash
+#!/bin/bash
+set -e
+
+echo "Installing redis-cli..."
+
+if command -v microdnf &> /dev/null; then
+    microdnf install -y redis && microdnf clean all
+elif command -v dnf &> /dev/null; then
+    dnf install -y redis && dnf clean all
+elif command -v yum &> /dev/null; then
+    yum install -y redis && yum clean all
+elif command -v apt-get &> /dev/null; then
+    apt-get update && apt-get install -y redis-tools && rm -rf /var/lib/apt/lists/*
+elif command -v apk &> /dev/null; then
+    apk add --no-cache redis
+else
+    echo "ERROR: No supported package manager found"
+    exit 1
+fi
+
+redis-cli --version
+```
+
+**2. Update Dockerfile** (`image/cli/Dockerfile`):
+```dockerfile
+COPY install /tmp/install
+RUN bash /tmp/install/install-redis-cli.sh && \
+    bash /tmp/install/install-python-packages.sh
+```


We shouldn't need this section as we don't allow a custom cli image

Hardik-Prajapati-10 and others added 28 commits April 6, 2026 16:51

MASCORE-13109

81e5c94

gitops-cp4d-service

8c0ad0f

Migrated 3 functions.

af353a0

image/cli/mascli/functions/gitops_suite_app_config - Migrated image/cli/mascli/functions/gitops_mas_config - Migrated image/cli/mascli/functions/gitops_deprovision_app_config - Migrated

merge

7bd494a

Fix Redis locking: correct function scoping and add configuration log…

a32da51

…ging

dummy

2d42cb0

dummy

8659c1a

redis-cli

2d32a90

redis-cli installed

3e97b81

Update install-redis-cli.sh

cf69f69

Enable redis module stream for RHEL 9 UBI compatibility

a059222

fix error handling to filter timestamp output

b7ffb33

Update gitops_utils

842eeeb

Update gitops_utils

975058b

Update gitops_utils

ef2b7c6

build

5c76068

New Lock Key Format

22796ff

Redis Lock Key

5e7e6b4

Without any password warnings or the -a **** flag in the display.

a1930a1

build

d22442f

Merge branch 'master' into mascore-13109-branch-lock

da35f00

Idempotent cleanup

b95f77a

Handles "No Changes" Scenario

e1bce1e

Redis Authentication and Exit Code Issues

36230ef

Update gitops_suite_workspace

2cfd021

migrated other functions

26ceff9

Removed fallback-to-Git mechanism

7b8f515

Redis Setup Documentation

74f7de3

Hardik-Prajapati-10 requested a review from a team as a code owner April 16, 2026 09:28

Hardik-Prajapati-10 requested a review from whitfiea April 16, 2026 09:32

Hardik-Prajapati-10 requested review from amitpandey0217, mnivedithaa and rbinns April 16, 2026 09:33

Hardik-Prajapati-10 self-assigned this Apr 16, 2026

removed debug logs

435b8cf

whitfiea changed the title ~~Implement Redis-Based Distributed Locking for GitOps Operations~~ [major] Implement Redis-Based Distributed Locking for GitOps Operations Apr 17, 2026

whitfiea requested changes Apr 17, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[major] Implement Redis-Based Distributed Locking for GitOps Operations#2189

[major] Implement Redis-Based Distributed Locking for GitOps Operations#2189
Hardik-Prajapati-10 wants to merge 29 commits intomasterfrom
mascore-13109-branch-lock

Hardik-Prajapati-10 commented Apr 16, 2026 •

edited

Loading

Uh oh!

whitfiea Apr 17, 2026

Uh oh!

whitfiea Apr 17, 2026

Uh oh!

whitfiea Apr 17, 2026

Uh oh!

whitfiea Apr 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Hardik-Prajapati-10 commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Description

Key Changes

1. Redis Locking Infrastructure (image/cli/mascli/functions/gitops_utils)

2. Container Image Updates

3. Migrated GitOps Functions (10 functions)

4. Documentation

Impact

Test Results

Manual Testing

Integration Testing

Configuration Testing

Backporting

Related Pull Requests

⚠️ Notes for Reviewers

Uh oh!

whitfiea Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

whitfiea Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

whitfiea Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

whitfiea Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Hardik-Prajapati-10 commented Apr 16, 2026 •

edited

Loading

1. Redis Locking Infrastructure (`image/cli/mascli/functions/gitops_utils`)