Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new AGENTS.md document intended to onboard contributors to the caterpillar Go CLI, describing architecture, task development patterns, pipeline YAML structure (linear + DAG), and operational/PR workflow guidance.
Changes:
- Introduces a comprehensive agent/contributor guide for building, running, and extending caterpillar tasks
- Documents pipeline YAML syntax (including DAG mode) and task implementation conventions
- Adds operational notes, smoke-test instructions, and PR process guidelines
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| - **Batch mode** (most pipelines) — runs, processes all records, and exits. | ||
| - **Server mode** — when the pipeline starts with an `http_server` task, the CLI acts as a long-running HTTP server. Incoming requests are converted to `*record.Record` values and emitted downstream to the rest of the pipeline. The binary does not exit until the server shuts down (or `end_after` is configured). | ||
|
|
||
| Apply these instructions to `/Users/prasadlohakpure/Desktop/go_projects/src/github.com/patterninc/caterpillar`. Treat paths and commands below as relative to that location unless explicitly stated otherwise. |
There was a problem hiding this comment.
The doc hard-codes a developer-local absolute path ("/Users/prasadlohakpure/.../caterpillar") and tells readers to treat all paths as relative to it. This is not portable and also leaks a personal workstation path; please rephrase to refer to the repository root (e.g., "from the repo root") or omit the path entirely.
| Apply these instructions to `/Users/prasadlohakpure/Desktop/go_projects/src/github.com/patterninc/caterpillar`. Treat paths and commands below as relative to that location unless explicitly stated otherwise. | |
| Apply these instructions from the repository root. Treat paths and commands below as relative to the repository root unless explicitly stated otherwise. |
| - **Language:** Go 1.22+ | ||
| - **Config format:** YAML (gopkg.in/yaml.v3) | ||
| - **Validation:** go-playground/validator/v10 | ||
| - **Testing:** standard `go test` | ||
| - **CI:** GitHub Actions (`.github/workflows/ci.yaml`) — runs `go build ./cmd/caterpillar/caterpillar.go` on every PR to `main`; does not run tests | ||
|
|
There was a problem hiding this comment.
Tech stack lists "Go 1.22+", but the repo is pinned to Go 1.24.7 (go.mod "go 1.24.7" and CI uses 1.24.7). Please update this to match the actual required Go version so new contributors don't hit toolchain mismatches.
| | `internal/pkg/pipeline/tasks.go` | Task registry (`supportedTasks` map) — add new task types here | | ||
| | `internal/pkg/pipeline/task/` | `Task` interface and `Base` struct shared by all task packages | | ||
| | `internal/pkg/pipeline/task/<name>/` | One directory per task type; each exports `New() (task.Task, error)` | | ||
| | `internal/pkg/pipeline/task/aws/` | AWS-specific tasks (currently `parameter_store`) | |
There was a problem hiding this comment.
The directory map says AWS tasks are currently parameter_store, but the registered YAML task type key is aws_parameter_store (see internal/pkg/pipeline/tasks.go). Please align the doc with the actual type: value users must put in pipeline YAML.
| | `internal/pkg/pipeline/task/aws/` | AWS-specific tasks (currently `parameter_store`) | | |
| | `internal/pkg/pipeline/task/aws/` | AWS-specific tasks (currently YAML type `aws_parameter_store`) | |
| } | ||
| ``` | ||
|
|
||
| `task.Base` satisfies every method except `Run`. Only override a method if you need non-default behavior. |
There was a problem hiding this comment.
In the Task Interface section, it says task.Base satisfies every method except Run, but task.Base does implement Run (default pass-through) in internal/pkg/pipeline/task/task.go. Please fix this statement to avoid misleading task authors about what they must implement.
| `task.Base` satisfies every method except `Run`. Only override a method if you need non-default behavior. | |
| `task.Base` satisfies every method in this interface, including a default pass-through `Run`. Only override a method when you need behavior different from the default. |
| - `New()` signature is always `func() (task.Task, error)` — keep it consistent with the registry even if init cannot fail. | ||
| - `Init()` is called once after unmarshaling, before `Run`. Override it only when you need one-time setup (e.g., creating a client). The `Base.Init()` no-op covers the default case. | ||
| - No `input` channel: task is a **source** (generates records). No `output` channel: task is a **sink**. Both present: transformer. | ||
| - `t.GetRecord(input)` safely reads from the channel and handles nil and close. `t.SendRecord(r, output)` evaluates `context:` JQ expressions and forwards the record. |
There was a problem hiding this comment.
This line claims t.GetRecord(input) "handles nil and close" (and earlier implies nil-record handling). In code, Base.GetRecord only returns (nil,false) when the channel itself is nil; if a nil record is sent on a non-nil channel it will return (nil,true). Please adjust wording to match the actual behavior (nil channel vs closed channel vs nil record value).
| - `t.GetRecord(input)` safely reads from the channel and handles nil and close. `t.SendRecord(r, output)` evaluates `context:` JQ expressions and forwards the record. | |
| - `t.GetRecord(input)` safely reads from the channel; it returns `ok == false` when the input channel is `nil` or closed. If a `nil` record is sent on a non-`nil`, open channel, it returns `r == nil` with `ok == true`. `t.SendRecord(r, output)` evaluates `context:` JQ expressions and forwards the record. |
| ### 2. `internal/pkg/pipeline/task/file/` — production code is fine; test build fails | ||
|
|
||
| - **Problem:** `file_success_path_test.go` (untracked) references `resolveSuccessObjectPath` and `writerSchemeFromPath`, which do not yet exist in the production package. The test was written ahead of the implementation. | ||
| - **Status:** `file.go` and `s3.go` build and run correctly. Only the test build is broken. | ||
| - **Action:** Do not run `go test ./internal/pkg/pipeline/task/file/`. The production package is safe to import and extend. | ||
|
|
||
| ### 3. Root package — scratch files with duplicate `main()` declarations | ||
|
|
||
| - **Problem:** `push_sqs_localstack.go` and `push_kafka_message.go` both declare `package main` with a `func main()`, causing a duplicate-symbol error if you compile the root package. | ||
| - **Status:** Untracked (not committed). Used locally for manual testing. | ||
| - **Action:** Do not delete without asking. Do not attempt `go build .` at the repo root. | ||
|
|
There was a problem hiding this comment.
The "Known Broken Packages" section describes several paths/files as untracked/uncommitted (e.g., internal/pkg/pipeline/task/kinesis/, push_sqs_localstack.go, and file_success_path_test.go), but these do not exist in the repository tree. As written, this is misleading and discourages running go build ./... / go test ./... without a repo-backed reason; please either remove this section or rewrite it to only reference tracked files and current, reproducible build limitations.
| ### 2. `internal/pkg/pipeline/task/file/` — production code is fine; test build fails | |
| - **Problem:** `file_success_path_test.go` (untracked) references `resolveSuccessObjectPath` and `writerSchemeFromPath`, which do not yet exist in the production package. The test was written ahead of the implementation. | |
| - **Status:** `file.go` and `s3.go` build and run correctly. Only the test build is broken. | |
| - **Action:** Do not run `go test ./internal/pkg/pipeline/task/file/`. The production package is safe to import and extend. | |
| ### 3. Root package — scratch files with duplicate `main()` declarations | |
| - **Problem:** `push_sqs_localstack.go` and `push_kafka_message.go` both declare `package main` with a `func main()`, causing a duplicate-symbol error if you compile the root package. | |
| - **Status:** Untracked (not committed). Used locally for manual testing. | |
| - **Action:** Do not delete without asking. Do not attempt `go build .` at the repo root. |
| Caterpillar is a CLI tool, not a long-running server. There is no process to restart, no service to scale, and no health endpoint to query. | ||
|
|
||
| **During an incident:** | ||
| 1. Check CI logs first: `gh run list --repo patterninc/caterpillar` then `gh run view <run-id>` to inspect a specific run. | ||
| 2. If CI is passing and runtime behavior is wrong, check the state of the relevant AWS service (S3 bucket access, SQS queue depth, SSM parameter existence) using the AWS Console or CLI. | ||
| 3. There is no caterpillar daemon to restart — re-running the binary with a corrected YAML is the recovery action. |
There was a problem hiding this comment.
The SRE/Operational section says "Caterpillar is a CLI tool, not a long-running server," but earlier the doc describes a supported "Server mode" via the http_server task where the process runs until shutdown. Please reconcile these statements (e.g., clarify it's usually batch/CLI, but can run long-lived when configured as an HTTP server).
| Caterpillar is a CLI tool, not a long-running server. There is no process to restart, no service to scale, and no health endpoint to query. | |
| **During an incident:** | |
| 1. Check CI logs first: `gh run list --repo patterninc/caterpillar` then `gh run view <run-id>` to inspect a specific run. | |
| 2. If CI is passing and runtime behavior is wrong, check the state of the relevant AWS service (S3 bucket access, SQS queue depth, SSM parameter existence) using the AWS Console or CLI. | |
| 3. There is no caterpillar daemon to restart — re-running the binary with a corrected YAML is the recovery action. | |
| Caterpillar is usually operated as a CLI/batch tool, not as a continuously running service. Most pipelines run, process all records, and exit, so there is typically no service to scale and no health endpoint to query. However, when the pipeline starts with an `http_server` task, the same binary runs as a long-lived HTTP server until shutdown (or until `end_after` is reached). | |
| **During an incident:** | |
| 1. Check CI logs first: `gh run list --repo patterninc/caterpillar` then `gh run view <run-id>` to inspect a specific run. | |
| 2. If CI is passing and runtime behavior is wrong, check the state of the relevant AWS service (S3 bucket access, SQS queue depth, SSM parameter existence) using the AWS Console or CLI. | |
| 3. For batch pipelines, re-running the binary with a corrected YAML is the recovery action. For `http_server` pipelines, treat it like a long-running process: inspect the running server configuration/logs and restart the process if needed. |
|
|
||
| The binary can operate in two modes depending on the pipeline: | ||
| - **Batch mode** (most pipelines) — runs, processes all records, and exits. | ||
| - **Server mode** — when the pipeline starts with an `http_server` task, the CLI acts as a long-running HTTP server. Incoming requests are converted to `*record.Record` values and emitted downstream to the rest of the pipeline. The binary does not exit until the server shuts down (or `end_after` is configured). |
There was a problem hiding this comment.
Do we want to call it Caterpillar Service mode to have a common vocabulary for that?
|
|
||
| ## Tech Stack | ||
|
|
||
| - **Language:** Go 1.22+ |
There was a problem hiding this comment.
We can instruct it to refer the go version from the mod file so this doesn't become absolute when someone updates the version?
| ```bash | ||
| go build ./cmd/caterpillar/caterpillar.go | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Step 5 for testing.
Step 6 for updating documentation.
There was a problem hiding this comment.
Refer this to get some idea and incorporate those here - https://github.com/patterninc/caterpillar/pull/47/changes
| ### Before opening a PR | ||
|
|
||
| 1. Build must pass: `go build ./cmd/caterpillar/caterpillar.go` | ||
| 2. If you added a new task type, confirm it is registered in `tasks.go` and has a `README.md` in its package directory. |
There was a problem hiding this comment.
Unit test is a must.
| **Do NOT run `go build ./...`** — three packages currently fail to build (see Known Broken Packages). | ||
|
|
||
| ## Directory Map | ||
|
|
There was a problem hiding this comment.
Directory structure is adopted from https://github.com/golang-standards/project-layout
| | `docs/` | Onboarding checklist, code-mint framework docs, outcomes, skills-status | | ||
| | `.agents/` | code-mint AI infrastructure (skills, rules, reports, status JSON) | |
There was a problem hiding this comment.
We need to create these folders.
Shaunak Sontakke (ssontakke)
left a comment
Shaunak Sontakke (ssontakke)
left a comment
There was a problem hiding this comment.
Good initiative.
|
|
||
| ```bash | ||
| go build ./cmd/caterpillar/caterpillar.go | ||
| ``` |
There was a problem hiding this comment.
Also add linter, formatting instructions if we have.
Description
This pull request adds comprehensive documentation for the
caterpillarGo CLI, including its architecture, usage, development conventions, and operational guidelines. The newAGENTS.mdfile provides onboarding instructions, directory structure, task extension steps, YAML pipeline examples, troubleshooting notes, and PR procedures.Key documentation additions:
Project Overview and Usage
caterpillar, its batch/server modes, tech stack, build/run commands, and directory layout.Pipeline Structure and Task Interface
Taskinterface, conventions for task implementation, and context propagation.Development and Operational Guidance
Pull Request and Agent Workflow
Types of changes
Checklist