The Best Branching Strategy for Multi-Tenant Detections

You’ve spent months building a robust detection engineering program across multiple SIEM platforms for your clients. But now, your team is drowning in a sea of unmanaged branches, conflicting rules, and deployment failures. Every time you try to update a detection for one client, you break something for three others. What started as an organized approach to Detection as Code has devolved into chaos.

The Multi-Tenant Detection Dilemma

Managing detection rules across numerous SIEM instances without a structured process leads to instability, duplicated effort, and a high risk of errors. Security teams and MSSPs struggle with questions like:

• Should we use a branch per SIEM or multiple repositories?
• How do we handle generic rules versus tenant-specific customizations?
• What’s the best way to implement a pull request workflow that ensures quality?
• How can we maintain sanity in our Software Development Lifecycle (SDLC)?

If these questions sound familiar, you’re not alone. As one security engineer put it: “Managing all this with branches is really overwhelming.” But there’s a solution that brings order to this chaos.

Why Your Branching Strategy Matters

A branching strategy is a set of guidelines for how code (in this case, detection rules) is organized, written, merged, and deployed using a version control system like Git. Without one, you’re essentially flying blind.

The GitFlow-Inspired Approach for Detection Rules

After evaluating several models, I recommend adapting the popular GitFlow branching strategy for multi-tenant detection engineering. This approach provides a robust framework for managing releases and hotfixes, which is crucial for security operations.

Core Branches

• main (or master): Your production truth. It contains only stable, tested, and deployed detection rules. This branch always reflects the latest production-ready state.
• develop: The primary integration branch where all completed features and fixes are merged before being bundled into a release. This branch contains the latest development changes.

Supporting Branch Types

• Feature Branches (feature/*)
  • Purpose: Develop new detection rules or modify existing ones
  • Workflow: Branch off from develop, and must be merged back into develop
  • Naming: feature/JIRA-123-new-phishing-rule or feature/tenant-a-specific-log4j-variant
• Release Branches (release/*)
  • Purpose: Prepare for a new production deployment with final testing
  • Workflow: Branch off from develop. Once ready, merge into both main and develop
  • Naming: release/v1.2.0
• Hotfix Branches (hotfix/*)
  • Purpose: Address urgent issues in production (e.g., false positives)
  • Workflow: Branch off from main at the specific problematic release tag. Once fixed, merge into both main and develop
  • Naming: hotfix/v1.1.1-fix-fp-storm

Repository Structure: The Great Debate

One of the most contentious questions in multi-tenant detection engineering is whether to use a single monolith repository or separate repositories for each SIEM platform.

Recommendation: Repo per SIEM

For multi-tenant environments, especially across different SIEM platforms (Splunk, Microsoft Sentinel, QRadar), the recommended approach is one repository per SIEM platform. Here’s why:

1. Logical Separation: As one security engineer noted, this “will help keep things logically separated.” It prevents confusion between rules written in different query languages (SPL, KQL, AQL).
2. Cleaner CI/CD Pipelines: It allows you to “keep your pipelines cleaner when you just have to bring the one repo in during CI/CD.” Each repo can have its own tailored deployment logic, secrets, and variables.
3. Easier Deprecation: If a SIEM is decommissioned, you can simply “archive the repo instead of having a git tree with a bunch of dead branches that need to be pruned. Less maintenance.”
4. Fine-Grained RBAC: You can implement role-based access control per repository. Maybe junior engineers can only contribute to the Splunk repo, while senior engineers have access to all platforms.

While you can use submodules to reference common code across repositories, be cautious – they add complexity that may outweigh their benefits in many scenarios.

Managing Generic vs. Tenant-Specific Content

Within each SIEM-specific repository, you’ll need a strategy for handling both generic rules (applicable to all tenants) and tenant-specific customizations:

Generic Rules

These form the core detection set for all tenants on that platform:

/
├── detections/
│   ├── credential-access/
│   │   ├── mimikatz.yml
│   │   └── pass-the-hash.yml
│   └── initial-access/
│       └── phishing.yml

Tenant-Specific Rules

These address unique requirements or environments:

/
├── tenants/
│   ├── client-a/
│   │   └── custom-detections/
│   │       └── legacy-app-alert.yml
│   └── client-b/
│       └── custom-detections/
│           └── azure-specific-alert.yml

The CI/CD pipeline can be configured to deploy rules from specific folders or tagged with specific metadata to the correct tenant instance. This approach allows you to maintain a single codebase while supporting customization.

The Pull Request: Your Gateway to Quality

A sane, auditable SDLC prevents broken rules from reaching production. Pull Requests (PRs) are non-negotiable in this process.

Best Practices for a Robust PR Workflow:

1. Mandatory Peer Review: Every code change should be reviewed and approved by at least one other person before merging. For critical branches like main and develop, enforce this with branch protection rules.
2. Automated Testing (CI): The PR should automatically trigger a CI pipeline that runs unit tests and automated testing on the detection rule. For Splunk users, tools like contentctl can validate rules against sample data.
3. Clear Descriptions: Use PR templates to ensure every submission includes:
   • What problem the detection addresses
   • How to test it
   • Expected behavior in production
   • Any performance considerations
4. Actionable Feedback: Use nested comments to have organized discussions around specific lines of code. Each comment should be resolved before the PR can be merged.

Putting It All Together: A Sample Workflow

Let’s walk through an example to see how this all works in practice:

Scenario: A new detection rule for a specific credential dumping technique is needed for all tenants using Splunk.

Step-by-Step:

1. Create an Issue: A ticket is created in Jira: SEC-451: Detect Mimikatz via LSASS Access.
2. Create a Feature Branch: In the splunk-detections repository, an engineer runs: git checkout develop && git pull && git checkout -b feature/SEC-451-mimikatz-lsass
3. Develop the Rule: The engineer writes the SPL query and saves it in a structured folder (e.g., /detections/credential-access/mimikatz.yml).
4. Unit Test Locally: Before committing, the engineer runs local tests with sample logs to ensure the logic is sound and performance is acceptable.
5. Push and Open a PR: The branch is pushed, and a Pull Request is opened to merge feature/SEC-451-mimikatz-lsass into develop.
6. CI Pipeline Triggers: The PR automatically kicks off:
   • Syntax validation of the SPL query
   • Automated tests against sample data
   • Performance benchmarks to ensure the rule won’t impact SIEM performance
7. Peer Review: Two team members review the logic, check for performance impacts, and suggest improvements using comments.
8. Merge: Once tests pass and approvals are granted, the branch is merged into develop.
9. Release Process: When the team is ready for the next release, a release/v2.5.0 branch is created from develop. After final QA in a staging environment, it’s merged into main and tagged, triggering a production deployment to all Splunk tenants. The release/v2.5.0 branch is also merged back into develop.
10. Hotfix if Needed: If the rule generates false positives in production, a hotfix branch is created from main, fixes are applied, tested, and then merged back to both main and develop.

Advanced Considerations

As your detection engineering practice matures, consider these advanced techniques:

Branch per SIEM Instance

For complex environments with highly customized SIEM instances, you might consider a branch per tenant approach within your SIEM-specific repositories. This allows for tenant-specific configurations while maintaining a shared codebase.

Secrets Management

Never store credentials or API keys in your repositories. Use environment variables or dedicated secrets management tools integrated with your CI/CD pipelines to securely provide endpoints and authentication details during deployment.

Feature Flags

Consider implementing feature flags to gradually roll out new or updated detections to select tenants before full deployment, allowing for controlled testing in production environments.

Conclusion: From Chaos to Control with Detection as Code

Adopting a structured branching strategy like GitFlow isn’t about adding bureaucracy; it’s about creating a predictable, scalable, and resilient process for “Detection as Code.”

By implementing:

• A Repo per SIEM to maintain logical separation
• A GitFlow-like model to manage the lifecycle of detection rules
• A strict Pull Request process with mandatory peer reviews and automated testing

You’ll transform your multi-tenant detection engineering from chaotic to controlled, enabling your security team to move faster with greater confidence.

Remember that the goal isn’t perfect implementation of a textbook branching strategy – it’s finding the right balance of structure and flexibility that works for your team and detection engineering requirements. Start with these principles, adapt as needed, and continuously improve your process as you learn.

Frequently Asked Questions

What is the best branching strategy for detection engineering?

The best branching strategy for detection engineering is a model adapted from GitFlow. This approach provides a structured framework using a main branch for stable, production-ready rules and a develop branch for integrating new features, supported by feature, release, and hotfix branches for organized development and urgent fixes.

Why should I use a separate repository for each SIEM platform?

You should use a separate repository for each SIEM platform to maintain logical separation and simplify your workflows. This structure prevents confusion between different query languages (e.g., Splunk’s SPL vs. Sentinel’s KQL), allows for cleaner and more specific CI/CD pipelines, makes it easier to deprecate a SIEM platform by simply archiving its repo, and enables more granular role-based access control (RBAC).

How do you manage rules for different clients in a single repository?

To manage rules for different clients (or tenants), you can create a dedicated directory structure within your repository, such as a top-level tenants/ folder with subfolders for each client (e.g., tenants/client-a/). Your CI/CD pipeline can then be configured to deploy generic rules to all tenants and specific rules from these folders only to the corresponding tenant’s environment.

What is the difference between a release branch and a hotfix branch?

A release branch is used for planned deployments, while a hotfix branch is for urgent, unplanned fixes. A release branch is created from the develop branch to prepare a new set of features for production. In contrast, a hotfix branch is created directly from the main branch to quickly address a critical issue (like a major false positive) in the current production code.

How does a Pull Request (PR) improve detection rule quality?

A Pull Request (PR) improves quality by acting as a gateway for changes, ensuring every rule is reviewed and tested before reaching production. A robust PR process includes mandatory peer reviews to catch logical errors, automated CI testing to validate syntax and performance, and standardized templates to ensure every change is well-documented and understood.

What if the GitFlow model is too complex for my team?

If GitFlow seems too complex, you can start with a simplified version and adapt it as your team grows. The key principle is to separate development work from your stable, production code. You could begin with just a main branch and short-lived feature branches, introducing develop and release branches later when your release cadence becomes more complex. The goal is to find a process that adds structure without unnecessary bureaucracy.

How should I handle sensitive information like API keys in my detection code repositories?

You should never store sensitive information like API keys, passwords, or credentials directly in your Git repositories. Instead, use a dedicated secrets management tool (like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault) and integrate it with your CI/CD pipeline. The pipeline can then securely inject these secrets as environment variables during the deployment process.

Have you implemented a branching strategy for detection engineering? What challenges have you faced? Share your experiences in the comments below.