Manage flows across multiple environments
To help you manage resources efficiently in qibb and establish a robust flow development process, we've outlined a best practices guide for creating, managing, and deploying your flow in multiple environments, such as Development (DEV) and Production (PROD) environments. This guide covers the use of Spaces and Git-based version control to ensure a structured workflow that facilitates review, approval, and deployment.
1. Set Up Spaces for each Environment
Spaces enable you and your team to efficiently organize resources. Technically, a space is a virtual environment built on underlying infrastructure where you can deploy multiple apps and manage them collaboratively with your team. Each space includes:
Access management, allowing owners to add other members for collaboration within the space.
Secret management, allowing members to securely store configurations and secrets, which can be accessed and used by apps within the space.
Monitoring capabilities, allowing members to inspect logs and runs for efficient debugging of contained apps.
Typically, spaces are used to resemble different projects or environments managed by different teams. For each client or project, we recommend creating dedicated Spaces for each environment. Below can be found an example structure, using a simple name scheme of <Customer> - <Project> - <Environment>, but it’s completely up to you how to name these Spaces.
Example Spaces Structure
Space Name | Purpose | Team Members | Secrets | |
|---|---|---|---|---|
| 1 | “Customer A - Project 1 - DEV” | This space allows developers to build, test, and debug new flows flexibly, enabling experimentation without affecting other environments. | Typically in this Space, you give your team the freedom to work flexibly while maintaining full control and visibility over their resources. For example:
| A set of secrets pointing to Development systems which the qibb flows are consuming. Note that the structure of these secrets should match that of other related environments, as the flows will reference them by variable names. |
| 2 | “Customer A - Project 1 - STAGING” | This space replicates the PROD environment as closely as possible, allowing the team to conduct final tests to ensure everything works as expected before release. It’s a controlled space for quality assurance (QA) and user acceptance testing (UAT), catching any potential issues before they go live. | Here, the same team as above may have access to this Space. In addition you may have additional users for reviewing, testing and approvals. For example:
| A set of secrets pointing to Staging systems which the qibb flows are consuming. Note that the structure of these secrets should match that of other related environments, as the flows will reference them by variable names. |
| 3 | “Customer A - Project 1 - PROD” | This is the live environment where the flows are accessed by end users. It must be stable, secure, and optimized since any issues here can impact real users directly. Only thoroughly tested flows reaches this environment. | Typically, this space is managed and accessed only by the team responsible for service availability and support. They carefully monitor and administer changes to the workflows and secrets. For example:
| A set of secrets pointing to Production systems which the qibb flows are consuming. Note that the structure of these secrets should match that of other related environments, as the flows will reference them by variable names. |
This setup allows your organization to administrate multiple environments as spaces, while having separate access control and team members, ensuring the development iterations are properly tested before they reach production.
2. Use version control to promote flow changes across different environments
Once you have setup your spaces to reflect each individual environment, you can proceed with deploying a pair of apps across each space. For each flow, you will need one app deployed per space.
Tip: Customize the Name, Description, Tags, Color, and Links of your deployed apps to improve their visibility and document their relationships to your environment and other resources.
Example Apps Structure
App Name | Deployed into Space | App Tags | App Links | |
|---|---|---|---|---|
| 1 | 🔵 “Ingest Flow DEV” | “Customer A - Project 1 - DEV” | DEV |
|
| 2 | 🔴 “Ingest Flow PROD” | “Customer A - Project 1 - PROD” | PROD |
|
Git Workflow
The given flows will be connected to a shared Git Remote Repository by following this guide for each flow app. The example Git workflow below uses separate Git branches to reflect environments. While using branches is recommended, you may also prefer a simpler Git workflow by utilizing just one main branch for all environments and have no merge process at all.
This process can be easily extended or customized. For instance, if you plan to do review and testing of changes before making it available in production, you could introduce an extra space and flow for Staging.
The process consists of the following steps:
Developers work on the Ingest Flow DEV and commit changes either automatically or manually into the local Git of the app.
This step is executed within the Flow Editor.Developers push changes from Ingest Flow DEV to the DEV branch of the Git Remote Repository.
This step is executed within the Flow Editor.Afterward, a Reviewer will review and merge the changes into the appropriate target branch, here the PROD branch.
This step is done on the remote repository using an external Git tool.Finally, the changes will be pulled from PROD branch into Ingest Flow PROD.
This step is executed within the Flow Editor.
Tip: Automatic and Manual mode for making commits
In the Flow Editor, Users can switch between Automatic or Manual Mode for making commits in the Git Config settings. The Manual Mode allows developers to create custom commit messages, while also giving them the responsibility to decide when changes should be checked in.
Additional Best Practices
Separate environment parameters from your flow
Your flow should not contain any hardcoded credentials or configuration parameters, instead it should reference it to variable names defined in the Secret Manager.
Each space should have the same set of variables defined.
Manage dependencies for your flow
You can pin the dependencies used in your flow, which helps you to ensure that the correct versions of nodes and modules are installed across your deployed apps. The definition of the dependencies will be stored in a package.json inside of the Git Repository.
To manage these dependencies, you can choose to manage them in them in the Flow Editor by navigating to Project Settings>Dependencies. Alternatively, you can choose to edit the package.json with an external tool on the Git Remote Repository.
For more information, see Managing projects and Project Dependencies.
Monitor your flows
While promoting changes across multiple environments, reviewing logs and secrets can provide valuable insights to ensure flows are functioning as expected after updates and assist in troubleshooting if issues arise.
For more information, see Log Browser and Run History.
By following these best practices, you can create a structured, scalable process for managing flows in qibb, ensuring that all changes are reviewed and approved before deployment to client environments.