Industry: Edge Computing, Industrial Connectivity & Data Integration
Region: Germany
Cloud Services Used: AWS (S3, EC2, EKS, VPN)
Headquartered in Germany, Cybus Connectware serves as a Factory Data Hub, connecting IoT and IT systems company-wide. Cybus empowers organizations to effortlessly collect, standardize, process, and distribute data, enabling diverse Industrial IoT use cases with a seamless global data flow across all facilities. To test their applications on various platforms with different scenarios and versions, the QA Team of Cybus requires rapid, highly configurable environments.
To address these specific needs, Build NYC Tech developed a comprehensive solution for on-demand infrastructure deployment and configuration. This platform integrates infrastructure provisioning using Terraform to ensure consistent and scalable resource allocation. Post-deployment, environments are dynamically configured using Ansible or Helm, running applications seamlessly on Docker (EC2 Deployments) or Pods (Kubernetes Deployments) to make them ready for immediate QA testing.
Problem: The DevOps team faced major bottlenecks deploying Connectware and its underlying agents across different environments for the QA team, as the number of agents could range anywhere from 0 to 999. Manually provisioning infrastructure for each release of Connectware and corresponding agent versions proved to be repetitive, time-consuming, and highly expensive.
Solution: We utilized Git as the definitive source for Terraform modules, ensuring that the modules remained modular, extendable, and easily updatable without needing to redeploy the entire core architecture. By leveraging Jinja templates, we dynamically generated custom Terraform scripts that inject these defined modules on the fly, enabling efficient, automated, and scalable infrastructure provisioning across wildly varying agent counts.
Problem: Configuring on-demand, large-scale infrastructure dynamically becomes complex when managing numerous virtual machines or Kubernetes clusters simultaneously. The manual configuration of these environments at scale was completely impractical.
Solution: We addressed this by using Jinja templates to programmatically generate dynamic Ansible scripts that automate the entire configuration lifecycle. These scripts seamlessly integrate and populate Docker Compose files or Helm values, enabling a swift, automated setup across large-scale environments while eliminating human error.
Problem: Terraform modules, Helm values, and Docker Compose files are subject to constant iterations by development teams. Without an efficient background update mechanism, the entire provisioning engine would need to be redeployed repeatedly to incorporate these blueprint changes, creating massive operational overhead.
Solution: The solution clones the relevant configuration repositories during the initial environment deployment. For every subsequent on-demand testing request, the engine pulls the latest Git updates automatically. This approach ensures that any architectural changes or module additions are seamlessly integrated into new test builds without requiring a full platform redeployment.
Problem: Because these environments are strictly for QA testing, identifying bugs, errors, and race conditions is a constant requirement. Manually checking logs and performance metrics across multiple virtual machines or isolated Kubernetes pods was physically unfeasible, creating an urgent need for a centralized monitoring layer.
Solution: Our solution configures each virtual machine’s Docker engine to capture and funnel logs using Fluentd, while monitoring container resource stats via cAdvisor. These logs and metrics are shipped to a centralized Grafana instance powered by Grafana Loki (for log aggregation) and Prometheus (for metric storage), allowing the QA team to track performance and troubleshoot system bugs in real time from a single pane of glass.
Problem: Each environment deployment task is a long-running, resource-intensive job. Concurrently running heavy Terraform and Ansible tasks adds massive compute load. For robust load testing, the platform needed to comfortably support a minimum of 10 complex deployment requests simultaneously.
Solution: We engineered a Python-based background task architecture powered by Gunicorn workers to handle incoming API requests efficiently. The application instance is hosted on a single AWS EC2 instance (type c7i.xlarge with 4 vCPUs). Through this optimized setup, we successfully handled 10 concurrent deployment requests of differing criteria while consuming just 32% of total CPU capacity.
Problem: Long-running, on-demand testing environments can occasionally fail due to misconfigurations, system crashes, or network drops. Tracking the active usage of every single tester manually is impossible, leading to forgotten or “zombie” deployments that continuously bleed cloud infrastructure budgets without adding business value.
Solution: We implemented an automated cleaner cron-job that executes every day after office hours to detect inactive or corrupted deployments. The script attempts to programmatically destroy failed deployments (with a 2-retry fallback policy) and instantly broadcasts the cleanup logs and financial savings to a dedicated team Slack channel.
Problem: During simultaneous deployment requests, parallel system processes attempt to pull blueprint updates from Git concurrently. This frequently caused conflicts due to the .git/index.lock file being engaged by whichever process arrived first, causing all other parallel deployments to fail and requiring manual lock-file deletion.
Solution: To manage this, we utilized Linux flock to implement a global file-locking mechanism for the Git pull operation. This ensures that only one deployment process can safely execute a pull operation at any exact millisecond. If a pull operation fails or is interrupted, the lock is automatically released, the stale lock file is programmatically purged if present, and subsequent deployment queues proceed smoothly without friction.
Problem: Provisioning and configuration logs from Terraform and Ansible are vital for QA engineers to verify if an environment was built correctly. Because log volumes fluctuate heavily based on the complexity of the request, multiple users needed a way to stream these compilation logs simultaneously without locking the target log file.
Solution: We used Python’s subprocess module to isolate the execution of Ansible and Terraform commands, capturing all stdout and stderr streams via a custom collector function that writes to a dedicated log file per deployment. We then built a dedicated streaming API that enables a single-write, multiple-read setup, letting users watch their builds live. Concurrently, a permanent copy of the log is archived in an AWS S3 bucket for auditing and post-mortem inquiries.
Problem: Because Cybus handles critical Industrial IoT and factory data routing, the infrastructure platform required rigid, multi-layered security measures far beyond basic user authentication to prevent unauthorized network access and protect system integrity.
Solution: We designed a secure, multi-tier AWS network architecture. AWS Cognito handles robust user authentication and authorization. The central management machine executing the heavy automation scripts is entirely isolated within a secure private subnet. User and QA engineer access to the underlying platform APIs is exclusively permitted through an AWS Client VPN gateway hosted in a public subnet. All spun-up testing deployments are isolated inside separate private subnets, ensuring critical infrastructure remains invisible to the public internet while remaining accessible to authorized QA operators over the VPN.