We’re working with a Fortune-100 company to apply simulation testing with Clojure and Datomic. The dev team heard about it and thought it might be a better way to test its unusually complex e-commerce system. It was.
Simulation testing filled a critical QA gap and compressed weeks of testing into a matter of hours. Using simulation testing with Clojure and Datomic, our team and the company's dev team worked together to:
We developed a case study to share the company's story. Here it is.
A Fortune-100 company uses Clojure and Datomic in its innovation lab to create real-world conditions and find bugs internal developer teams can’t find, improving the customer experience for one of the most recognized brands in the world.
The team responsible for the enterprise’s e-commerce system came to Cognitect with a problem. For nearly a year, its developers had been looking for a better way to test an unusually complex system without slowing release schedules. They were under pressure to get new features to market faster without exposing millions of global users to new bugs.
As part of its consulting services, Cognitect worked with the client’s developers to set up simulation testing, creating intelligent user-agents to navigate system functions. The agents take actions only real users could take. They also behave like real users: unpredictably and as a crowd. Compared to other testing methods, simulation testing provides a more accurate view of how the system would perform in the real world.
The team chose Clojure for its simplicity. It brings powerful data-processing capabilities and uses far less code than other programming languages. Datomic’s flexible schema and built-in history add testing speed and quick access to key information to support innovation. Together, they create an intelligence layer that rides on top of the company’s existing database services.
Since Datomic data is immutable – never overwritten or erased – user-agents grow smarter over time, building on their actions and keeping a clear record of everything they do, each step of the way. The same agents can be used to test old builds. Once a testing script is run and the results are captured, the team can validate new information against the old capture without running the script again.
This more predictive model gives the team the best of both worlds: the ability to test intelligent actions real users would take and do it quickly in a repeatable, scalable way. As a result, they compressed weeks of testing into a matter of hours and found bugs their internal developer teams hadn’t found.
Cognitect built an interactive dashboard to visualize test results. Managers use it to see at a glance if a build was good or if it introduced a new problem. Developers can drill down quickly into each failure to identify the root cause, fix it and track the success or failure of subsequent tests.
The dashboard also revealed unexpected global issues across the system, such as failures caused by dependencies on other enterprise components or conflicts within code written by multiple developers working on related features simultaneously. Before simulation testing, these workflow patterns went undetected, silently adding friction and delays to products, teams and innovation.
Simulation testing filled a critical gap in the company’s quality assurance (QA) process, increased the speed of product iteration and delivery, and improved confidence in the quality of each release.
Today, simulation testing runs around the clock, across multiple builds and QA environments. The development team can see the results of each new feature, change or repair on a daily basis. Each build is automatically subjected to real-world conditions, and live test results flag issues that may affect users if it were released. With a quick look at the dashboard and review of any failures, the team can confidently decide if today is the day to release a new version to millions of eager users.
For more information about how simulation testing can empower your development team to innovate and bring new products to market faster and with higher quality, contact Cognitect at firstname.lastname@example.org or visit our consulting page.
Michael T. Nygard is Vice President at Cognitect, whose products and services allow companies to build applications and move data securely to the cloud with ease, speed and few resources. He also served as technical director for Verizon Business, leading its “Innovation Incubator” team in its heyday. He has authored and contributed to books about applying technology to reduce effort, boost productivity and optimize financial outcomes. He is a regular speaker at QCon, GOTO and No Fluff Just Stuff conferences.