The world of commodity trading is unpredictable and complex, yet at the same time slow and ridden with manual processes and legacy IT systems. With spiking volatility, increased uncertainty, and an urgent need to improve supply chain security, the commodity industry is set for a massive transformation of its digital systems. Industry players are therefore rushing to modernize their IT infrastructure - and it starts with how trading companies think about software development.
The global arms race of market intelligence is heating up
Digitalization is everywhere, and trading is one of the most dynamic segments of the commodity industry. In fact, trading desks today face an epic technology arms race that foreshadows what the rest of the commodity industry has in store as well. As traditional competitive moats erode, trading houses are investing billions of dollars in frontier analytics to protect their trading edge in highly competitive markets.
This investment frenzy has spawned a whole ecosystem of market intelligence companies fighting to build the latest solutions using alternative data sources (“altdata”) and of course artificial intelligence. These tools provide traders with the most recent insights, but they are still third-party software tools that quickly become commoditized and add little new value. However, not having the tools is still not an option. There is a “forced adoption” of these tools as quickly opting out puts you at a disadvantage, but the ROI on these investments also spiral downwards as all players purchase the same (or very similar) tools.
But market intelligence is just not enough
The best traders, however, recognize that it is not just market intelligence that provides an edge; it is about the ability to execute change when the markets change. Sophisticated traders have employed small armies of ”quants” (quantitative analysts) in recent years to develop increasingly advanced trading algorithms, but competitors quickly catch up once opportunities are identified. The race therefore changed from having the best insight, to being able to execute on those insights the fastest.
However, this race for speed has unveiled yet another challenge: advanced trading algorithms are only as good as the data they train on, and the data sources in trading keep changing. Traders therefore need a highly complex software architecture, specialized software services, and extensive data governance processes in order to manage these data source dynamics.
Traders require modular software ecosystems to move faster than they do today
Many companies have in the past built their software solutions for data management in-house, or hired battalions of consultants to build (and maintain) an ecosystem of custom data management solutions. However, as the number of data sources and solutions grows, so does the army of people it takes to manage them. When margins are wide, such linear scaling may still work. Actually, this strategy was the only practical option for many companies when technology was still young and immature. However, when looking at what is rapid technology developments in in Cloud, DevOps, Cybersecurity, and now AI, many companies ask themselves:
Should our company build this software infrastructure from scratch, and then maintain it all in-house? Can we really win by delivering every layer of our IT architecture ourselves?
Go Deeper: 7 Myths of DIY Platforms
Trading companies are learning from the modern software industry
Look at any modern software company; they don’t try to build software that covers everything they need, from the infrastructure to the user interface. Instead, the best modern software companies build composable IT architectures consisting of a mosaic of interoperable software modules.
Whether it is the use of popular programming languages (e.g., Python, Java), low-cost and scalable Cloud infrastructure (e.g., AWS, Azure, GCP), open-source tools (e.g., Kubernetes, scikit), and now AI tools with readily accessible API’s (e.g., OpenAI), all modern software consists of multiple layers of other software components. And that’s the way modern trading companies also need to think.
For traders to win, their competitive edge in the market will come from the ability to develop, deploy, and roll out new solutions faster and faster as the “shelf life” of these solutions shrinks. Without an agile and modular software architecture however, constantly ramping up production of new tools, adapting to market changes, and executing on new insights becomes a near impossible task.
The best companies will be those able to orchestrate dynamic software ecosystems
Developing an agile and dynamic software architecture for commodity trading may indeed seem a Herculean undertaking. The architecture must handle massive data sets with high quality standards, and simultaneously scale to meet the ever-changing demands of the business. The solution space must also integrate with multiple internal and external systems and comply with complex global regulations. However, an increasing number of traders are adopting a more modular approach to their IT architecture in order to not only manage but to win the race for more adaptive trading as the market evolves.
What traders must optimize their IT infrastructure around
Traders face all kinds of technical challenges when building IT architectures. Designing these architectures must therefore optimize for a number of parameters.
- Data integration and management: Integrating and managing large amounts of data from various sources, such as financial markets, logistics systems, asset data, and internal systems, requires an IT architecture designed as an integral part of the company’s overall corporate strategy.
- Interoperability: Continuous system integration requires highly efficient interoperability, which can be difficult to achieve, especially with legacy systems. Lack of standardization, governance policies, and even data labeling will make it challenging to connect systems across organizations.
- Data Governance: With the increasing amount of data generated and shared among different parties, data governance and data quality have become critical challenges to handle. Data governance policies and systems must ensure that data is accurate, consistent, and accessible to all parties, while data quality monitoring must provide trust that data is accurate and reliable for decision-making.
- Transparency and Traceability: As digitalization brings a new level of transparency and traceability to the commodity trading process, companies must adapt and comply with the latest standards and regulations for traceability, sustainability, and ethical standards. This continuous advancement will put significant strain on the traditional IT architectures, and thus legacy system constraints need to be challenged and re-programmed.
- End-user capabilities: As digitalization brings about new technologies and automation, many seasoned traders today may need to be retrained and reskilled to work with new tools as they come to market. At the same time, many young graduates currently entering the trading industry have the programming skills and technical knowledge base to adopt new and more modular software environments quickly. These new talents will demand access to the tools to work in more digital environments, and traders must provide the best talents with the best digital tools for achieving success.
- Cybersecurity: As the solution architecture gets complex in commodity trading, so do the cyber threats. However, unless you also have a top-tier security team in-house, the solutions you build also risk falling victim to increasingly advanced cybercrime. Therefore, companies need to partner with highly specialized vendors that can cover any critical security weak spots, and design an IT architecture that mitigates risks and provides ability respond to new threats as they emerge.
The trading companies that can manage efficiently coordinating an IT architecture built on modularized software components will be competitive as technology advances further.
Learn More from the Power Industry
In conclusion, IT architectures based on modularized, interoperable software designs are already critical aspects of digitalization in commodity trading. Yes, modularization can present significant challenges related to IT standardization, integration complexity, legacy systems, data governance, security, scalability, and maintenance, but monolithic architectures are no longer a viable option. Modular and highly agile trading architectures will create more responsive and adaptive trading organizations than the traditional "do-everything-yourself" approach.
At Cognite, we work with global commodity traders to build more agile and efficient solution ecosystems, accelerating the development, deployment, and roll out of new analytics solutions directly with the trading organization. Leveraging Cognite Data Fusion® (CDF) to handle the solution orchestration and data source integration, trading companies can move faster, build more advanced solutions, and protect their competitive advantage, especially in high-volatility environments.
"Want to learn more about the use of CDF in commodity trading? Contact author: andreas.proesch@cognite.com"
Read our last article on the topic: 5 Data Challenges in Commodity Trading
Learn more about Cognite and Cognite Data Fusion on www.cognite.com