GETTING AHEAD WITH MINE DATA PLATFORMS
Susan Wick and Sean Hunter dispel common myths about in-house mining data platform development and examine alternatives for sustainable organizational performance.
If data is the new oil, then mining companies need a system to manage that flow throughout their organizations and deliver value to where it’s needed, when it’s needed. The imperative is well understood, but the question that many companies are grappling with is whether to develop that system (or parts of) it in-house, or opt for an off-the-shelf data management solution?
There’s often an assumption that there is a competitive advantage to be gained from creating a technology or system that is unique to a company. Miners are, after all, exceptional in the operations they run and the challenges and opportunities they face. But time and trial have proven that this is not always the case.
In reality, many underestimate not only the initial cost, skills and changes required to create a useful data platform, but also the ongoing operational implications. If longevity is the name of the game, then focusing effort and resources on a project that is outside of the business’ core interests may not be the optimal move.
A data platform is a centralized, secure system for the storage, handling, processing and analysis of an organization’s data. Bringing it together in one place provides a basis for artificial intelligence (AI) and machine-learning (ML) technologies to be applied, as well as operational optimization programs.
Today, the most common approach is for mining companies to piece together their own data platforms using one solution for data storage – say a data lake or warehouse – which is connected to another solution for data processing, and another for analysis and reporting. It might also connect to various application-specific solutions, for instance, software for mine design, scheduling, asset maintenance and other duties.
The result is a custom-built web of hundreds, if not thousands, of point solutions which, while made functional in the short term via scripts and application programming interfaces (APIs), are not necessarily designed to be interoperable or compatible in the long run.
Over time as different products are introduced, older ones updated with new features and functions, and as the organization grows, it becomes increasingly difficult to maintain that integration and to get the insights needed from the data in an accurate and timely manner. Ultimately, most custom-build projects like this fail because their interfaces are not user friendly, and they do not deliver the value the organization hoped for.
As CEO Susan Wick explained: “It’s a well-known business axiom that companies should stick to their core areas of expertise. Focusing on things outside of that can weaken the structure of the organization.
“If we think back to the early years of mine planning software, several mining companies tried to develop their own solutions, but eventually they all switched to commercial products, because developing and maintaining solutions in-house just wasn’t sustainable.”
Eclipse is the developer of the SourceOne enterprise knowledge performance system.
Sean Hunter, director of product development, concurred with Wick: “We’ve seen a number of mining companies develop their own data lakes in recent years, but most stop there. They don’t go on to build the infrastructure and models needed to support that. They then run into problems with data accessibility or find the system has limited integration with third-party applications, none of which are very elastic. But because they’ve invested so much time and capital into the project, they have no choice but to continue maintaining it.”
DIY: WHAT’S INVOLVED?
The initial cost of building a data platform varies depending upon the elements the company wants to custom-develop, the skillsets and tools it already houses as well as its existing infrastructure and computing power. Wick estimated the capital cost for most development projects by mining companies runs into tens of millions of dollars. But data platforms that inform core business functions are long-term investments and therefore require ongoing maintenance and expansion to support the business’ own evolution.
“Developing and maintaining a platform requires deep expertise and a team that is dedicated to it 24/7,” she said. “Apart from the fact that there is a shortage of skilled workers in the mining industry, particularly data specialists, mining is a cyclical business and when times get tough, companies tend to make cuts in areas that are not a core competency. If they have a collection of software solutions rather than a whole holistic platform that needs maintaining then that can become very difficult, especially if there are overarching auditability needs that must be adhered to.”
The importance of usability and flexibility cannot be understated. For useful insights to be gained and operationalized, and for reporting to be effective, the data must be accessible to all users. Developing intuitive user interfaces for data platforms requires a very specific set of skills.
Hunter explained: “Without a good user interface, only super-users can navigate the system. This causes users to delegate work and the data team can become an organizational bottleneck. But giving users access to data also raises questions around security and verification, data sharing and permissioning. Once companies understand the ongoing costs and capabilities required to run these systems, then they start to see the value in working with commercial developers.”
Another issue is the growing need for real-time data. This data is complicated to handle and can require the creation of a microservice which demands both IT and data knowledge. A microservice can be tricky to setup and maintain if a company chooses to do so itself.
Ultimately, there is little point in spending millions of dollars to develop and maintain a platform if users find it too difficult to use and eventually revert to manual data handling and processing. Or, if it cannot support growth of the company and its objectives over time.
“Our solution, SourceOne, is the opposite of this,” explained Wick. “It’s a unified data solution that provides everything an enterprise needs to ensure its optimal performance. SourceOne underpins all the legacy and point solutions that mining companies have and, by handling the organization’s data in a holistic way, there are benefits to be gained around collaboration, data preservation, security and longevity.”
Hunter added: “SourceOne was developed around the user experience, considering how people work, how they interact with the platform and what they need it to do. Inside of the system, there are ways to communicate and send messages, assign tasks and work collaboratively through processes. Every tool within SourceOne works seamlessly together. That allows us to create custom dashboards for different users and teams quickly and easily, and overlay them with specialized charts, text and real-time data.”
THE CASE FOR COMMERCIAL TOOLS
Oftentimes, companies will assume there is no off-the-shelf data solution that can meet their needs. That’s another reason why they might choose to create one themselves – a process that can take years. However, there are flexible commercial solutions, like SourceOne, that can be configured to meet differing needs. Rather than starting from scratch, these solutions act as a springboard, allowing miners to harness their data faster, and innovate only in areas where it’s truly necessary.
SourceOne is designed to break down the functional silos that occur within companies. It allows data to be shared across departments and for teams to communicate and collaborate in real time. SourceOne uses open and standard data formats, and users have the option to view data in their preferred format. For example, one department could import a geometry file in one format, such as .OMF, and a user in another department could consume it in a different format, such as .DWG files. SourceOne supports this capability and synchronizes the files.
The system includes layers of automations which support this and also allow certain tasks and workflows to be completed automatically or semi automatically.
Hunter explains: “For instance, an automation might be used to keep a file in SourceOne in sync with a file outside of SourceOne. Having these capabilities allows data to flow freely between systems and provides users with an up-to-date picture of what’s going on in the business at all times.”
SourceOne can run on-premise or in the cloud via the SourceOne desktop application. Importing data into the app means it then becomes immediately available for use with web-based APIs and other cloud-based technologies.
Hunter added: “The cloud isn’t required to run SourceOne, but it can take advantage of it if it’s available. This is useful for mines in remote locations that can’t quickly or reliably use the public cloud.”
Properly organized, good quality data is the basis for achieving optimal outputs from AI and ML technologies. These systems require vast amounts of data to train their algorithms and provide reliable predictions, and SourceOne’s ability to centralize and standardize data supports that capability.
Standardization is also important when it comes to corporate reporting. Most mines, even those within a single organization, tend to run standalone when it comes to software and reporting but, having a unifying system that spans the value chain allows insights to be shared and efficiencies created.
“We worked hard to ensure that SourceOne would be quick and easy to deploy and scale, regardless of the size or type of organization,” said Hunter. “The platform can support as intricate or as basic an operation as a company wishes, and that can be expanded as their needs and demands change; it’s essentially a tool to make businesses more agile.”
SHORT-TERM WINS, LONG-TERM BENEFITS
Agility is vital in an industry whose purpose and objectives are fast-changing. The speed of technological development is only going increase in the coming decades and, to prepare for and capitalize upon that growth, companies need data management solutions that are flexible, scalable and, most importantly, sustainable.
The decision to build or buy a solution must be made on an individual basis, but there is significant evidence that commercial implementations are usually more successful than in-house builds. Why go to the effort and expense of systems development if there is no real need or advantage to be gained?
Wick summarized: “If a mining company can get ahead of the game by using an off-the-shelf tool which helps achieve its current goals and supports growth for tomorrow, whatever that might look like, then that’s a significant advantage,” she said.
“The beauty of SourceOne is that it doesn’t require miners to move away from their preferred suppliers or applications, and they don’t have to change the underlying functionality within their organization. SourceOne acts as a bridge to make those applications more useful. It’s really a win-win situation.”
About the authors: Susan Wick and Sean Hunter are CEO and director of product development, respectively, for Eclipse Mining Technologies.