Over the past few weeks, I’ve shared foundational knowledge about how data is stored and managed in the pipeline industry. My first post introduced ArcGIS Pipeline Referencing (APR) and explained some options operators have in adopting it. My second post worked to demystify the confusing world of pipeline data models (there’s a lot to consider).
In this post, I will outline important information you need to consider when choosing a data model for your organization, including:
- Limitations of current data models;
- How APR is addressing these limitations; and
- Questions you should ask yourself to help assess the best data model for your organization (should you choose to move to APR).
Limitations of Existing Models
A data model is defined as: “An abstract model that organizes elements of data, and standardizes how they relate to one another and to properties of real world entities.”
In the pipeline space, real world entities include not only the pipe and valves that make up the pipeline system, but all of the things that happen to and around the pipe. Things such as repairs & replacements, surveys & patrols, one-call, cathodic protection, ILI & CIS assessments, HCA & class location, land ownership & right-of-ways, and crossing information all have components that, in one form or another, need to be captured in your GIS.
The differing needs of these complex data representations expose limitations in legacy systems. And in a world where critical business decisions must be made from this data, identifying limitations and addressing them is an important step as we move to next-generation solutions.
Limitation #1: Data volume
As the years have progressed, the operational and regulatory needs surrounding pipelines have increased. These needs are driving new levels of inspections and analyses on pipeline systems – resulting in more data, both in terms of volume and complexity. The legacy systems were simply not designed to handle the volume of data current programs produce.
An example is the case of Inline Inspection (ILI) and Close Interval Survey (CIS) data. A single ILI or CIS inspection results in hundreds of thousands of records. With assessment intervals recurring every 1-7 years — and operators performing dozens of inspections each year — the resulting records from these inspections alone add millions of records to the database. This doesn’t include follow-up inspections, digs, and run comparison activities.
When you couple the sheer volume of records with complexities surrounding data management and the need to provide a high-performance environment, limitations in the system are quickly exposed. These limitations force operators to make difficult data storage decisions, often choosing to remove subsets of data from the system of record. This is sub-optimal to say the least; it significantly impacts your ability to view and analyze important trends in the data.
Limitation #2: Engineering Stationing
Engineering stationing is important, complex, and rooted in pipeline data management. Before modern GIS, operators kept track of the location of pipelines and associated appurtenances using engineering stationing on paper or mylar sheets. With a vast majority of pipelines that are in use being constructed before the existence of modern GIS technology, large volumes of data were referenced with this approach.
Engineering stationing doesn’t benefit all operators; however, companies that manage gathering and distribution assets find this method burdensome … and dare I say unnecessary?
When traditional data models were developed, the need to adhere to legacy engineering stationing outweighed the need to redesign the entire system to favor a spatial-first approach. But as technology has improved, and more users have embraced spatial data, new methods to blend modern (spatial-first) and legacy (stationing-first) models have emerged. Operators need this flexibility when managing their assets.
Limitation #3: Native support for the Esri Platform
The emergence of the Pipeline Open Data Standard (PODS) represents the last major shift in data storage solutions for pipelines, and it happened nearly 20 years ago. At that time, the GIS landscape was both immature and fragmented. As a by-product, PODS was designed specifically to be GIS-agnostic. In the nearly two decades since, Esri has emerged as the predominant provider of spatial data management, and they have developed a suite of solutions that enable stronger collection, management, and analysis of data.
Chances are your organization embraces Esri for spatial data management and content dissemination, which begs the question: “If your organization has standardized on Esri technology, does it make sense to employ a data structure that does not natively support the environment?” (Hint: probably not.)
Addressing and Improving Limitations
The core of APR has been engineered to address important limitations currently felt due to the existing designs of PODS and APDM. APR directly addresses the three limitations described above.
Improvement #1: Data volume
Understanding the need to support large datasets, time-aware data, and the ability to offload the storage of data to other systems, APR has been engineered to handle the high volume of data more efficiently, with a focus on scalability. To achieve this, available rules can be configured to allow a more fine-grained approach to managing data during routine line maintenance. No longer are implementations limited to keeping the data referenced to the LRS or detaching it.
Changes like these allow operators to keep more data in the system, providing a baseline for more powerful analysis and decision making.
Improvement #2: Engineering Stationing
As explained above, engineering stationing is firmly rooted in pipeline data management, but it’s not required for all operators. New construction, gathering systems, and vertically-integrated distribution companies are finding the rigorous application of stationing to be unnecessary overhead. If your current database repository requires it, and your organization doesn’t rely on it, you are taking on unnecessary data management cycles – costing valuable time and money.
APR not only provides the ability to manage data in stationed and non-stationed methods: its flexibility allows for both stationed and non-stationed lines to exist in the same model. Let that sink in for a bit: Operators that have deployed two separate data management systems can now consolidate the management of these assets! This functionality benefits a majority of the clients I’ve worked with over the years.
Improvement #3: Native support for the Esri Platform
As I stated in my previous post, APR is (possibly most importantly) integrated with the ArcGIS platform. You can perform complex long transactions on your data, analyze it in ways that have not been possible before, keep track of product flow using the Facility Network, and get the data in the hands of your organization with methods that are integrated, fluid, and connected.
Considerations for Implementation
If you’re considering implementing ArcGIS Pipeline Referencing (APR), knowing why, and which data model to use with it is has more to do with your business than with IT — success can be achieved with either model.
But how do you decide which one is best for your organization? Here are some questions to consider as you’re laying the foundation for your next-generation GIS implementation.
1) Business focus: What segment of the vertical are you in?
If you are a distribution company with transmission assets, the decision is pretty clear: you should choose Utility and Pipeline Data Model (UPDM). It’s designed as a distribution-first model, allowing you to integrate the management of distribution and transmission assets in a single model.
If your company is ‘upstream’ of distribution, the answer gets a bit trickier. Both models are adequate, but my vote tends to lean towards PODS for a few reasons:
- Out-of-the-box PODS supports APR slightly more natively for operators without distribution assets than UPDM.
- Are you a liquids operator? As UPDM is focused on gas utility and transmission, the PODS model will provide a better solution for those moving liquid products.
- As an organization delivering a comprehensive model to the industry, PODS is a thriving community of operators and vendors working together to design a comprehensive model for the industry. This collection of subject matter expertise is invaluable to operators – and provides an opportunity to share your experience with like-minded individuals.
2) Existing model: What are you using now?
As you consider moving to APR, understand that it’s going to require a data migration. The existing system will need to be mapped and loaded into the new solution. If you are currently using PODS and are a gathering, midstream, or transmission company, APR with PODS is probably the best solution to implement. It’s likely that your existing data will migrate more seamlessly, and the model will make more sense to those that manage and interact with the data.
If your organization is primarily gas distribution, and you’ve implemented a PODS model for a small subset of high-pressure assets in the system you manage, consider UPDM. You can take advantage of the intended benefits and consolidate those assets into a common platform.
3) Other programs: ILI, CIS, other survey, Cathodic Protection
If your company has a strong investment in recurring inspections, PODS again rises as the preferred model, especially considering the efforts of the PODS Next Generation initiative around how to efficiently store and process this data moving forward.
4) Interoperability
With the growing importance of importing and exporting data (due to acquisitions, divestitures, etc.), analysis, and reporting, a system that promotes standard mechanisms to exchange data becomes increasingly more important. With the work the PODS organization is putting into a data interchange standard, it again rises as the preferred model.
There isn’t just one approach, but there is a best approach for your organization
While this change is beneficial for operators, many things need to be considered before you commit to an approach. I hope my series of posts provides some clarity for you. To stay up-to-date on the data model landscape and the tools surrounding it, I encourage you to follow the PODS association and Esri. The work of these two organizations in the pipeline space is a great thing for our industry.
If you’d like to discuss any of these concepts further, or would like to have a conversation about which model is best for your implementation, please get in touch with me here. I, and the rest of the Energy team at Latitude, are eager to offer our years of expertise to help you.
Contributors