Showing 3 result(s) for tag: gis

How to search for data in a non-spatial database [Geocortex Tech Tip]

Maps allow you to visualize data in meaningful ways and expose patterns that can’t be seen anywhere else. One of the challenges, though, is that your most important business data typically lives in another system or database. This can become even more challenging when it’s data stored outside your geodatabase.

In this Geocortex Tech Tip, Drew Millen shows you how to search for data in a non-spatial database (such as Oracle or SQL), find the spatial relationship, and display it on a map. 

Watch on YouTube

 

Video Transcription

“Hi everybody, I’m Drew with Latitude and in this Tech Tip we’re going to look at searching for non-spatial data. That’s data stored in Oracle or SQL Server… somewhere that’s not in your geodatabase. We’re going to look for that, find the spatial relationship, and display it on a map, so let’s see how we do that with Geocortex.

What we’re looking at here is a very basic Geocortex viewer application that’s been configured with a single layer called “Land Use”. This contains polygons of different types of land uses and what I’m interested in is this “Arts and Recreation” land use polygon, which contains park information for Los Angeles County. I also have a database table - in this case, an Excel spreadsheet of trees. Now notice that I‘ve got records of all the different types of trees that exist, but I don’t have location information for these. In other words, this is a non-spatial database table. This could live in Oracle or SQL Server, but for the sake of this demonstration it’s just an Excel table.

We’ve got a facility that tells us which park this tree belongs to, but we still don’t have its “XY” location on the map. What I want to find out is where I can find certain trees in my county, so, what parks do I have to visit to discover certain types of trees.

Now in this application, I’ve got a data link between my parks layer, or my land use layer, and the tree database. So, if I have a park selected, and I view the additional details for [the park]; I can see the spatial details associated with that park and I can also see the trees that are within that park, but I’m not quite there. What I want to find out is which parks contain which trees... and remember, my trees don’t have “XY” locations.

How do I solve this? Well, I’ve already set it up so that we can do a search against this Excel table. So, if I do a search for the word “macadamia”, for example, I will find search results from that Excel table, but I still don’t have the location on the map where these macadamia nut trees exist. I need to create a “join” between these search results and a spatial layer on the map to find the underlying spatial feature. In other words, the park that the trees live within.

What I can do is come back to Geocortex Essentials Manager where I’ve configured this application. And to connect to this Excel spreadsheet, I’ve established a data connection. You can see the connection string that we’ve used here simply points to the spreadsheet. If you’re connecting to Oracle or SQL Server, there’s different syntax that you would use for your connection string, but the same idea exists.

Now that we have that data connection, we can set up what we call a “Search Table.” And a search table gives us a select clause: in other words, which fields are we interested in returning from that table when the user issues a search. In this case, we want the user to be able to search on the common name of the tree (like my example when I typed in the keyword “macadamia”) and find all the attributes from the LA Parks trees in this database. So that search is set up.

I’ve also got the land-use layer in my site configured with a datalink. This datalink means that the layer is joined to this data connection, so that every time I click on a park on the map, I see the associated records from my Excel spreadsheet. Recall, however, that I want to do the reverse. So, our current datalink makes sure that every time I select a park on the map I’m grabbing the trees and joining it on the facility column. Notice that facility column is the name of the column that we're using in the spreadsheet to represent the park that the tree exists within.

There’s this section down at the bottom, here, that allows me to add a search, so that’s the reverse of what we’re currently doing, and it allows me to use one of the searches that I’ve configured to find these features from the land use layer that match my search criteria from my datalink.

I’m going to give this a display name. We’ll just use “Park Trees Search,” and the search table that I’m searching on is the only one that we’ve configured in our site earlier, so it’s this Park Trees Search table. And then the field that we want to join is called “Facility,” and that maps to the name of the land use polygon. So that’s where we get our many-to-one relationship from. I’ll go ahead and save the site with those configuration changes and then refresh our viewer.

Now I’m going to issue a search for the word “macadamia” like I did before, and I’ll find the same four results from my Excel spreadsheet. But now when I drill into a result, we can see the facility that it belongs to. It exists in two different parks: there’s “Runyon Canyon Park” and “Runyon Canyon Park – MRCA”. If I click on one of those it’s going to take me to the park where I can discover these macadamia trees.

Hopefully this quick Tech Tip has shown you how you can configure a non-spatial data source to be searchable inside your viewer and still return spatial results. Thanks for watching!”

Explore more Geocortex Essentials functionality in the Discovery Center.

Discover Geocortex


GIS Health Assessment: A new way to think about your system

When we think about the health of our GIS, many of us are used to beginning with the infrastructure. After all, it is what drives the technical performance of your system. The problem with starting at the infrastructure level, though, is that it’s difficult to get a complete picture of which aspects of the GIS are most important to your users.

While the GIS infrastructure is extremely important, not all the resources in your environment are created equal. You might have some layers or services that are used 3-4 times a week, and others that are accessed thousands of times each week. While you want to do all you can to ensure the entire system is performing as it should, there are only so many hours in a day. With an infrastructure-first approach, you’re often unable to hone in on what the most important apps, layers, services, servers, and ArcGIS instances are. 

 

A new way to think about GIS health

It’s time we flip the traditional infrastructure-first approach and begin thinking about GIS health through the lens of end-user productivity. Your GIS is there to help your users do their jobs, so that’s where your analysis should start.

Whether it’s explicitly or implicitly, you’re going to be measured on the productivity of the users you build apps for, not the response time of a specific server. Without the users, there is no need for the GIS infrastructure.

By starting with what your users are trying to accomplish, you’ll be able to map your key business processes and user flows to the GIS infrastructure and resources that are most important to supporting them. Looking at your GIS from users’ perspectives allows you to better understand how it is being used day-to-day and identify the critical resources needed to support your monitoring and optimization efforts.

With so many moving pieces in your GIS, you don’t have time to treat everything equally.  Focusing your efforts will let you be much more productive and spend more time working on high-value activities.

When we talk about a user-first approach to GIS health, there are two major areas that you need to be considering:

Performance: While closely tied to infrastructure performance, what we mean here is the performance of your end-users. Are they able to do their jobs effectively with the tools and applications you’re building? Are your users taking longer than expected to complete certain tasks?

When these things crop up, a user-first approach will help you target your efforts and fix issues quicker. A good example would be if an application had a poorly performing layer. This would be an infrastructure performance issue, but if you understand what specific layers and services are used in that application, you will know where to look to address the issue.

Usability: If your GIS infrastructure is performing as expected, the next area to examine is usability. Usability is all about whether your applications are configured and designed in a way that makes sense for what your users need to do. Strong infrastructure performance combined with poor usability is still poor performance (remember, performance is about end-user performance, not infrastructure).

An example of how usability can affect performance is when a common tool is not in a prominent location in your app. If it’s difficult to find, users will waste time looking for it, take longer to complete a task by using a different method, or abandon it entirely. This is also true when incorrect layers are loaded by default – users end up wasting time searching for the layers they need.

Completing a user-first health assessment

Once you’ve adopted a user-first approach to GIS health, you’re ready to perform a user-first health assessment. What you’re trying to accomplish is mapping out the business processes and use cases that you manage with your GIS to the specific GIS resources that support them.

First, you’ll want to identify the different user groups that leverage your GIS. By user group, we mean a group of users that have common work patterns and engagement with your applications. This could be a group of people (or one person) with the same task in your head office, or it could be a specific field crew that uses an app on a tablet. The key here is to identify people who use the GIS in similar ways.

We’ve created a checklist to help you perform a health assessment; it’ll help you map what your different user groups need to accomplish to the GIS resources and infrastructure required to support their work.

The checklist contains areas to detail the users and what they need to do, the app(s) they use, the most-used layers, the services the app(s) consume, which ArcGIS products are used and how they’re configured, and the server(s) that support it.

Get your GIS health assessment checklist now  

What to do with your health assessment

Once you’ve completed your GIS health assessment, you can use the information you’ve gathered to proactively monitor the GIS resources that are the most important. Tools like Geocortex Analytics allow you to configure personalized dashboards that provide a snapshot of the resources you want to monitor.

You can also configure alarms and notifications in some systems monitoring tools. Because you know what you need to monitor, you can set thresholds for warning signs of potential issues and have notifications sent to your email.

Next, identify anomalies among your use patterns. If certain users are performing notably better or worse than the average, you can dive into the specifics of how those individuals are using the applications you’ve built. Replicate the superior use patterns and examine the weaker patterns to gauge if there is a potential gap in training or understanding of certain functions.

If you want to learn how all of this is possible with Geocortex Analytics, we’d like the chance to show you! We’ve recently added great new features (including individual user reporting) and made significant improvements to performance and reliability. Get in touch with us using the button below.

Let's chat


ArcGIS Pipeline Referencing: Choose the best data model

Over the past few weeks, I’ve shared foundational knowledge about how data is stored and managed in the pipeline industry. My first post introduced ArcGIS Pipeline Referencing (APR) and explained some options operators have in adopting it. My second post worked to demystify the confusing world of pipeline data models (there’s a lot to consider).

In this post, I will outline important information you need to consider when choosing a data model for your organization, including: 

  • Limitations of current data models;
  • How APR is addressing these limitations; and
  • Questions you should ask yourself to help assess the best data model for your organization (should you choose to move to APR).
 

Limitations of Existing Models

A data model is defined as: “An abstract model that organizes elements of data, and standardizes how they relate to one another and to properties of real world entities.”

In the pipeline space, real world entities include not only the pipe and valves that make up the pipeline system, but all of the things that happen to and around the pipe. Things such as repairs & replacements, surveys & patrols, one-call, cathodic protection, ILI & CIS assessments, HCA & class location, land ownership & right-of-ways, and crossing information all have components that, in one form or another, need to be captured in your GIS.

The differing needs of these complex data representations expose limitations in legacy systems. And in a world where critical business decisions must be made from this data, identifying limitations and addressing them is an important step as we move to next-generation solutions.

Limitation #1: Data volume

As the years have progressed, the operational and regulatory needs surrounding pipelines have increased. These needs are driving new levels of inspections and analyses on pipeline systems - resulting in more data, both in terms of volume and complexity. The legacy systems were simply not designed to handle the volume of data current programs produce.

An example is the case of Inline Inspection (ILI) and Close Interval Survey (CIS) data. A single ILI or CIS inspection results in hundreds of thousands of records. With assessment intervals recurring every 1-7 years -- and operators performing dozens of inspections each year -- the resulting records from these inspections alone add millions of records to the database. This doesn’t include follow-up inspections, digs, and run comparison activities.

When you couple the sheer volume of records with complexities surrounding data management and the need to provide a high-performance environment, limitations in the system are quickly exposed. These limitations force operators to make difficult data storage decisions, often choosing to remove subsets of data from the system of record. This is sub-optimal to say the least; it significantly impacts your ability to view and analyze important trends in the data.

Limitation #2: Engineering Stationing

Engineering stationing is important, complex, and rooted in pipeline data management. Before modern GIS, operators kept track of the location of pipelines and associated appurtenances using engineering stationing on paper or mylar sheets. With a vast majority of pipelines that are in use being constructed before the existence of modern GIS technology, large volumes of data were referenced with this approach.

Engineering stationing doesn’t benefit all operators; however, companies that manage gathering and distribution assets find this method burdensome … and dare I say unnecessary?

When traditional data models were developed, the need to adhere to legacy engineering stationing outweighed the need to redesign the entire system to favor a spatial-first approach. But as technology has improved, and more users have embraced spatial data, new methods to blend modern (spatial-first) and legacy (stationing-first) models have emerged. Operators need this flexibility when managing their assets.

Limitation #3: Native support for the Esri Platform

The emergence of the Pipeline Open Data Standard (PODS) represents the last major shift in data storage solutions for pipelines, and it happened nearly 20 years ago. At that time, the GIS landscape was both immature and fragmented. As a by-product, PODS was designed specifically to be GIS-agnostic. In the nearly two decades since, Esri has emerged as the predominant provider of spatial data management, and they have developed a suite of solutions that enable stronger collection, management, and analysis of data.

Chances are your organization embraces Esri for spatial data management and content dissemination, which begs the question: “If your organization has standardized on Esri technology, does it make sense to employ a data structure that does not natively support the environment?” (Hint: probably not.)

Addressing and Improving Limitations

The core of APR has been engineered to address important limitations currently felt due to the existing designs of PODS and APDM. APR directly addresses the three limitations described above.

Improvement #1: Data volume

Understanding the need to support large datasets, time-aware data, and the ability to offload the storage of data to other systems, APR has been engineered to handle the high volume of data more efficiently, with a focus on scalability. To achieve this, available rules can be configured to allow a more fine-grained approach to managing data during routine line maintenance. No longer are implementations limited to keeping the data referenced to the LRS or detaching it.

Changes like these allow operators to keep more data in the system, providing a baseline for more powerful analysis and decision making.

Improvement #2: Engineering Stationing

As explained above, engineering stationing is firmly rooted in pipeline data management, but it’s not required for all operators. New construction, gathering systems, and vertically-integrated distribution companies are finding the rigorous application of stationing to be unnecessary overhead. If your current database repository requires it, and your organization doesn’t rely on it, you are taking on unnecessary data management cycles - costing valuable time and money.

APR not only provides the ability to manage data in stationed and non-stationed methods: its flexibility allows for both stationed and non-stationed lines to exist in the same model. Let that sink in for a bit: Operators that have deployed two separate data management systems can now consolidate the management of these assets! This functionality benefits a majority of the clients I’ve worked with over the years.

Improvement #3: Native support for the Esri Platform

As I stated in my previous post, APR is (possibly most importantly) integrated with the ArcGIS platform. You can perform complex long transactions on your data, analyze it in ways that have not been possible before, keep track of product flow using the Facility Network, and get the data in the hands of your organization with methods that are integrated, fluid, and connected.

Considerations for Implementation

If you’re considering implementing ArcGIS Pipeline Referencing (APR), knowing why, and which data model to use with it is has more to do with your business than with IT -- success can be achieved with either model.

But how do you decide which one is best for your organization?  Here are some questions to consider as you’re laying the foundation for your next-generation GIS implementation.

1) Business focus: What segment of the vertical are you in?

If you are a distribution company with transmission assets, the decision is pretty clear: you should choose Utility and Pipeline Data Model (UPDM). It’s designed as a distribution-first model, allowing you to integrate the management of distribution and transmission assets in a single model.

If your company is ‘upstream’ of distribution, the answer gets a bit trickier. Both models are adequate, but my vote tends to lean towards PODS for a few reasons:

  1. Out-of-the-box PODS supports APR slightly more natively for operators without distribution assets than UPDM.
  2. Are you a liquids operator? As UPDM is focused on gas utility and transmission, the PODS model will provide a better solution for those moving liquid products.
  3. As an organization delivering a comprehensive model to the industry, PODS is a thriving community of operators and vendors working together to design a comprehensive model for the industry. This collection of subject matter expertise is invaluable to operators – and provides an opportunity to share your experience with like-minded individuals.

2) Existing model: What are you using now?

As you consider moving to APR, understand that it’s going to require a data migration. The existing system will need to be mapped and loaded into the new solution. If you are currently using PODS and are a gathering, midstream, or transmission company, APR with PODS is probably the best solution to implement. It’s likely that your existing data will migrate more seamlessly, and the model will make more sense to those that manage and interact with the data.

If your organization is primarily gas distribution, and you’ve implemented a PODS model for a small subset of high-pressure assets in the system you manage, consider UPDM. You can take advantage of the intended benefits and consolidate those assets into a common platform.

3) Other programs: ILI, CIS, other survey, Cathodic Protection

If your company has a strong investment in recurring inspections, PODS again rises as the preferred model, especially considering the efforts of the PODS Next Generation initiative around how to efficiently store and process this data moving forward.

4) Interoperability

With the growing importance of importing and exporting data (due to acquisitions, divestitures, etc.), analysis, and reporting, a system that promotes standard mechanisms to exchange data becomes increasingly more important. With the work the PODS organization is putting into a data interchange standard, it again rises as the preferred model.

There isn’t just one approach, but there is a best approach for your organization

While this change is beneficial for operators, many things need to be considered before you commit to an approach. I hope my series of posts provides some clarity for you. To stay up-to-date on the data model landscape and the tools surrounding it, I encourage you to follow the PODS association and Esri. The work of these two organizations in the pipeline space is a great thing for our industry.

If you’d like to discuss any of these concepts further, or would like to have a conversation about which model is best for your implementation, please get in touch with me here. I, and the rest of the Energy team at Latitude, are eager to offer our years of expertise to help you.