Geo-Elektrika has moved to geoeletrika.wordpress.com

Geo-Elektrika has moved to geoeletrika.wordpress.com 

What's eating up my ArcGIS Online credits?

My presence on ArcGIS Online is small:  Choptank River Heritage.

It runs on the Esri Home License - ArcGIS Pro and an Online site with 100 credits - for $100 per year.  I use it to improve my ArcGIS skills while publishing historic maps to support my Choptank River Heritage volunteer blog about Caroline County, Maryland, history

I was surprised to find that 7 percent of my annual credits were used up in the past 14 days, mostly for storage:

Storage of what?  You have to drill down deep to find that.  The help doc is here.  The gist is:

Click View Status > Credits > Storage > [Storage Usage Chart] Feature Storage > [Feature Storage Usage Report] Database Storage.  Then download the CSV.

Now I see it.  I can certainly get rid of this stuff:

And pay out another $100 to keep my historic maps and GIS skills development going.

GDAL to convert GeoPDF to GeoTIFF - the Windows version

I needed to convert GeoPDF files to GeoTIFF.  I'm not command-line savvy.  So I was hesitant to try this.  When the gdal_translate command failed no matter what I tried, I nearly gave up.  (I can just screen-shot the PDFs.  Right!)  

But I kept digging.  I found that not all GDAL installs support PDF.  I had to get a specific set of GDAL binaries for Windows.  Here's how and why:


I'm doing volunteer/non-profit work to publish map services that show georeferenced map layers for historic maps of  Caroline County, Maryland.  I'm using MapTiler, Tileserver, QGIS, and ArcGIS Online for low-cost production and publishing.  Recent products are:

- 1875 and 1897 county and towns via AGOL and
- 1897 county in OpenLayers.

Next task is to publish similar services and maps from the USGS Topo GeoPDFs.  I need to convert the GeoPDFs to GeoTIFF in order to publish the tiles as a service.  (Esri has done this already.  But only for paying customers.)

I found a tutorial for using GDAL's gdaltranslate utility.  After installing GDAL and cursing my way through the godawful Windows command line window, I kept getting GDAL errors.  GDAL wasn't recognizing the PDF format.

Why not?  I was following the tutorial steps exactly.  I checked forums and found that GDAL has PDF support in some releases but not in others -- it was working for Ubuntu installs, but not for all Windows installs.   The tutorial used the Ubuntu installer.  I had used the OSGeo4W installer for GDAL.  That one does not include PDF support.  Gotta love opensource.

I found a pointer to GDAL binaries I needed at gisinternals.com.  I downloaded the zipfile with the files that are right for my Windows system.  Simply unzipped the file in a folder.  Then ran the SDKShell.bat file.  No need to install anything.  That worked for me.  Gotta love opensource.

OK. I'm not a GeoHipster. I'm a GIS Worker. This is what I do.

But worry not, GIS worker: Spatial might no longer be special, but projections, datums, and legacy file formats will continue to be very, very special.
      - Brian Timoney, MapBrief

Okay... GIS is not sexy like your iPhone locator app.  But it will always be special.  In the same way that your plumbing and electrical and HVAC systems are special.  Next time they stop working, ask yourself just how special they are.  Next time your iPhone map is missing Main Street, ask yourself how special GIS is.

Consumer and business geo apps and services keep evolving - cutting edge today and obsolete tomorrow.  BigData today - legacy file format tomorrow.  But projections, datums, and georeferencing will always be there to provide ground truth. 

Esri and the ArcGIS platform have been around for 30 years, defining and documenting GIS work.  So when we needed to understand what GIS workers are doing across a variety of industries, we looked at the list of tools in ArcGIS Toolbox as a compendium of GIS workflow and process skills.  

We needed the ArcToolbox list itself as a data set -- not a PDF poster or searchable web site.  I asked Esri for an ArcToolbox list as Excel table or other word processing format, but they could not provide.  So my friend at Esri, Nick Toscano, and I compiled the table ourselves.   That's almost 900 "GIS tools" in an Excel data table.  Easy to sort, filter, annotate, categorize in ways that meet our needs.  

Here it is.

Some of the tools are pretty arcane.  But most are plain markers for the nuts-and-bolts work of GIS professionals in towns, cities, and counties who have to deal with projections, datums, and legacy file formats every day.  Not the next wave or next-big-thing.  Just the actual data and task at hand.

How will we use this?

• Survey and assess the experience of our GIS workers.
• Plan GIS skills training and investment.
• Design new workflows to make common tasks more efficient.
• As a quicker reference for building geoprocessing services. 

My Very Short History of LiDAR, GIS, and CAD

In a letter to my son, the young architect.  With a reading assignment from Dad.

Alan,

Are you architects seeing more about the integration of these three technologies?

LiDAR started more than a decade ago in remote sensing from satellites, first tested from the shuttles for terrain and elevation modeling.  When I was in the Univ. of Maryland GIS program, they were working on calibrating LiDAR for remote sensing of vegetation - different levels of tree canopy to assess forest health.

When I was working GIS in for the city of Philadelphia, the GIS staff started seeing LiDAR in the urban environment.  GIS and LiDAR vendors were partnering with the city to use LiDAR for 3D modeling of the downtown area.  Then they started using portable LiDAR sensors - the size of a wheelbarrow - to survey the subway spaces in Center City.  And the interiors of some buildings.  The focus was public safety and emergency response.

Now we see LiDAR moving into the consumer market - interior design and real estate are the focus of the two articles below.  Made possible because LiDAR sensors are smaller and cheaper.  (Where have we heard that before... )

Dad

The Integrated Built Environment Data Model - LiDAR News

3D Models for the Real Estate Industry - LiDAR News

The architect's reply just in:

Dad,

We see this happening but don't use it directly.  There is talk of it being used in surveying but not for design. 

We have our own information-imbedded software (Building Information Modeling - BIM) which we use to design with and to communicate material quantities and interfaces.  The first article mentioned intergratig LiDAR and BIM. 

We are seeing companies which market the 3D scanning service for interiors and you can imagine that we could use that for modeling existing conditions, but the issue is that most of the time we need those existing models to contain information - BIM - and to be manually input. 

The 3d rendering company we work with has a portable version of this and they use it to scan interiors because all they need to do their work is geometries and surfaces. 

I can definitely see an integration of this technology with our BIM but we need and additional layer of information. The BIM software that we use is called Revit.  

"Trusted-Crowd Sourcing" for Rural County Road Assets Survey and Inventory

Working as a GIS volunteer for Caroline County, Maryland, I created the first-ever web map of RACPro road survey data used to inventory signs, crosspipes, and utilities.   Why does this matter?

1.  County road survey can move away from 10-year-old "distance measuring instruments" (DMIs).  DMIs replaced the hand-wheel and use vehicle odometer readings to survey road assets like signs, crosspipes, and utilities.

2.  Instead of one expensive DMI ($900 each), county personnel could now use any mobile device to collect GPS-generated data.

3.  The survey data is no longer locked into a proprietary data format (.rac).  Our county owns just one copy of the RACPro software that can display this data using 10-year-old MapPoint technology.

4.  The data and map were locked into the RACPro desktop app and were not shareable.  I exported the data to Excel and published it as a web map at ArcGIS Online:

5.  Because of limited resources -- especially personnel equipped with obsolete DMIs and proprietary software -- our county has only surveyed and inventoried a small portion of county roads.  It would now be possible to train other volunteers, such as Service Learning high school students, Eagle Scout candidates, and senior citizens to use their mobile devices to survey and inventory county road 

View Larger Map

Are geolocation using mobile device GPS accurate enough for this type of survey?

I think it is,when used along with the web map app.  Access to the web map with a streets or satellite imagery basemap would allow a survey volunteer with minimum training to place the asset correctly on the map.  And to edit or update the map point as needed.  This kind of interaction with the data and the map are not possible with the DMI and desktop software now in use.

Can volunteers accurately collect this kind of data?  

Maybe better than the current method.  Volunteers can use the mobile device's camera to document their collection.  Quality controllers can use the photo context to verify the record and edit the data in ways that are impossible with the current technology.

Tell me what you think.

"Slashmaps for Mapbox" Drupal Module is cool. And I can do that with ArcGIS Online.

This post shows how I created map galleries for my Drupal web site like the galleries you get from the "Slashmaps for Mapbox" Drupal module.  

I worked on this because my maps are already in ArcGIS Online.  And because I'm a Drupal novice and found the new mapping module intimidating. 

Here is a quick-ref list of all the Drupal map galleries that are described and compared below:

1.  FCC.gov/maps built on the Slashmaps for Mapbox Drupal module.

2.  My ArcGIS Online map galleries based on the <iframe> tag:
     a.  Drupal Map Gallery - Demo 1:  grid view 
     b.  Drupal Map Gallery - Demo 2:  list view
     c.  Drupal Map Gallery - Demo 3:  carousel
     d.  Drupal Map Gallery - Demo 4:  gallery hosted elsewhere and embedded here (gridview)

3.  Marten Hogeweg's Esri_MapGallery Drupal module prototype

Are my ArcGIS Online map galleries as good as the Slashmap for Mapbox module?  You can look for yourself at the links above.  I'll do a capabilities comparison in a future post.

The FCC Announcement:  Slashmap for Mapbox Module for Drupal

I'm in the process of migrating my Choptank River Heritage web site, which already includes a map gallery, to Drupal.  So I was interested in the FCC announcement back in March about their new map gallery.  And their collaboration with the Drupal community to build the Drupal module called Slashmaps for Mapbox.

I took a look at fcc.gov/maps:

And I was suprised at how similar it looks to my own map galleries built from ArcGIS Online at
maps.choptankeriverheritage.org and stories.choptankeriverheritage.org :
  

Since my Choptank River Heritage maps are already published in ArcGIS Online, I looked for a Drupal module that connects to ArcGIS Online.

Is There a Drupal module for ArcGIS Online?

I googled and found Marten Hogeweg's Drupal sandbox for ArcGIS Online.  It was still rough.  So I contacted Marten and asked if Esri will develop and support a Drupal module for ArcGIS Online.  He said there's interest.  And we plan to meet up at the Esri UC to talk more about this.  (Contact Marten or me if you want to join in.)

Meantime, Marten did some quick work over the weekend and committed a new Esri MapGallery module to Drupal git for me to try out.   Here is the very rough result so far.  Marten has already made improvements (such as removing the huge title banner) that I will incorporate in the next few days.

Do I really need a Drupal module for a map gallery?  How about a simple <iframe> tag ?

I also wondered if I couldn't do a simpler Drupal integration using the <iframe> tag.  Same as we see already for embedding ArcGIS Online maps into web pages:

And for embedding Google Maps into web pages:

My four <iframe> Map Galleries for Drupal

Esri's first "map gallery" was a simple display of maps that belong to an ArcGIS Online user group.  Nothing else to do.  Last summer (2011), Esri released its Javascript template that can be published in an organization's own web site.  This required simple Javascript configuration and publishing the html, css, and js files to your web server.  This is still an option.  Since then, Esri has created other ways to publish map galleries in your own web site.  All of them run off of the user groups at ArcGIS Online.  

Here is the list of demo map galleries in my Drupal web site that leverage the <iframe> tag to embed map group contents from ArcGIS Online.  Each of my Drupal pages gives a short explanation of the configuration and subtle differences between them:

1.  Drupal Map Gallery - Demo 1:  grid view is simple, usable as-is with no changes to <iframe> sizing.

2.  Drupal Map Gallery - Demo 2:  list view, <iframe> size has to be tweeked to hide annoying side panel, and the gawdy title bar is not configurable.

3.  Drupal Map Carousel - Demo 3:  map carousel, requires <iframe> sizing but otherwise easy, and pretty cool.

4.  Drupal Map Carousel - Demo 4:  <iframe> of the Esri gallery template already published on my non-Drupal web site.  Gives the most configuration control but requires some knowledge of HTML and CSS.


Let me know of your experience with the Mapbox module or these ArcGIS approaches.