Saturday, June 28, 2014

GDAL to convert GeoPDF to GeoTIFF - the Windows version

I needed to convert GeoPDF files to GeoTIFF.  I'm not command-line savvy.  So I was hesitant to try this.  When the gdal_translate command failed no matter what I tried, I nearly gave up.  (I can just screen-shot the PDFs.  Right!)  

But I kept digging.  I found that not all GDAL installs support PDF.  I had to get a specific set of GDAL binaries for Windows.  Here's how and why:


I'm doing volunteer/non-profit work to publish map services that show georeferenced map layers for historic maps of  Caroline County, Maryland.  I'm using MapTiler, Tileserver, QGIS, and ArcGIS Online for low-cost production and publishing.  Recent products are:

1875 and 1897 county and towns via AGOL and
- 1897 county in OpenLayers.

Next task is to publish similar services and maps from the USGS Topo GeoPDFs.  I need to convert the GeoPDFs to GeoTIFF in order to publish the tiles as a service.  (Esri has done this already.  But only for paying customers.)

I found a tutorial for using GDAL's gdaltranslate utility.  After installing GDAL and cursing my way through the godawful Windows command line window, I kept getting GDAL errors.  GDAL wasn't recognizing the PDF format.

Why not?  I was following the tutorial steps exactly.  I checked forums and found that GDAL has PDF support in some releases but not in others -- it was working for Ubuntu installs, but not for all Windows installs.   The tutorial used the Ubuntu installer.  I had used the OSGeo4W installer for GDAL.  That one does not include PDF support.  Gotta love opensource.

I found a pointer to GDAL binaries I needed at gisinternals.com.  I downloaded the zipfile with the files that are right for my Windows system.  Simply unzipped the file in a folder.  Then ran the SDKShell.bat file.  No need to install anything.  That worked for me.  Gotta love opensource.







Friday, January 3, 2014

OK. I'm not a GeoHipster. I'm a GIS Worker. This is what I do.


But worry not, GIS worker: Spatial might no longer be special, but projections, datums, and legacy file formats will continue to be very, very special.
      - Brian Timoney, MapBrief

Okay... GIS is not sexy like your iPhone locator app.  But it will always be special.  In the same way that your plumbing and electrical and HVAC systems are special.  Next time they stop working, ask yourself just how special they are.  Next time your iPhone map is missing Main Street, ask yourself how special GIS is.

Consumer and business geo apps and services keep evolving - cutting edge today and obsolete tomorrow.  BigData today - legacy file format tomorrow.  But projections, datums, and georeferencing will always be there to provide ground truth. 

Esri and the ArcGIS platform have been around for 30 years, defining and documenting GIS work.  So when we needed to understand what GIS workers are doing across a variety of industries, we looked at the list of tools in ArcGIS Toolbox as a compendium of GIS workflow and process skills.  

We needed the ArcToolbox list itself as a data set -- not a PDF poster or searchable web site.  I asked Esri for an ArcToolbox list as Excel table or other word processing format, but they could not provide.  So my friend at Esri, Nick Toscano, and I compiled the table ourselves.   That's almost 900 "GIS tools" in an Excel data table.  Easy to sort, filter, annotate, categorize in ways that meet our needs.  

Here it is.

Some of the tools are pretty arcane.  But most are plain markers for the nuts-and-bolts work of GIS professionals in towns, cities, and counties who have to deal with projections, datums, and legacy file formats every day.  Not the next wave or next-big-thing.  Just the actual data and task at hand.

How will we use this?
  • Survey and assess the experience of our GIS workers.
  • Plan GIS skills training and investment.
  • Design new workflows to make common tasks more efficient.
  • As a quicker reference for building geoprocessing services. 



Friday, December 20, 2013

My Very Short History of LiDAR, GIS, and CAD

In a letter to my son, the young architect.  With a reading assignment from Dad.



Alan,

Are you architects seeing more about the integration of these three technologies?

LiDAR started more than a decade ago in remote sensing from satellites, first tested from the shuttles for terrain and elevation modeling.  When I was in the Univ. of Maryland GIS program, they were working on calibrating LiDAR for remote sensing of vegetation - different levels of tree canopy to assess forest health.

When I was working GIS in for the city of Philadelphia, the GIS staff started seeing LiDAR in the urban environment.  GIS and LiDAR vendors were partnering with the city to use LiDAR for 3D modeling of the downtown area.  Then they started using portable LiDAR sensors - the size of a wheelbarrow - to survey the subway spaces in Center City.  And the interiors of some buildings.  The focus was public safety and emergency response.

Now we see LiDAR moving into the consumer market - interior design and real estate are the focus of the two articles below.  Made possible because LiDAR sensors are smaller and cheaper.  (Where have we heard that before... )

Dad



The architect's reply just in:

Dad,

We see this happening but don't use it directly.  There is talk of it being used in surveying but not for design. 

We have our own information-imbedded software (Building Information Modeling - BIM) which we use to design with and to communicate material quantities and interfaces.  The first article mentioned intergratig LiDAR and BIM. 

We are seeing companies which market the 3D scanning service for interiors and you can imagine that we could use that for modeling existing conditions, but the issue is that most of the time we need those existing models to contain information - BIM - and to be manually input. 

The 3d rendering company we work with has a portable version of this and they use it to scan interiors because all they need to do their work is geometries and surfaces. 

I can definitely see an integration of this technology with our BIM but we need and additional layer of information. The BIM software that we use is called Revit.  



Sunday, October 20, 2013

"Trusted-Crowd Sourcing" for Rural County Road Assets Survey and Inventory


Working as a GIS volunteer for Caroline County, Maryland, I created the first-ever web map of RACPro road survey data used to inventory signs, crosspipes, and utilities.   Why does this matter?

1.  County road survey can move away from 10-year-old "distance measuring instruments" (DMIs).  DMIs replaced the hand-wheel and use vehicle odometer readings to survey road assets like signs, crosspipes, and utilities.



2.  Instead of one expensive DMI ($900 each), county personnel could now use any mobile device to collect GPS-generated data.


JAMAR distance measuring instruments typically pay for themselves in increased productivity in a matter of days


3.  The survey data is no longer locked into a proprietary data format (.rac).  Our county owns just one copy of the RACPro software that can display this data using 10-year-old MapPoint technology.



4.  The data and map were locked into the RACPro desktop app and were not shareable.  I exported the data to Excel and published it as a web map at ArcGIS Online:


5.  Because of limited resources -- especially personnel equipped with obsolete DMIs and proprietary software -- our county has only surveyed and inventoried a small portion of county roads.  It would now be possible to train other volunteers, such as Service Learning high school students, Eagle Scout candidates, and senior citizens to use their mobile devices to survey and inventory county road 


View Larger Map

Are geolocation using mobile device GPS accurate enough for this type of survey?

I think it is,when used along with the web map app.  Access to the web map with a streets or satellite imagery basemap would allow a survey volunteer with minimum training to place the asset correctly on the map.  And to edit or update the map point as needed.  This kind of interaction with the data and the map are not possible with the DMI and desktop software now in use.

Can volunteers accurately collect this kind of data?  

Maybe better than the current method.  Volunteers can use the mobile device's camera to document their collection.  Quality controllers can use the photo context to verify the record and edit the data in ways that are impossible with the current technology.

Tell me what you think.

Wednesday, June 27, 2012

"Slashmaps for Mapbox" Drupal Module is cool. And I can do that with ArcGIS Online.


This post shows how I created map galleries for my Drupal web site like the galleries you get from the "Slashmaps for Mapbox" Drupal module.  

I worked on this because my maps are already in ArcGIS Online.  And because I'm a Drupal novice and found the new mapping module intimidating. 

Here is a quick-ref list of all the Drupal map galleries that are described and compared below:

1.  FCC.gov/maps built on the Slashmaps for Mapbox Drupal module.

2.  My ArcGIS Online map galleries based on the <iframe> tag:
     a.  Drupal Map Gallery - Demo 1:  grid view 
     b.  Drupal Map Gallery - Demo 2:  list view
     c.  Drupal Map Gallery - Demo 3:  carousel
     d.  Drupal Map Gallery - Demo 4:  gallery hosted elsewhere and embedded here (gridview)

3.  Marten Hogeweg's Esri_MapGallery Drupal module prototype

Are my ArcGIS Online map galleries as good as the Slashmap for Mapbox module?  You can look for yourself at the links above.  I'll do a capabilities comparison in a future post.

The FCC Announcement:  Slashmap for Mapbox Module for Drupal

I'm in the process of migrating my Choptank River Heritage web site, which already includes a map gallery, to Drupal.  So I was interested in the FCC announcement back in March about their new map gallery.  And their collaboration with the Drupal community to build the Drupal module called Slashmaps for Mapbox.

I took a look at fcc.gov/maps:


And I was suprised at how similar it looks to my own map galleries built from ArcGIS Online at
maps.choptankeriverheritage.org and stories.choptankeriverheritage.org :
  

Since my Choptank River Heritage maps are already published in ArcGIS Online, I looked for a Drupal module that connects to ArcGIS Online.

Is There a Drupal module for ArcGIS Online?

I googled and found Marten Hogeweg's Drupal sandbox for ArcGIS Online.  It was still rough.  So I contacted Marten and asked if Esri will develop and support a Drupal module for ArcGIS Online.  He said there's interest.  And we plan to meet up at the Esri UC to talk more about this.  (Contact Marten or me if you want to join in.)

Meantime, Marten did some quick work over the weekend and committed a new Esri MapGallery module to Drupal git for me to try out.   Here is the very rough result so far.  Marten has already made improvements (such as removing the huge title banner) that I will incorporate in the next few days.

Do I really need a Drupal module for a map gallery?  How about a simple <iframe> tag ?

I also wondered if I couldn't do a simpler Drupal integration using the <iframe> tag.  Same as we see already for embedding ArcGIS Online maps into web pages:


And for embedding Google Maps into web pages:



My four <iframe> Map Galleries for Drupal

Esri's first "map gallery" was a simple display of maps that belong to an ArcGIS Online user group.  Nothing else to do.  Last summer (2011), Esri released its Javascript template that can be published in an organization's own web site.  This required simple Javascript configuration and publishing the html, css, and js files to your web server.  This is still an option.  Since then, Esri has created other ways to publish map galleries in your own web site.  All of them run off of the user groups at ArcGIS Online.  

Here is the list of demo map galleries in my Drupal web site that leverage the <iframe> tag to embed map group contents from ArcGIS Online.  Each of my Drupal pages gives a short explanation of the configuration and subtle differences between them:

1.  Drupal Map Gallery - Demo 1:  grid view is simple, usable as-is with no changes to <iframe> sizing.

2.  Drupal Map Gallery - Demo 2:  list view, <iframe> size has to be tweeked to hide annoying side panel, and the gawdy title bar is not configurable.

3.  Drupal Map Carousel - Demo 3:  map carousel, requires <iframe> sizing but otherwise easy, and pretty cool.

4.  Drupal Map Carousel - Demo 4:  <iframe> of the Esri gallery template already published on my non-Drupal web site.  Gives the most configuration control but requires some knowledge of HTML and CSS.


Let me know of your experience with the Mapbox module or these ArcGIS approaches.

Wednesday, June 20, 2012

Why do I still use both Chrome and Firefox? For bookmarks on desktop, laptop, and mobile.

Web browser bookmarks are important to me.  I use them every day for my professional GIS work.  I need fast access to ArcGIS and Python resources on the web.  I need to get back to training and tutorials.  I need to flip back and forth between web sites, Gmail, and calendar.

I put time and effort into managing and updating web browser bookmarks.  And I need to access them on my laptop, my office workstation, and my Android phone. 

Chrome gives me web bookmarks on my laptop and workstation -- effortlessly.


Firefox puts my bookmarks on my phone.



Neither browser does desktop, laptop, and mobile.  

Why can't Chrome or Firefox get the whole thing right?  I don't know.

Chrome -- Log in and get bookmarks everywhere ... except on my phone.

Some of my co-workers resist the Chrome login idea.  They don't want Google tracking their every web move.

But Chrome with login gives me updated bookmarks on both my laptop and workstation - or any other computer where I choose to log in to my Google account.  But not on my Google/Android phone.

The Android browser is crappy.  No bookmarks management at all.  There is no Chrome-like browser for Android mobile.  At least not the OS running on my DroidX --  Andoid 2.4.3.  There is hope at Android 4.0, which offers Chrome-to-Phone Beta and other Chrome toys.  (If I'm wrong, please leave a comment.)

Firefox -- Manually import Chrome bookmarks to get them onto my phone.

The Firefox mobile browser is pretty nice, with tabs, good handling of Javascript and HTML5.  Best of all, it replicates my bookmarks in the same way I organize them on my desktop.


Not without a cost.  I have to a weekly export-import from Chrome to Firefox to keep all this up to date.


Why don't I use IE?

Because IE mixes together web bookmarks and desktop "favorites".  And that screws up everything.

Is there a better way?

Friday, June 15, 2012

ArcGIS Online pricing - what is a Service Credit worth?

Confused about AGOL pricing?  So was I.

I got on the phone with Esri today.   Here is is our Q&A:


ArcGIS Online for Organizations, Level 1 purchase gets "2500 Service Credits".  But what does that mean?

Service credits are expended when ArcGIS Online functionality is deployed.  With the final release, we are providing a dashboard for the administrator, so that they may review how the service credits are being utilized.

Here are some examples:

1)      Service Credits are not used when you upload data or services to your instance of AGOL
2)      Service Credits are not consumed when using a esri basemap service in your application
3)      Service Credits are used when ‘mash’ up a shapefile, map service, table your company/agency uses in a service
4)      Service Credits are used when you create and store a feature service
5)      Service Credits are used when you create or store a tile or geospatial data service(layer, map package)
6)      Service Credits are used when you use a geoprocessing service (i.e. …batch geocodes), others to be added
7)      Service Credits are used when you do data transfer

This info would be more useful if it read "XX [Number of] service credits are used when..."

Here are examples of AGO Credit Consumption


  • Data Transfer – Data transferred out as hosted services or downloading data files
    • 6 credits / 1 GB data
  • Geocodes –
    • 12.5 geocodes / credit
    • 80 credits / 1,000 geocodes
    • 100,000 geocodes / 8000 credits
  • Tile & Data Storage – 1.2 credits per GB of storage per month, 14.4 credits/yr  per GB of data
  • Feature service –
    • 2.4 credits per 10 MB of storage per month, 28.8 credits/yr  per 10 MB
    • 2880 credits/yr  for 1GB of storage
  • Tile generation – 1 credit per 1000 tiles generated


Is there a more complete table online somewhere -- to help me with service budgeting and making a decision about how much to purchase?

No.  But you might want to download the trial and give it a try. There is a dashboard that shows you how you are using your credits.