Using Cloud Optimized Geotiff part 1

  • blog

The Cloud Optimized Geotiff (COG). Is it helping you to stop unnecessarily downloading data? It should.

One of the biggest Earth Observation ‘things’ in 2019 was that USGS is delivering Landsat 8 as COGs (that have a SpatioTemoral Asset Catalog – STAC as well)

But old practices die hard and sometimes, despite the knowledge of new tools/processes (or whatever) that are available we revert back to what we know. Visit the Earth Explorer on USGS website (login, sigh) and search for your data. Inspect it, download it, use it. Or there are a multitude of other ways; a really nice one is the semi automatic classification plugin for QGIS. I wrote about using it here.

Now though there is a new, perhaps simpler, option. You can stream these Rasters directly into QGIS (min requirement 3.4 LTR – soon to be 3.10 LTR). This obviously saves you downloading locally and potentially storing in perpetuity (we should all think about how big our digital footprints are).

You can do this via the Data Source Manager – more on this shortly. How is this possible? Amazon Web Services host this data, free to access for everyone.

https://landsatonaws.com/

You still need to find the data though. If you know the path and row then you could step through the data structure until you arrive at the image you need.

What we are after here is the link to the direct data so we can load it into QGIS. An example (on the West Coast of the North Island of New Zealand) is here:

http://landsat-pds.s3.amazonaws.com/c1/L8/073/087/LC08_L1TP_073087_20190818_20190902_01_T1/LC08_L1TP_073087_20190818_20190902_01_T1_B2.TIF

If you click the link you will be prompted to download the tile directly. BUT we don’t want to do that. We want to use this link and stream the data. Again there are plenty of options to search for tiles, such as:

https://search.remotepixel.ca – is excellent, but it is not for commercial use.

http://demos.fmeserver.com/landsat-sentinel-selector/ – simple to use.

You could download the tiles and rows vector data and use that to work out which path and row you need for your area of interest.

https://eos.com/landviewer/ – this is a brilliant way of finding data (not just Landsat).

There are plenty of other web sites, choose the one that works best for you. Another option is to access the data programmatically. sat-api from development seed is a very good choice: https://github.com/sat-utils/sat-api

I am going to assume at this point you have an aws landsat 8 link (if not, just copy the one above). In QGIS (3.4+) select layer – Data Source manager. Select the Raster option and the Protocol HTTP(s) radio button. Paste link to the data in the URL box as shown below:

Click on Add and repeat for bands 4 and 3 – you only need to change the end part of the file:

ttp://landsat-pds.s3.amazonaws.com/c1/L8/073/087/LC08_L1TP_073087_20190818_20190902_01_T1/LC08_L1TP_073087_20190818_20190902_01_T1_B3.TIF

ttp://landsat-pds.s3.amazonaws.com/c1/L8/073/087/LC08_L1TP_073087_20190818_20190902_01_T1/LC08_L1TP_073087_20190818_20190902_01_T1_B4.TIF

Click on close and the data should appear as three grayscale images. Use the Raster – Miscellaneous – Build Virtual Raster to stack these bands together to create an RGB.

Make sure you set the order of bands 4,3,2 (for true colour image). Save the file. Click on run and the following should appear in QGIS.

This is super nice – the vrt file we just created is 2.62kb. Close this project down and open a completely new project. Grab the vrt file you just created and drag it into the map.

Astonishingly you now have the file. Visualize it as needed. Clip it, save it (if you must as a GeoTiff). As long as you have a connection to the internet AND that the data remains hosted on aws for free then this should always work. The really nice thing is that you could find another (cloud free!) date for this Landsat tile and take a copy of the vrt file just created, change the paths and you have another scene of data. Hopefully you can see the potential to build up direct links to data pretty quickly.

If you are still downloading Landsat 8 data, perhaps make 2020 the year you stream the data as a COG.

This post was inspired by this post:

https://www.cogeo.org/qgis-tutorial.html

Next time I’ll look at accessing a COG file using Python, and how we can answer a very common question – can I download a subset of a tile?

I run through this, and a lot more, on my Geospatial Python programming courses. If you are interested in finding out more visit www.acgeospatial.co.uk/training or email me on info@acgeospatial.co.uk

I am a freelancer able to help you with your projects. I offer consultancy, training and writing. I’d be delighted to hear from you. Please check out the books I have written on QGIS 3.4

https://www.packtpub.com/application-development/learn-qgis-fourth-edition

https://www.packtpub.com/application-development/qgis-quick-start-guide


I have grouped all my previous blogs (technical stuff / tutorials / opinions / ideas) at http://gis.acgeospatial.co.uk.

Feel free to connect or follow me; I am always keen to talk about Earth Observation.

I am @map_andrew on twitter