So I aim to write one post every day about this topic (this post was supposed to be finished last evening but has stretched through until this morning) until all the parts are completed, there are probably three or four parts. This is easy because I am doing georeferencing at the moment on a particular volume of the NZ Rail Maps project.
Firstly, the choice of GIS project CRS. This relates more particularly to the GIS and strictly speaking is not relevant to the georeferencing task but is worth considering if your GIS project intends to use the Linz WMTS layers at any stage.
When you are creating maps in the GIS you need to consider the CRS of the project and this is very important if you are using Linz’s WMTS live aerial photo layers. I use these layers for tracing all the current features of the maps and have been doing so for only the past year. Prior to that, I downloaded the individual map tiles for every rail corridor from the LDS website, but now I only need to download what I actually use for georeferencing.
Linz only supplies its WMTS layers in two CRSs, which are EPSG:2193 (NZTM2000) and EPSG:3857 (WGS84). If your project is not in one of these CRSs, using WMTS is only possible if the GIS can transform the CRS on the fly, and this can be fraught with challenges. In my case, I selected EPSG:3857 for my maps some years ago because of its compatibility with Google Earth KML. Even though I no longer import KML files into my maps in order to ensure there will be no licensing challenges in the project, I have stayed with EPSG:3857 as the project CRS.
If your project is not able to use one of the available CRS that is supported directly by Linz WMTS then downloading all of the tiles you need to cover your corridors and importing them into the GIS to provide the contemporary base imagery for the area being mapped, if this is what you are using, is the alternative. You will need some of this imagery as the georeference base for Gimp in any case. The difference with using it in a GIS and using it in Gimp is that the GIS will be able to load JPEG or TIFF files directly without expanding them out of their compressed state. However, when you import the tiles into Gimp, it breaks everything into separate pixels without the benefit of JPEG compression, and as far as I can tell, the resulting file size does not have the same level of compression that is possible in JPEG. I am not totally sure of this, and I know Gimp has its own compression built in, but I have to be able to account for a small volume of tiles becoming a big volume of Gimp project data.
Secondly it is very important to ensure Gimp is configured with adequate tile cache as the transform tools for handling large images will use up a lot of swap space. My computer has 32 GB of system RAM and 168 GB of SSD based swap space that Gimp can use. At the OS level, the SSD is configured as a Linux swap partition, and then in Gimp, I set the tile cache to 200 GB and it uses the 32 GB of RAM and 168 GB of the swap partition (which in practice is larger than this to allow for other system or application use). As a rule of thumb you could expect to be able to handle a Gimp project with a filesize up to 30 GB with this amount of tile cache. My experience has been not to push larger file sizes (generally I go with no more than 100 layers in my projects as another rule of thumb) since Gimp is more likely to crash during a save operation resulting in data loss. I have also experimented with another computer that has just as much SSD swap available but only 8 GB of RAM and this computer is far less able to handle 20 GB or larger project files and will often crash trying to load them, even though it is nowhere near using up all the swap space. It seems that both the system RAM and SSD swap are important but the amount of RAM makes a bigger difference to performance and stability than the amount of swap.
Thirdly, when you are doing transforms in Gimp you’ll want to consider the best interpolation algorithm. Interpolation is what happens when you increase or decrease the size of a graphical layer. This is because the existing pixels are either being increased or decreased in number, and information from the existing pixels has to be sampled to make the new pixels. If you do a scale that is a whole number multiple of the existing size then each pixel just gets cut into the appropriate number of pieces needed, for example scaling by 2x each size means each pixel becomes four new pixels. But in real life we work with all sorts of scale multipliers and this process becomes a lot more complex. The interpolation algorithms are designed for various capabilities and Gimp has a few different ones supplied. When I started doing this stuff I tried with the most intensive and therefore slowest interpolations, LoHalo and NoHalo, but after a while I found that using Cubic gave just as good a quality with a much faster transformation speed and that has been what I have used ever since.
It’s worth noting that I found the scaling process counter-intuitive until I worked it out in my head after a while. To go from a pixel size of 0.3 metres to 0.15 metres, you scale the layer up 200% each side. I had always thought you would shrink each pixel down to half its previous axis by scaling at 50%. However, what you are actually doing at that point is creating new pixels with 50% of the previous information in them and so actually losing resolution rather than increasing it. When you are scaling up 200%, you are doubling the number of pixels so that each pixel contains less information and this is doubling the resolution. You end up with a layer that contains four times as many pixels, but when you tell the GIS to draw them at half the original size, the layer occupies the same area of the screen as the original.
The key reason we want to scale 0.3 m to 0.15 m, or perhaps 0.4 m to 0.1 m, or 0.75 m to 0.15 m, is that we want to avoid downsampling our historical layers too much. I found this out early in the piece with one project having to use 0.3 m background, putting a 1:5500 corridor survey image onto it, scaling that down in the transform too much, and losing too much detail. So these considerations have been worked out by trial and error. I now decide that base imagery where the size of a pixel is more than 0.15 m is a bit too big. I am still working with a lot of 0.3 m base imagery around NZ and scaling it to 0.15 m is the best way to deal with it. The result is a larger tile size and it is easier just to go with those larger tiles than to try dividing up the new tile into four tiles of the original size, because they have to be given a new naming scheme, and because you have to work out the coordinates of the top left corner of something that is spaced halfway along one side of the original tile, which makes for extra work. I tried this for a while but eventually I came around to understanding that a bigger tile, that is actually drawn by the GIS at the same physical area as the original by telling it the pixels are half the original size, is not enough of a deal to warrant getting pedantic about the size of the new tiles.
There is one more thing to think about and that is the merits, if your station doesn’t fit within one tile, of making the mosaic tiles that you import into the GIS, cover more than one of the original tiles footprint. This is just done for the sake of convenience reducing the number of files you have to copy or store. Each tile (JPEG is the format I use) apart from the graphical file which takes up megabytes on the disk, has three sidecar files, which each take at most a few kB. You are not really saving a lot of disk space by reducing the number of files, but if you are creating the sidecar files for your mosaic tiles by hand, you’ll appreciate having a smaller number of steps (I wrote a script to copy the sidecars). The main downside is the GIS will take longer to refresh larger tile sizes. Gimp has some sort of limit to its export functionality or plugin that will limit the actual size of your exported tile. When you combine multiple tiles into one bigger one, you do have to create each new tile as a rectangle and look at how a bigger area can be divided into multiple pieces. You also have to know the top left corner coordinates of the new big tile and the easiest way to do that is to put the top left corner of the new tile in the same place as one of the background tiles you got from LDS. Well, I’ll try to explain these little details more as we work through the practical steps.
One more thing is to make Gimp easy for you to use. For me, that is by setting up the right shortcut keys so that I can make things happen really quickly. The key ones for me are:
- Zoom level – this is on the View menu, “Zoom In” and “Zoom Out” in the keyboard shortcuts interface. I have these assigned to + and – on the numeric keypad, which just allows a single key to be used to zoom. I have also experimented with using the Ctrl key with the – and = keys on the main keyboard.
- Canvas rotation – this comes from the Flip and Rotate submenu on the View menu. In the keyboard shortcuts interface, within the View menu section, the actions that I find useful are “Rotate 15 degrees clockwise”, “Rotate 15 degrees counter-clockwise” and “Reset Flip & Rotate”. These are assigned to Ctrl+., Ctrl+, and Ctrl+/ in my case.
- Preview opacity – the transform tools we are going to use all have previews which we use to line up the overlays with, before pushing a button to carry out the actual transform. We have to be able to compare the background and historical images at various levels of opacity and it’s convenient to be able to go quickly up or down the levels. This is found in the Tools section of the keyboard shortcuts interface where it is programmed with the actions “Tool’s Opacity: Increase by 1” and “Tool’s Opacity: Decrease By 1”. In practice, the steps are actually 10%, which is actually quite satisfactory, and there are also actions for 10, which I would use if it hadn’t turned out that the 1 action actually turns out to be 10. I set these to be the , and . keys. The reason for using these keys is they look like left and right arrows because they are the < and > keys when shifted.
So next time we will look at actually doing the steps with Gimp, based on a real situation, this is creating a historical mosaic for Whangarei that is being done right now for NZ Rail Maps. The blog post will be written alongside the actual steps that are being done to create the historical mosaic to cover this station. This is part of Volume 1 of NZ Rail Maps that is being updated at the present time. The next post will appear later today because the schedule is actually one post a day and this post is being finished half a day late but the schedule will be caught up.