Boring, nerdy sail stuff: How to convert SAS Planet cache files

The Red Mosque of Colombo.  If you are not sailing nerd, this is the last interesting thing you will see in this post.  Run away now.

Your friendly cruise director, Jen, has taken the day off. Unfortunately for you, you're left with Matt to fill the blog void.  And this week, instead of a relentless recap of our boring lives (especially here and now, stuck in Galle), 99.9% of you are going to get treated to something you've probably never wanted: A tutorial on how to convert cache files in SAS Planet. If you've never heard of SAS Planet before, this is probably your cue to shake your head, close the browser, and try your luck again in a week or two. If however, for the remaining 2 or 3 of you, you DO know about SAS Planet, then stick around.

UPDATED Dec 1, 2020- Check the bottom for another option in addition to what's detailed below. 

Why you might want to convert your cache structure:

SAS Planet ("SAS") is a wonderful tool for offline map and imagery viewing. It allows for virtually unlimited storage of all sorts of map types (Google Satellite, Navionics, Bing, ARC.GIS, and many others).  It also allows for integration with a GPS feed to show current position, speed and heading overlaid on any of the stored maps. 

However, as the number of your stored maps grows, the storage container for the maps, or "cache", becomes unwieldy and difficult to move, copy, or share with others. This is because the original cache structure (assuming you are using the standard "SAS.Planet" cache type) was set up on a series of nested folders to organize and contain the literal millions of small images that are stitched together to create the maps shown on the computer. Each image file is only 1-2 Kb, but a good sized SAS cache might contain 10 million of these small files. The problem arises when you go to backup or copy your cache. Computers are becoming faster and faster with passing data, but the problem here lies in the overhead associated with the Windows (or any other) operating system. 

When copying a file, you, the user, only worry about dropping and dragging, or hitting Ctrl-C then Ctrl-V. However behind the scenes the operating system is having this sort of conversation:

OS: Hey, Disk, User wants to copy file x to location y.

Disk: OK.

OS: Does that location exist?

Disk: Nope

OS: OK, create it.

Disk: OK, created.

OS: Are you ready to receive the data?

Disk: Yes

OS: Here's the data [this is where the 1.8 KB gets sent along]

Disk: OK, got it.

OS: Did you get all of it?

Disk: Hold on, let me check... Yep, it's here.

OS: OK, that's it. Goodbye.

When copying a large file, that "conversation" overhead  is almost unnoticeable.  However, when copying millions of small files, that overhead is where the computer spends most of its time relative to the small image file that it passes along. Consequently, copying or backing up a 10 GB cache might take hours, whereas a 10 GB movie file might only take a minute or two.

Luckily, more recent versions of SAS have provided support for a different cache format known as "SQLite3". Instead of storing millions of individual image files, the cache has a limited nested directory structure which ends in one or more small database files. Each database can contain thousands of images.

Let's use my Google Satellite cache as an example. When converted to a SQLite3 format, the cache went from 3,406,505 Files and 114,444 Folders to 1,681 Files and 992 Folders. That is a whopping 99.96% reduction in the number of files!  This allows the operating system to copy and back up the cache files much more efficiently.

How to convert your cache:

So now that you understand the need to convert the cache, let's get in to how to actually get it done. Within SAS, there is Cache Conversion utility.  However, it's rather manual and takes a bit of setup to get it to work properly. That said, it's not terribly complicated and the payoffs are large, so it's worth doing.

The first thing to understand is that the SAS Cache Conversion utility only allows you to convert one map source at a time. So, depending on the number of map types in your cache, you may need to repeat this process several times. Also, to prevent confusion, it's best not to have SAS attempt to download more new files while you're are doing this process, so for now set your data source to "Cache Only".

Step 1- Navigate to the main SAS directory.  This is easily found by opening Windows File Explorer and navigating to your SAS Planet folder.  In this main directory you should see a sub-directory called "cache". This is where your existing cache files reside. Create a new directory called "cache_sqlite". At this point you should now have a "cache" sub-directory and a " cache_sqlite" sub-directory. You only need to create this "cache_sqlite" directory once.  This is the new directory where all of our new SQLite3 map specific caches will reside. 

Proper set up of the directory structures is the key to a successful conversion

 

Step 2- Take a look in your "cache" folder   Within "cache" you should see many subfolders with names like "sat", "both", "navionics", "vesat", etc. Each one of these sub folders represents a different map type that you have downloaded at some point.  Most names are pretty self explanatory, but a few are more cryptic.  Here are a few you may not immediately recognize:

•  "sat" --> Google satellite imagery
• "Both" --> Google Hybrid images (this is the overlay that shows town names, streets, businesses, etc.)
• "vesat" --> Bing satellite imagery

Determine which map type you want to convert. Let's use the Google satellite imagery as an example. Drill down into the sub-directories under that map type ("sat") until you reach the final level that contains the image files. Make a note the type of image file that the map type uses (possible to be .jpg, .png, .bmp or even others). Google satellite imagery uses .jpg. Come back up to main SAS directory.

Step 3 - We now need to make a new sub-folder in our "cache_sqlite" directory  that is the EXACT same name as the sub-folder in the "cache" directory that we want to convert. So if we are going to convert Google Satellite imagery, we need to create a "sat" sub-folder within our "cache_sqlite" folder.  The sub-folder names must be identical or the SAS program won't recognize the new location.

This completes the "set up" required, and we are now ready to start the SAS Cache Conversion utility.

Conversion utility screen

 

 Step 4 - Within the SAS program, navigate to Operations -> Cache Manager

This will bring up a bring up a window where we need to enter some info for the conversion to work properly.

·         In "Source Cache", leave the first drop-down set to "Folder".

·         In "Path", click on the ellipses button to the right of the blank field and navigate to and select the source folder, namely your "sat" subfolder within the main cache folder.

·         Set the "Format" drop down to "Sas.Planet"

·         Set the "Extension" dropdown to ".jpg" (this is the file type you discovered when you drilled down into the cache in Step 2 of the set up process)

·         Ignore all the other fields in the top half, or "Source Cache" area.

·         In the "Dest Cache"", leave the first drop-down set to "Folder".

·         In "Path", click on the ellipses button to the right of the blank field and navigate to and select the destination folder, namely your NEW "sat" subfolder within the newly created "cache_sqlite" folder.

·         Set the "Format" drop down to "SQLite3"

·         Ignore all the other fields in the bottom half, or "Dest Cache" area.

You are now ready  to run the conversion. Click the "Start" button. A progress window will pop up. It will take a bit of time for the system  to determine what it needs to process, but after a few minutes (possibly longer if your cache is really big), the progress window will start showing how many files it has processed.  Just for planning purposes, a large cache of roughly 6 million files (58 Gb) took almost 13 hours to process. However this was on a machine limited to USB2 and the cache was on an external disk. I would assume that the ability to use USB3 would have sped things up considerably.

 

Setting up SAS to use the new SQLite cache type.

Step 5 - Once the process is complete, if you want to check that everything went smoothly, you need to tell SAS to start using the new cache. This can be done as follows:

·         Navigate within the SAS program to Settings -> Options

·         Select the "Cache" tab

·         Make sure the "SQLite3 cache folder" is set to "cache_sqlite"

·         Change the "Default cache type" dropdown to "SQLite3"

·         Click "Apply"

You should now be able to see your newly created cache.  At this point you should ONLY be able to see the one mapset you just converted (make sure the selected basemap is properly set).  All the other map types will be grey, as that data has not been converted yet. 

Step 6 - To finish the conversion for other map types, simply repeat steps 2-4 as needed.

Phew! Did you really read all that anyway? If so, I gotta suggest you find yourself a hobby.

 

UPDATE Dec 1, 2020:

Another option for this process is to use the program called "Sat2Chart" developed by Paul Higgins.  The program is primarily used to create .kap and mbtile charts from imagery sources like Google earth and Sas Planet.  However, Paul has also recently included a utility that automates the cache conversion process detailed in my post.  The utility is called "Convert_Cache" and is found within the "Sat2Chart" program.  You can download the program here: http://gdayii.ca/Downloads/  

To run Sat2Chart, you will also need to download and install OoRexx 5.0.0, which is also available from the download location given above.

When the Convert_Cache window opens, there is a "Tutorial" button you can click  that walks you through the operation and setup of the utility.

Galle (Sri Lanka) - Still Paused

 

Peace Pagoda

During most of the eight years we have lived on Perry, we have been at anchor rather than at a dock. It's also been somewhat rare that we have stayed in the same spot for more than a few weeks.  And on the occasions that we haven't been moving around, we are usually hauled out and working hard to get some necessary work finished or we are on a road trip.

View of the harbor from the Peace Pagoda

This time is different (a common theme for most people these last few months) and we find ourselves on a dock, in one place, and without  major boat jobs that otherwise fully consume the day. Normally this would be the perfect opportunity for one of the aforementioned road trips, but that's off the table at the moment as well. We don't feel secure leaving our boat in the harbor here for very long due to the line chafing, fender popping surges that can come through the harbor.

Sushi made from locally caught tuna (bought, not caught, by us)

So what to do? Perry still finds ways to demand our attention: A cleat that needs rebedding after six months of strain, the discovery of some  small termites that set up shop in the engine room (good thing we don't have much wood), new faucets in the bathrooms, and some leaks seeping in from the torrential rains of the Northwest monsoon. But for the most part, we fill our days with walks around the Galle area, including the Peace Pagoda pictured at top and the rocking surf at the beaches nearby.

Our big day out was a trip to the U.S. Embassy in Colombo to renew a passport. We took the train, which was a convenient and scenic way to travel. Colombo is a rambling and bustling city, and deserving of more than just the few hours spent there running errands. Hopefully, as the seasons change and the swells diminish, the surge will decrease and we will be able to spend more time away from Galle.  

As there have been no reported cases of community transmission of Covid-19 for several months, things seem more normal. Not many people in Galle wear masks anymore except in some stores where they're required (although the train police did walk the aisles ensuring that passengers wear masks). The security guards taking your temperature before entering the supermarket no longer have the look of urgent concern in their eyes, replaced now by the bored indifference of having to repeat the same task hundreds of times each day. 

Unfortunately, even as normal life in some aspects resumes, it's clear that the country is hurting from lack of tourists and the pause in the economy during the lock down.

Paratha making--yum!

Conrad has been creating small aquariums (aquaria?) with a multitude of aquatic plants, a Betta fish and some colorful shrimp. Many of these items are much cheaper here than in the United States, though finding a place to house them is a challenge on a boat. He easily rattles off scientific names and fish/shrimp/plant care instructions. He (with help from Matt) has also used some of the wiring and mechanical knowledge he has gained over the years to put some LEDs in one of the tanks and create a filter (from an old olive jar, some sponges and other various bits and bobs) for the other. It's a great supplement to his homeschooling.