Boring, nerdy sail stuff: How to convert SAS Planet cache files

The Red Mosque of Colombo.  If you are not sailing nerd, this is the last interesting thing you will see in this post.  Run away now.

Your friendly cruise director, Jen, has taken the day off. Unfortunately for you, you're left with Matt to fill the blog void.  And this week, instead of a relentless recap of our boring lives (especially here and now, stuck in Galle), 99.9% of you are going to get treated to something you've probably never wanted: A tutorial on how to convert cache files in SAS Planet. If you've never heard of SAS Planet before, this is probably your cue to shake your head, close the browser, and try your luck again in a week or two. If however, for the remaining 2 or 3 of you, you DO know about SAS Planet, then stick around.

UPDATED Dec 1, 2020- Check the bottom for another option in addition to what's detailed below. 

Why you might want to convert your cache structure:

SAS Planet ("SAS") is a wonderful tool for offline map and imagery viewing. It allows for virtually unlimited storage of all sorts of map types (Google Satellite, Navionics, Bing, ARC.GIS, and many others).  It also allows for integration with a GPS feed to show current position, speed and heading overlaid on any of the stored maps. 

However, as the number of your stored maps grows, the storage container for the maps, or "cache", becomes unwieldy and difficult to move, copy, or share with others. This is because the original cache structure (assuming you are using the standard "SAS.Planet" cache type) was set up on a series of nested folders to organize and contain the literal millions of small images that are stitched together to create the maps shown on the computer. Each image file is only 1-2 Kb, but a good sized SAS cache might contain 10 million of these small files. The problem arises when you go to backup or copy your cache. Computers are becoming faster and faster with passing data, but the problem here lies in the overhead associated with the Windows (or any other) operating system. 

When copying a file, you, the user, only worry about dropping and dragging, or hitting Ctrl-C then Ctrl-V. However behind the scenes the operating system is having this sort of conversation:

OS: Hey, Disk, User wants to copy file x to location y.

Disk: OK.

OS: Does that location exist?

Disk: Nope

OS: OK, create it.

Disk: OK, created.

OS: Are you ready to receive the data?

Disk: Yes

OS: Here's the data [this is where the 1.8 KB gets sent along]

Disk: OK, got it.

OS: Did you get all of it?

Disk: Hold on, let me check... Yep, it's here.

OS: OK, that's it. Goodbye.

When copying a large file, that "conversation" overhead  is almost unnoticeable.  However, when copying millions of small files, that overhead is where the computer spends most of its time relative to the small image file that it passes along. Consequently, copying or backing up a 10 GB cache might take hours, whereas a 10 GB movie file might only take a minute or two.

Luckily, more recent versions of SAS have provided support for a different cache format known as "SQLite3". Instead of storing millions of individual image files, the cache has a limited nested directory structure which ends in one or more small database files. Each database can contain thousands of images.

Let's use my Google Satellite cache as an example. When converted to a SQLite3 format, the cache went from 3,406,505 Files and 114,444 Folders to 1,681 Files and 992 Folders. That is a whopping 99.96% reduction in the number of files!  This allows the operating system to copy and back up the cache files much more efficiently.

How to convert your cache:

So now that you understand the need to convert the cache, let's get in to how to actually get it done. Within SAS, there is Cache Conversion utility.  However, it's rather manual and takes a bit of setup to get it to work properly. That said, it's not terribly complicated and the payoffs are large, so it's worth doing.

The first thing to understand is that the SAS Cache Conversion utility only allows you to convert one map source at a time. So, depending on the number of map types in your cache, you may need to repeat this process several times. Also, to prevent confusion, it's best not to have SAS attempt to download more new files while you're are doing this process, so for now set your data source to "Cache Only".

Step 1- Navigate to the main SAS directory.  This is easily found by opening Windows File Explorer and navigating to your SAS Planet folder.  In this main directory you should see a sub-directory called "cache". This is where your existing cache files reside. Create a new directory called "cache_sqlite". At this point you should now have a "cache" sub-directory and a " cache_sqlite" sub-directory. You only need to create this "cache_sqlite" directory once.  This is the new directory where all of our new SQLite3 map specific caches will reside. 

Proper set up of the directory structures is the key to a successful conversion

 

Step 2- Take a look in your "cache" folder   Within "cache" you should see many subfolders with names like "sat", "both", "navionics", "vesat", etc. Each one of these sub folders represents a different map type that you have downloaded at some point.  Most names are pretty self explanatory, but a few are more cryptic.  Here are a few you may not immediately recognize:

•  "sat" --> Google satellite imagery
• "Both" --> Google Hybrid images (this is the overlay that shows town names, streets, businesses, etc.)
• "vesat" --> Bing satellite imagery

Determine which map type you want to convert. Let's use the Google satellite imagery as an example. Drill down into the sub-directories under that map type ("sat") until you reach the final level that contains the image files. Make a note the type of image file that the map type uses (possible to be .jpg, .png, .bmp or even others). Google satellite imagery uses .jpg. Come back up to main SAS directory.

Step 3 - We now need to make a new sub-folder in our "cache_sqlite" directory  that is the EXACT same name as the sub-folder in the "cache" directory that we want to convert. So if we are going to convert Google Satellite imagery, we need to create a "sat" sub-folder within our "cache_sqlite" folder.  The sub-folder names must be identical or the SAS program won't recognize the new location.

This completes the "set up" required, and we are now ready to start the SAS Cache Conversion utility.

Conversion utility screen

 

 Step 4 - Within the SAS program, navigate to Operations -> Cache Manager

This will bring up a bring up a window where we need to enter some info for the conversion to work properly.

·         In "Source Cache", leave the first drop-down set to "Folder".

·         In "Path", click on the ellipses button to the right of the blank field and navigate to and select the source folder, namely your "sat" subfolder within the main cache folder.

·         Set the "Format" drop down to "Sas.Planet"

·         Set the "Extension" dropdown to ".jpg" (this is the file type you discovered when you drilled down into the cache in Step 2 of the set up process)

·         Ignore all the other fields in the top half, or "Source Cache" area.

·         In the "Dest Cache"", leave the first drop-down set to "Folder".

·         In "Path", click on the ellipses button to the right of the blank field and navigate to and select the destination folder, namely your NEW "sat" subfolder within the newly created "cache_sqlite" folder.

·         Set the "Format" drop down to "SQLite3"

·         Ignore all the other fields in the bottom half, or "Dest Cache" area.

You are now ready  to run the conversion. Click the "Start" button. A progress window will pop up. It will take a bit of time for the system  to determine what it needs to process, but after a few minutes (possibly longer if your cache is really big), the progress window will start showing how many files it has processed.  Just for planning purposes, a large cache of roughly 6 million files (58 Gb) took almost 13 hours to process. However this was on a machine limited to USB2 and the cache was on an external disk. I would assume that the ability to use USB3 would have sped things up considerably.

 

Setting up SAS to use the new SQLite cache type.

Step 5 - Once the process is complete, if you want to check that everything went smoothly, you need to tell SAS to start using the new cache. This can be done as follows:

·         Navigate within the SAS program to Settings -> Options

·         Select the "Cache" tab

·         Make sure the "SQLite3 cache folder" is set to "cache_sqlite"

·         Change the "Default cache type" dropdown to "SQLite3"

·         Click "Apply"

You should now be able to see your newly created cache.  At this point you should ONLY be able to see the one mapset you just converted (make sure the selected basemap is properly set).  All the other map types will be grey, as that data has not been converted yet. 

Step 6 - To finish the conversion for other map types, simply repeat steps 2-4 as needed.

Phew! Did you really read all that anyway? If so, I gotta suggest you find yourself a hobby.

 

UPDATE Dec 1, 2020:

Another option for this process is to use the program called "Sat2Chart" developed by Paul Higgins.  The program is primarily used to create .kap and mbtile charts from imagery sources like Google earth and Sas Planet.  However, Paul has also recently included a utility that automates the cache conversion process detailed in my post.  The utility is called "Convert_Cache" and is found within the "Sat2Chart" program.  You can download the program here: http://gdayii.ca/Downloads/  

To run Sat2Chart, you will also need to download and install OoRexx 5.0.0, which is also available from the download location given above.

When the Convert_Cache window opens, there is a "Tutorial" button you can click  that walks you through the operation and setup of the utility.

Galle (Sri Lanka) - Still Paused

 

Peace Pagoda

During most of the eight years we have lived on Perry, we have been at anchor rather than at a dock. It's also been somewhat rare that we have stayed in the same spot for more than a few weeks.  And on the occasions that we haven't been moving around, we are usually hauled out and working hard to get some necessary work finished or we are on a road trip.

View of the harbor from the Peace Pagoda

This time is different (a common theme for most people these last few months) and we find ourselves on a dock, in one place, and without  major boat jobs that otherwise fully consume the day. Normally this would be the perfect opportunity for one of the aforementioned road trips, but that's off the table at the moment as well. We don't feel secure leaving our boat in the harbor here for very long due to the line chafing, fender popping surges that can come through the harbor.

Sushi made from locally caught tuna (bought, not caught, by us)

So what to do? Perry still finds ways to demand our attention: A cleat that needs rebedding after six months of strain, the discovery of some  small termites that set up shop in the engine room (good thing we don't have much wood), new faucets in the bathrooms, and some leaks seeping in from the torrential rains of the Northwest monsoon. But for the most part, we fill our days with walks around the Galle area, including the Peace Pagoda pictured at top and the rocking surf at the beaches nearby.

Our big day out was a trip to the U.S. Embassy in Colombo to renew a passport. We took the train, which was a convenient and scenic way to travel. Colombo is a rambling and bustling city, and deserving of more than just the few hours spent there running errands. Hopefully, as the seasons change and the swells diminish, the surge will decrease and we will be able to spend more time away from Galle.  

As there have been no reported cases of community transmission of Covid-19 for several months, things seem more normal. Not many people in Galle wear masks anymore except in some stores where they're required (although the train police did walk the aisles ensuring that passengers wear masks). The security guards taking your temperature before entering the supermarket no longer have the look of urgent concern in their eyes, replaced now by the bored indifference of having to repeat the same task hundreds of times each day. 

Unfortunately, even as normal life in some aspects resumes, it's clear that the country is hurting from lack of tourists and the pause in the economy during the lock down.

Paratha making--yum!

Conrad has been creating small aquariums (aquaria?) with a multitude of aquatic plants, a Betta fish and some colorful shrimp. Many of these items are much cheaper here than in the United States, though finding a place to house them is a challenge on a boat. He easily rattles off scientific names and fish/shrimp/plant care instructions. He (with help from Matt) has also used some of the wiring and mechanical knowledge he has gained over the years to put some LEDs in one of the tanks and create a filter (from an old olive jar, some sponges and other various bits and bobs) for the other. It's a great supplement to his homeschooling.

Yala National Park - Look what we (caution: bad pun ahead) spotted!

 

Yala National Park borders the Indian Ocean

We took our first road trip in Sri Lanka, even if it was only for two days. What's been stopping us? First, we needed to be allowed out of the port. Check. Second, we had to figure out how to leave the boat safely.

Sea Eagle

 

This second requirement turned out to be mostly a mental barrier borne of past chafed lines, popped fenders, and a marred hull paint job. At times big surges would come into the harbor and toss the boat around. But the forecast predicted calm water and a friend on another boat here generously agreed to keep an eye on our boat. So we felt comfortable enough to leave Perry for a couple days and one night to take a trip to Yala National Park.

 

Water buffalo

Yala is one of the top attractions in Sri Lanka for good reasons. It's a bit smaller than 400 square miles and is filled with wildlife, including one of the highest densities of wild leopards in the world.

 

Asian elephant enjoying water lilies

 

Yala is one of the best places to have the chance to see a leopard in the wild, but you still have to be pretty lucky to spot one. You can tell that finding leopards is the main focus for the drivers/guides. They all keep in touch with each other on their phones and when there is a sighting, it's a mad dash, with tires screeching and dirt flying.

There were a dozen-plus jeeps at our leopard sightings but on the weekends there can be a few hundred jeeps in the park

 

We did an afternoon safari on our first visit and a morning safari the next day. In the chaotic zooming around at the news of our first leopard, our driver went too fast over a big bump. Conrad and I both flew up and hit our heads on the roof of the jeep (I managed to get 3 pretty good sized knots in various spots on my head). The next day, one of the guides/trackers from another jeep said that people have actually cracked their skulls in the past and suggested that we should tell our driver not to speed (because there is a speed limit in the park).

Spoiler alert: We saw a leopard.

We felt bad that the leopard was seemingly chased non-stop by the the jeeps, but he seemed completely unfazed by the hullabaloo. I'm not sure if that's because the dozen or so vehicles were actually a much smaller number than usual or because he is just used to it. The apparent harassment is a double-edged sword because the tours and visitor dollars (the entrance fee on the website is literally in U.S. dollars) help ensure that the park will continue to provide protection to the animals. But maybe one of these days a leopard will go all Sean Penn on its 'paparazzi'. If so, there will be someone there with a camera to record it.
 

We were lucky enough to see a leopard on both our safaris. Our driver said it had been a week since he had seen one, so he was excited too (we think it was the same leopard both days because it was in the same general area). 

On the second safari (which came with an unpopular 5 a.m. pick-up), a leopard was spotted sleeping in a tree. Promising something more interesting than just a hanging tail and some half-obscured fur, our guide suggested to wait. We waited about an hour for the leopard to come down out of the tree (did you know that teenage boys can be impatient about waiting around doing nothing? and in this case, their mom too...). Finally, the gamble paid off as it got warmer and the sun started to creep onto the leopard. He got up to get a drink from the grimy-looking pond nearby, before strolling off into the underbrush.

 

We were glad that we waited, but who knows what else we missed. At least my headache from literally hitting the roof is mostly gone now.

Malabar Pied Hornbills

Kingfisher

Purple Heron

We also spotted this Spotted Deer

Mom and baby

Carpenter bee - about the size of a pecan

Our hotel had its own wildlife. Matt was surprised (he may have shrieked a little) as he was resting on the bed when a tree frog dropped next to him from somewhere. We figured out later that they like to sit in the air-conditioning unit above the bed. After we turned the AC on and the unit started cooling down, they were less happy inside and wandered out. And then--gravity. 

That evening, one fell on my foot while I was sleeping (I definitely shrieked). I'm glad I decided to move my head to the other side of the bed. The kids loved the frogs ('they're so cute') but we all had to be extra careful not to squish them when going to the bathroom in the dark and it is disconcerting to turn on the lights and see five of them staring at you from various places in and around the toilet.

Freedom!

 

The lighthouse at
Galle Fort

After four and a half months of being confined to the Port in Galle, we have been granted shore passes. Although we still don't have visas, we can now leave the port on our own. This means we can finally go shopping on our own and actually see some of Sri Lanka.

Dutch architecture
in Galle Fort

We have been taking full advantage of our new-found freedom. The boys have not even complained about the length of our walks (the first day we walked 6 miles in the pouring rain and 8 miles the second day).

 

Galle Fort, apparently usually bustling, was fairly empty and many of the businesses and eateries were closed because of the lack of tourists. That doesn't mean we were completely unable to find an outlet for our restaurant and ice-cream needs.  Unfortunately, our freedom has come with a literal cost: we managed to spend way more money in our first few days out than when we were trapped in the port with almost nothing to buy.

We are still getting used to being able to buy groceries when we need them, rather than sprinting through the store trying to buy enough for the next three weeks.

In addition to old buildings,
there are some huge mature trees

We also enjoyed some 'fun' time for the first time in months: A nice beach is just around the corner from the port and the boys were able to get in the ocean. Conrad even managed to surf some decent waves. 

These fishermen have challenging
surf to navigate when going out

For now, we plan to stay in Sri Lanka while we're allowed, while we wait for countries to open up along the route back to the U.S. 

We visited the Mahamodara Sea
 Turtle Hatchery Centre

A Shikra (small bird of prey) with
a broken wing that Conrad rescued.

Our friend Ruwan cutting
some King Coconuts for us

Ruwan and family

Waffles, anyone?

When the swell is up, the waves splash
over the rocks and onto this twenty-plus foot hill

'Waffles', because we can't seem to make up our minds about what to do next--stay in Sri Lanka or go to the Seychelles. Of course, waffles are a bit of a sore subject on our boat since I nixed the waffle-maker when we moved aboard and Matt still hasn't quite gotten over it.

Matt put out an anchor to keep us pulled away from
the dock, so we don't bang into the wall when the surge is up

On the one hand, we have been in Sri Lanka for over four months and still don't have visas. We have been out of the port area only a handful of times, always with an entourage and only to get groceries and other essentials. We don't know when or if we will get visas and are going a bit stir crazy. Also, our new paint job has taken a beating and we cringe every time the boat jerks against the dock lines. We could possibly go up to Trincomalee and anchor, but not without visas.

We had a visit from a special guest. Reverend
Mendis lives in Chicago and happened to be in Sri Lanka
 now. Friends of my parents contacted him about us.

The Seychelles (where several of our cruising friends are now) has granted us access to enter their country (and the Maldives would allow us to stop on the way), but we would be fighting wind, current and weather a good chunk of the way. Once there, we would need South Africa and, preferably Madagascar, to open up for us at the right time (basically October). With the virus on the upswing in those countries and their borders closed, the situation is far from settled.

So we go back and forth on what to do, depending on the day and how bad the surge is. In the meantime, we have gotten better at shopping for three weeks at a time. Each of the four of us gets a shopping cart and then Matt gets to use his packing skills to make everything fit 'just so' in the fridge and freezer.

My last few pictures are sort of a metaphor (on some days) for our outlook on the world situation right now: rubbish. Everyone probably feels that way occasionally these days, right? Unfortunately, the downwind location of the harbor makes it the final resting point for tons of garbage that flows in from other places. It's a never-ending battle.