CDROM.txt 6.7    HOW TO BUILD APRS MAPS FROM THE USGS CD ROM      19 Nov 94

New in 6.7:  Brought README.100 into this file to reduce the # of txt files

NOTE:  Special guidance for mapmakers is in README\MOREMAPS.txt

BEGIN:  All information on using the original 2,000,000:1 USGS CD ROM has
been moved to the end of this file, since by now, (fall 94) almost all
of the US population centers have been made into APRS maps at this level of
detail.  This FILE now begins to discuss the use of 100,000:1 USGS CD ROMS
that contain data on roads all the way down to your street.  Because of the
tremendous amount of data available in this data base, USGS sells the data on
a set of 13 CD's.  To see which CD covers your area, hit 1 on your APRS screen
to get to the whole USA map.  Then press MAPS-ALT command to see the CDROM.map
file which is under the USA map.  This display shows you the areas of the 13
different ROMS.  My yellow lines are crude, but the data in the files does
truncate exactly at the state lines.  To obtain one of these CD ROMS, send $32
for each ROM to the USGS Earth Science Info Center, 507 National Center,
Reston, VA 22092. 703-648-6045 or call 1-800-USA-MAPS.
BE SURE TO SPECIFY THE OPTIONAL FORMAT!!!!!!!!!!!!!!!! (vice standard)

WARNING:  Since both Ken, W7KKE and myself are continuing to improve these
programs, and keep modifying each others code, the following programs are
identical, except for the latest mods installed by each writer.  Look at the
latest file date, to see which one is the most recent:

     Ken's:   CONV100.bas   =   Bob's 100toGRF.bas  (Currently, use KEN's)
              DEFRAG.bas    =         JOIN.bas

JUST BECAUSE WE HAVE NOW WRITTEN ROUTINES TO HELP YOU MAKE MAPS FROM BOTH THE
2,000,000 and 100,000 SCALE USGS CD ROMs, DO NOT GIVE UP ON MAKING YOUR OWN
MAPS USING ANY OF THE OTHER METHODS.  THE 2,000,000 SCALE CD IS ONLY GOOD FOR
64 MILE MAPS because THEY DONT HAVE ENOUGH DETAIL TO MAKE SMALLER MAPS, and
they HAVE TOO MANY POINTS TO MAKE MAPS MUCH LARGER!  The 100,000 scale maps
generate so many points that an APRS map of city streets can be no larger than
about 1 mile in dense neighborhoods!

BACKGROUND:  There are 3 problems with the USGS data that causes excessive
points in an APRS map.  First, lines begin and end at EACH intersection.  For
am interstate, for example, you end up with dozens of short road segments
wasting 3 point pairs per exit.  Second, these short segments are not always
consecutive in the file, making it extremely time consuming to try to join
them.  Third, the USGS data guarantees a point every .01 inch at the original
maps scale (1/3 mile for the 64 mile maps) whether it is needed or not.  A 10
mile long straight road is stored as 30 points in USGS vice 2 points in APRS.
At the 100,000:1 scale, .01 inch is only 100 feet!


VERY FINE DETAIL USING THE USGS 100,000:1 DETAIL CD ROMS

    W7KKE wrote a program called CONV100 that I have modified to 100toGRF.bas
which can take data from the 100,000 scale CD ROMS.   It generates so many
roads in an urban environment, that just a 1 square mile map has over 5000
points.  I have upped the APRS maximum limits to accomodate up to 2999 points
but you should avoid making these large size maps unless you absolutely
need every street, because the large size also means APRS takes longer to
load maps!  Since we are vehicle tracking for radio purposes, a whole map
1 mile square is only one one-thousandth of the potential coverage area of
one APRS digipeater, so you can see that the effort to make these maps is
not worth the time in my opinion. (unless you are making the detail for
a commercial application and are willing to spend the time!)  Here is how
to do it:

1.  Run the USGS EXTRACT program and follow its instructions to output the
appropriate map data for the 15 x 15 minute area that you are interested in.
These files will be of the form XXXRDSnn for ROADS and xxxHYFnn for hydro
where the nn is for multiple files to cover the area.  In my urban area,
each of the ROAD files covers about a 2 mile area.

2.  Run 100toGRF.bas and once you give it one of the above file names, it will
automatically generate a series of 10 output files of the form PREnRD.GRF for
n different ROAD files and two other files preWB.grf and preST.grf for
waterbodies and streams.  100toGRF.bas outputs these files in the same format
as the original 2,000,000 that MAPFIX knows how to read.  The PRE is a 3 or 4
letter prefix that you specify for this series to keep it separate from other
efforts.  So far, the number 4, 9 and 10 maps in my area have been empty...
but the pre8RD.GRF file is the largest by a factor of ten!  It contains all
of the neighborhood streets.

3.  Run MAPFIX and use the B command to establish the boundary of the area
that you want to develop.  Choose a scale of 2400, 3000, 3600, 4800 or so
that will cover the total area you eventually want covered.  For city streets,
you should use at least 2400 or more.  Next use the alt-USGS command to
bring in the first of the above files.  Then do the alt-SMOOTH and the
alt-JOIN commands to minimize the map.  Then SAVE IT WITH A NEW NAME.  Then
repeat this step for all the maps that were output from the 100toGRF.bas
program.  After you have saved each of these, you will now have almost a dozen
maps each containing a different feature, but in APRS format.  WARNING!:
NEVER do a second ALT-U USGS BUILD on the same map, because it will clear
all current points and start a new map from scratch.

4.  Next, run my JOIN.bas program, or Ken's better DEFRAG.bas program on EACH
of the files generated by MAPFIX in the above step 3.  These programs go
through the files over and over again, trying to join together all of the
disjointed road segments (of the same class) generated by the USGS data base.
Hopefully you may see a 20% or more reduction in points.  Re-save each of
these reduced maps under a new name so that you do not distroy your originals
if there is a problem.

NOTE:  Be sure to also review the steps at the end of this file on how MAPFIX
is used to read the 2,000,000:1 CD ROMS.  Since the 2,000,000:1 process was
developed first and is the basis for MAPFIX routines, you might find some
helpful details or clarifying information.  Ken Swaggart, W7KKE, wrote the
conversion routines to put the 100,000:1 data into the 2,000,000:1 format.
His instructions are in README.100. (and are similar to these)

4.  Now, combine these maps into one APRS map, using the MAPFIX alt-IMPORT
command.  This command will bring just one, or all of the map features of one
file into the current file.  In an urban area, the level 8 road file will be
larger than all the others combined.  You can omit it to allow your map to
cover a larger area or include it for the finest detail available.

Good luck.  I am just beginnning to play with this level of detail, and to
this point, it is far more practical to use a digitizer!  For more detail on
the use of MAPFIX with CD ROMs, also read the 2,000,000:1 section below,
since it evolved and was the basis for APRS maps before we began messing with
the 100,000:1 data.

TO SEE KEN's ORIGINAL EXPLAINATION ON HOW TO DO THIS, SEE HIS README.100 FILE.


PROCEDURES FOR 2,000,000:1 CD ROM THAT COVERS THE WHOLE US

1.  Use USGS XTRACT.EXE program to extract the CD ROM data in the GRAPHIC
    format for your area from one of the dozen or so major USGS areas.  Then
    select up to 6 files for extraction onto your HARD disk, one for each of
    the data types, BOUNDARIES, ROADS, WATER BODIES, STREAMS, RAILROADS, and
    HYPSOGRAPHY (continental divide).  These files will take about 5 MBytes.
    To save HD space, and also to end up with the minimum of points, you will
    probably only use the ROADS, WATERBODIES, and STREAMS files.  Note the
    filename assigned by XTRACT.

2.  Run MAPFIX.BAS program and load the best existing APRS map that covers
    the desired area.  This map is only used to help you locate the area of
    interest.  Next, move the cursor to the center of interest and select
    a SCALE using the BOUNDARY (B) command.  This command will show you the
    maximum map possible for a given scale in Points/per degree.  I have found
    600 to be about right.  450 in sparse states and 300 in UTAH and NEW Mex.
    If you use any less, to get larger maps, you will end up with far too many
    points.  I used 450 for the SouthCarolina map, and it took me 6 hours to
    get the resulting 3000 points down below 1500!  In the East, I used 900
    for a 35 mile map around Philidelphia.  Making maps any smaller is probably
    not worth the effort, since there will be so few roads in it..

3.  When you have the B command set to the desired number of Pixels-per-degree,
    press the alt-U key.  MAPFIX will allow you to round the ORIGIN values, to
    reasonable nearly whole numbers or fractions of a degree so that later
    manual operations with a map and ruler will have round numbers to deal
    with.

4.  Next APRS asks you for the type of file (RD, ST, WB, etc) and then the
    exact path to the XTRACTed USGS data.  This path should include the first
    part ot the filename up to the (_) underline that separates the RD or WB
    etc final two characters.  MAPFIX will then scan the entire indicated file
    for all points within the selected border.  While converting the points to
    APRS format, MAPFIX will also compare the slope of each new point with the
    previous point and ignore the point if it has approximately the same slope
    (within 1.2)  The total number of points will be from 2000 to 4000.

4.  Next MAPFIX has a SMOOTHER command which will run another slope comparison
    process.  You can select the SMOOTHER factor.  A factor of 1 will only
    throw out points that are exactly the same slope (few if any).  A factor of
    1.2 (best) will throw out points whose slope differs by less than 1.2.
    Bigger values will throw out  more points (usually including some necessary
    ones).  I never use a value greater than 1.5.  To test the smoothing
    algorithm, run the TEST SMOOTHER command which will mark every point that
    would be eliminated with a red dot, and every point that will be kept with
    a yellow dot; but without actually eliminating any points.  Look at the map
    closely and see if you can accept the decisions of the SMOOTHER.  Select
    your chosen factor and run SMOOTHER. (alt-S)

5.  Next use the alt-JOIN command.  This will cause MAPFIX to scan through
    the map looking for consecutive map features that end and then start
    at the same point.  These two features will be joined into one longer
    continuous feature.  At this point, you can save the map, quit MAPFIX
    and run MYdefrag.bas which extends this same philosophy by comparing
    each feature to EVERY other feature (not just consecutive) still trying
    to match end points of one feature with the start of another.  This
    process can usually eliminate another couple of hundred points.  The
    result is saved in MAPTEMP.map.

6.  Now you have the smallest APRS map that can be done automatically.
    Reload MAPFIX, reload the MAPTEMP.map (or other optional name) and Move
    the MAP POINTER to the beginning FEATURE using the RESET command and
    use the gray +/- keys to cycle through EVERY point in the file.  Use the
    alt-D key to delete all unnecessary points.  Look for complete duplicates
    of some lines too.  This should get you down to below 2000 or so.

7.  Notice that you only have one category of features so far (usually roads).
    Unless you require county lines I recommend that you ignore them.  The
    USGS BOUNDARIES file contains even more points than the ROADS file!  AND
    the SMOOTHER routine does not work well, because property lines are very
    jagged, unlike smooth curves in roads.  The same thing goes for WATER
    BODIES and STREAMS.

10. To allow you to add some detail from these other USGS files, MAPFIX has
    the USGS OVERLAY feature which will overlay the USGS data on the MAPFIX
    screen without doing anything.  Then you can use the NEW-FEATURE and ADD
    commands to draw APRS features over the top of these temporary points.
    The philosopy here is that 96% of amateur interest is on roads, and you
    can sure burn up points on waterways and county lines.  For rivers, about
    the only points I use are where the water goes under a bridge.  Most folks
    have no idea where a river is out in the boonies (and could care less).
    Of course, if you are making a map for boats with GPS, then start the
    process with the WATERBODIES file and add just enough roads for them to
    find the boat ramps..
       You will first need to overlay on the 64 mile scale to see where things
    are, but then zoom into the 8 mile scale at least and do the overlay again
    to get enough resolution for drawing a particular feature.  You can use the
    NEW, ADD and MOVE commands, but any other commands will redraw the map and
    wipe-out the overlay.  I have added the Ctrl-R command to toggle the map
    REDRAW process on and off.  This may help you to add lines without destroying
    the USGS overlay.  You can do the overlay again and again.

WARNING!:  NEVER do a second ALT-U USGS BUILD, because it will clear memory
    and start from scratch.


CAUTION:  Do not expect too much from this CD ROM.  ALthough the 2,000,000:1
    USGS database seems to have resolution down to 1/3 rd of a mile, remember
    that that 1/3rd of a mile was actually 1/100th of an inch on the original
    map!  Your 64 mile section was only a two inch square!!!!!  AND THE DATA
    WAS PROBABLY DERIVED BY A HIGH SCHOOL SUMMER STUDENT using a mouse!

      I have found this data to be very crude, and although the maps lines
    look smooth, you are getting about the same quality as if you were
    looking at a state highway map through a magnifying glass!  If you want
    good detail, use MAPFIX and a digitizer on a much better original map,
    such as a 100,000:1 or even 24,000:1 (which is the standard USGS 7.5
    minute map).  There you can get down to your driveway!

IT STILL TAKES AN EXCESSIVE AMOUNT OF MANUAL EFFORT TO REDUCE THESE MBYTES
OF USGS DATA DOWN TO A reasonably quick loading 15K APRS MAP!  IT TAKES ME 4
to 12 HOURS PER MAP!  By now (Fall 94) I have done most of the US to the
state highway level of detail.  IF YOU WANT ME TO DO A SPECIAL MAP OF YOUR
AREA, to the resolution in the 2,000,000:1 ROM, SEND $$$ or convince me it is
in the general best interest of all HAMS.  THAT ALONE SHOULD KEEP YOU
INTERESTED IN DOING YOUR OWN.  HI HI

FOR THE 100,000:1, I will only do these maps commercially, because the 3 days
worth of effort only results in a map that covers a few miles.  Only a few
HAMS will even live inside one of these small areas.  If you are a MAP-ology
expert, or an ace programmer and can help us figure out a way to better
automate the process, then we would love to make the whole process easier
for everyone.  Contact Ken Swaggart, W7KKE in Calif, or myself.

*****************************************************************************
README.100   (as of 2/13/94)

                   1:100,000 CD ROM Map Making

                            W7KKE
                         Ken Swaggart
               w7kke @ ka6eyh.#nocal.ca.usa

The 1:100,000 line segment maps contained on the USGS CD ROM uses
a different format than the 1:2,000,000 CD ROM maps. My first
utility, CONV100.BAS, converts the 1:100,000 map "Optional Format"
extracted files to the .GRF file format which can be read by the APRS
MAPFIX.BAS program. When you import these map files into MAPFIX you
will notice they are extremely fragmented into small line segments
which are not adjacent. The second utility, DEFRAG.BAS takes the saved
MAPFIX file and defragments it.

Here are the step by step instructions:

  1. Create a directory on your hard drive where you will be work
on your maps. Stick my programs in this directory and copy
MAPFIX.BAS from the APRS directory into this working map directory.
I use C:\APRSMAPS as my working map directory.

  2. Extract "Transportation" and "Hydrography"  for your area of
interest from the 1:100,000 CD ROM using the extraction program
which comes on the CD ROM. I zoom down to the 15' scale on the USGS
extraction program before I extract the map file. Copy these files
into your working map directory.

  3. Run CONV100.BAS.  The program will extract each feature, such
as Interstates, US Highways, etc., as separate files with the
filename extension of .GRF.

  4. Run MAPFIX.BAS. Import the first converted .GRF file  by using
the @U command in MAPFIX.BAS. After the file is imported you and on
your MAPFIX screen use the Join command (@J) to join adjacent line
segments for this feature. Save this feature as its own .MAP file
and repeat the process for each map feature you created.

  NOTE: Make sure you use the same origin and scale each time you
        import the .GRF file using the @U command.

  5. Run DEFRAG.BAS to de-fragment the .MAP files you have created.

     If DEFRAG.BAS crashes, you can recover most of your defragmented
     segments. Look for SCRATCH1.TMP and SCRATCH2.TMP. Rename the
     larger of these two and delete the remaining .TMP file.

  6. Use a text editor, such as DOS's EDIT, and examine your
defragmented .MAP files and tweak as you see fit. You may find that
some line segments consist of only one point - edit those out.

  7. Run MAPFIX.BAS again and import your defragmented .MAP file.
Check it over for any obvious errors, correct those, then run the
smoother function (@SMOOTH) to pare down the .MAP file.

  Note: I generally do not run smoother on the water boundaries and
streams, but hand edit to reduce the points. Another option which
works well is to use the U command to overlay the water boundaries
and streams, then manually draw in the rivers with the @new and @A
commands. This results if far fewer points than the automatic
method.

 8. Save your smoothed .MAP file and import your next feature and
repeat the process. (I only run the smoother on each separate
feature file so I can control what gets smoothed.) Sometimes I have
to bump the smoother factor way up to pare down the points to fit
the total point limit for APRS.

  9. Use a text editor, such as DOS's EDIT, to assemble the final
map from all the defragmented and smoothed .MAP files created for
each map feature type.

  10. Run MAPFIX again and import the resulting complete .MAP file
with all your features. If it is too large, which it probably is,
delete points until you have a map which will fit within APRS
limits.

  - There is some small errors (less than 1/4 mile) toward the north east
and north west sections of the map. This is due to the conversion scheme
I use to get from the UTM grid to lat/long. The two grids do not line up
exactly. When you join two adjacent 15' maps, you will see that the roads
are off by about 100 meters. Use mapfix to tweak the lines so they join.


Please send me any program enhancements/fixes you make to either
program. I'm sure there are a lot better ways to defragment the
.MAP files - there MUST be!

PS - watch out for "Streams" - when I extract the ones around here
the resulting .MAP has thousands of points!

