Announcement

Collapse

TGC Tour - Congratulations!

Congrats to @Joe_S (GROSS & NET) winning the TGC1 2020 SUMMER TOUR - John Deere Classic!

Join this weeks TGC1 2020 SUMMER TOUR - The Memorial Tournament with 2 rounds at Valhalla Louisville (Pro Tee): https://golfsimulatorforum.com/forum...ial-tournament
See more
See less

LiDAR Data Help

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • LiDAR Data Help

    I'm very new to this community and would like to recreate my home course in the game. I've watched several course design tutorials and came to the conclusion that using LiDAR is really the way to go for accurate course design. I've gone through and splined the first hole of my home course, located in Kennewick, WA, in OSM . I wanted to test things out on that first hole and I'm struggling to find any LiDAR data out there for my area.

    Is this just a case of LiDAR data not currently existing for this area? If so, do any of you course designers have any other suggestions on getting topographical data that can be imported? Any help or guidance is much appreciated!

  • #2
    I did a quick check and there seems to be quite a bit of data. Go to

    https://viewer.nationalmap.gov/basic/

    and watch Chad's video on how to use the website:

    https://www.twitch.tv/videos/414113375

    the sticky thread of Chad's Tools on this subforum (TGC2019 design) is a treasure trove of info. It just takes patience to read through it.

    Comment


    • #3
      Originally posted by s0707 View Post
      I did a quick check and there seems to be quite a bit of data. Go to

      https://viewer.nationalmap.gov/basic/

      and watch Chad's video on how to use the website:

      https://www.twitch.tv/videos/414113375

      the sticky thread of Chad's Tools on this subforum (TGC2019 design) is a treasure trove of info. It just takes patience to read through it.

      That's the website I go to, but when I select the area with Box/Point and try to find product under "Elevation Source Data (3DEP), it returns zero results. In Chad's video he says you must select that option. Were you able to pull up the source data for anywhere in Kennewick?

      Comment


      • #4
        I did a quick check as well and am finding lots of results. I am not sure if your course is within this square that I have highlighted, but I find 80 results.

        Make sure you have Lidar/Point Cloud checked in the Datasets section. I would recommend finding your course and getting the exact geographical coordinates in order to narrow it down more specifically.

        Shout if you are still having trouble.

        Click image for larger version

Name:	Kennewick.JPG
Views:	136
Size:	121.1 KB
ID:	277196 Click image for larger version

Name:	Kennewick1.JPG
Views:	115
Size:	128.9 KB
ID:	277197




        Comment


        • #5
          I see two very similar options on the left pane of the web page: "Elevation Products (3DEP)" and "Elevation Source Data (3DEP) - Lidar, IfSAR" -- select the latter one.

          Comment


          • #6
            My course is in the highlighted box shown in this image. Search gives me zero hits. It seems I may just have to wait until lidar data becomes available for our area. It looks like most of the lidar data we've got is all along the river and it's wildlife refuges.
            Attached Files

            Comment


            • #7
              Originally posted by s0707 View Post
              I see two very similar options on the left pane of the web page: "Elevation Products (3DEP)" and "Elevation Source Data (3DEP) - Lidar, IfSAR" -- select the latter one.
              I'm able to find data from 3DEP-lidar but when I go to download it the download link just fails. Am I just being a dunce?
              Link: ftp://rockyftp.cr.usgs.gov/vdelivery...7_LAS_2019.laz

              Comment


              • onlevel7
                onlevel7 commented
                Editing a comment
                I was able to download the .laz file that you linked and processed the lidar in chads tool but it doesn't look like data matches the golf course at all.

              • binisha
                binisha commented
                Editing a comment
                And now it downloaded for me. Weird. Time to see if I can figure other things I was doing wrong out.

            • #8
              Hmmm... Yeah, after zooming in on your golf course I don't get any lidar results on the National Map web page. opentopography.org shows that there's lidar data to the East of you, but not on your golf course. It's possible the lidar data is coming up the pipeline in a few years. I read somewhere that the USGS wants to lidar the whole country by 2023 (IIRC).

              However, I went to this site:

              https://prd-tnm.s3.amazonaws.com/Lid...r/index.html#/

              And after clicking on "define area of interest" and zooming in on your golf course, some DEM (digital elevation model) popped up. There's 3-meter data, which you might want to use if you can't find lidar data. It would be a matter of converting the DEM to lidar and classifying the data as ground so Chad's tool can use it. I just did this for a 9-hole course in my hometown in Mexico, but it's a little involved. The DEM data I got was 30-meter apart, and it gives me a general idea of elevations, much better than my memory of 40+ years ago. If you want to explore this option, I can tell you what I did.

              Comment


              • onlevel7
                onlevel7 commented
                Editing a comment
                That would be awesome if you could give me a rundown of how to convert. I did download one of the available options and it was geotiff format. I searched on whether or not it was possible to convert that to .las but came up empty.

            • #9
              Can you do python? That's one of the steps I had to use to do the classification of the data as ground points, and that may be the hardest one. Though you might find a tool different than python to do the classification.

              Anyway, I used the 'demzip.exe' tool from

              http://lastools.org/

              This converted my raster "ellipsoidal alos" .tif file into a laz file. IIRC I used the options "-longlat -wgs84" or "-longlat -nad83". Make sure to specify these after specifying your input file "-i" and your output file "-o".

              This produced a .laz file (i.e. compressed lidar-format data), but the data points have classification=0 which means "not yet classified". Chad's tool doesn't like this. So, the next step is to set the data points to have classification=2, which means ground. Every lidar data point has the X, Y, Z information (coordinates+altitude) plus additional info -- classification is one of that additional info. classification 0=created-never-classified, 1=unclassified, 2=ground, 3=low vegetation, 4=medium vegetation, etc.

              lastools.org has the 'lasground.exe' tool that is supposed to help you classify the data points and you can use "-not_airborne" option so that it classifies everything as ground. The problem is lasground.exe doesn't like long/lat for coordinates, and I could never find a way to have demzip.exe create the laz file with the "UTM" system -- lasground kept telling me that it was long/lat data and that the classification would most likely fail -- it kept copying the input file straight to the output file without changing it.

              Enter python and laspy... laspy is a python library that you can use to manipulate LAS files. The file produced by demzip is a LAZ file (compressed LAS data), so you can uncompress it with lastool's laszip.exe. That's an easy step.

              What did I do with laspy? I ran some manual commands (can be put together into a script) that opened the input file, copied the points data array into an new output array, setting the classification=2 in every record of the output array; then created an output file with a copy of the header from the input file, set the output data array, and closed the file, which writes the data to disk. I then used that file in Chad's tool, which auto-magically accepted the data.

              One thing I didn't get right, but Chad's tool came to the rescue, is the epsg code. I got a code from http://epsg.io, but Chad's tool said it was incorrect and it would use instead another epsg which ended up producing the right result. Chad's too is quite amazing. Thank you, Chad.

              How to get python and laspy? If you're not sure and are adventurous, I can write more instructions.

              Comment


              • onlevel7
                onlevel7 commented
                Editing a comment
                I mean I'm a complete noob when it comes to python, but I can teach myself programming languages fairly easily. I installed python on my rig and I went to lastools.org and downloaded demzip.exe, which actually just downloaded lastools.zip and extracting it doesn't have any executable files in it. I would need some pretty heavy hand holding to get me started and I wouldn't expect you to go through all of that.

            • #10
              I had to do the same thing for a lot of my courses here in Japan because I could only find DEM data at 5 meters. I ended up getting a free trial of GlobalMapper including the LiDAR tool and with that I could convert DEM data to LiDAR and classify it to get it into Chad's Tool. It was a bit of a hassle to get the trial license but once I got that, I could convert it fairly easily. I ended up downloading and converting data for about 50 courses here in Japan for any that I may ever want to build. The data is not perfect, some turned out really well while others seemed really off, but it was at least a good starting point.

              Comment


              • onlevel7
                onlevel7 commented
                Editing a comment
                So I've got the trial version of GlobalMapper and my .dem project loaded. How do I go about converting that to .las?

                EDIT: So I was able to convert it to LAS, but now how do I go about classifying it? You mentioned in another thread that you were able to do it relatively simply in Global Mapper. Thanks!
                Last edited by onlevel7; 07-01-2020, 11:56 PM.

              • onlevel7
                onlevel7 commented
                Editing a comment
                Well shit, now it's saying I've exported the maximum number of files of this type with a trial license. You said you were able to do 50 courses? It let me extract once and said I hit my limit

            • #11
              onlevel7 , here's the python script that will allow you to force the classification of all points to be ground. The forum doesn't let me upload the script, but you shoud be able to copy and paste it into notepad or any other text editor you like. Make sure to save the file with the .py extension. Here's the script, which I called classify_dem_las_as_ground.py:

              ######## script begins after this line; no need to copy this line but it'll turn into a python comment line if you accidentally copy it.
              import numpy
              from laspy.file import File
              import sys

              def print_usage():
              ....print( "Usage:" )
              ....print()
              ....print( " <path-to-your-python-executable> classify_dem_las_as_ground.py <input-filepath> <outout-filepath>" )
              ....print()
              ....print( "For example:" )
              ....print()
              ....print( " c:\python38-32\python.exe classify_dem_las_as_ground.py input_data.las output_data.las" )
              ....print()
              ....print( "Also, note this script cannot process .laz files. They have to be uncompressed to an .las file first" )

              if len( sys.argv ) < 3:
              ....print( "Please provide input and output file" )
              ....print_usage()
              ....sys.exit( 1 )

              input_filepath = sys.argv[1]
              output_filepath = sys.argv[2]

              print( "Processsing: input file = '%s'; output_file = '%s'" % ( input_filepath, output_filepath ) )

              print( "Opening input file %s" % input_filepath )
              input_file = File( input_filepath, mode='r' )
              if not input_file:
              ....print( "Error: unable to open input file %s. Somehow laspy could not parse the file" % input_filepath )
              ....sys.exit( 1 )

              # create the output array
              output_point_records = numpy.array( input_file.points )

              for output_point in output_point_records:
              ....output_point[0][5] = 2

              print( "set classification of all data points to 2 (ground)" )

              # open output file and write the modified data. Providing the input file header
              # makes a copy of it into the output file's header.
              output_file = File( output_filepath, mode="w", header=input_file.header )
              output_file.points = output_point_records
              output_file.close()

              print( "Wrote output file %s" % output_filepath )
              sys.exit( 0 )


              ##################### end of the script

              Note that there are 4 spaces indenting instructions inside 'if', 'for' loop, and 'def' constructs. Those are important for python otherwise the script breaks. See EDIT below... the 4 dots must be substituted with 4 spaces.

              To make things easier you can copy the script to the folder where you have the .LAS file to classify. Then open a DOS command prompt window (you can search in Win7 Start, or Win10 Cortana), and change to the drive and do a cd to the folder where you have your .LAS file. For example

              D:
              cd \las_files

              This is an example of running the script

              C:\python38-32\python.exe classify_dem_las_as_ground.py input_las_file.las output_las_file.las

              C:\python38-32 is the folder where you installed your python.

              You may not have the laspy or perhaps even the numpy libraries available. You can do the following to download them:

              C:\python38-32\python-exe -m pip install laspy numpy

              and so on. If you installed Chad's python distribution and ran 'C:\python38-32\python.exe -m pip install -r requirements.txt' then you should have laspy and numpy available as Chad's python code distribution downloads them.

              Good luck! Hope it works for you.


              EDIT !!!!! - Aargh... The forum posting code is removing the spaces that indent instructions inside 'if', 'for', and 'def' constructs.... I've updated it with 4 dots instead of 4 spaces. After copying and pasting the text to your text editor, make sure to substitute the 4 dots with 4 spaces... There have to be exactly 4 spaces that indent instructions inside the constructs. Otherwise the script will break.
              Last edited by s0707; 07-02-2020, 06:23 PM.

              Comment


              • #12
                @s0707

                OK, I think I'm getting close here. I was able to successfully install both laspy and numpy, and ran the script. It is spitting out ValueError:mmap length is too large. Any ideas?

                EDIT: Just figured out that I downloaded the 32 bit version of python. After a complete uninstall and fresh install of 64, it has now output a .las file for me. We'll see how she goes!

                EDIT #2: Followed step by step with the twitch stream and after processing Lidar (took a lot longer to process than it did on his stream) it brought up the window to draw a box around my course. Issue is the window was completely white. No split screen, just a fully white window titled Select Course Boundaries.

                I'm going to try following step by step with the course he does in the stream to see if I can get past that point.
                Last edited by onlevel7; 07-03-2020, 01:01 AM.

                Comment


                • #13
                  That could be the EPSG code. I don't know how to figure that one. In my case, for the golf course in my hometown in Mexico, I went to

                  http://epsg.io

                  And clicked on the map and got WGS84 4326. I tried 4326 and Chad's tool said it was incorrect, but it was going to try 32615, and that turned out to work well.

                  You could try those numbers, or try a search for "washington" in epsg.io, which returns a large number of entries

                  Comment


                  • #14
                    I finally got it to show an image on both sides, but the mapquest image just barely clips off my course. Is there a way to force the long and lat that chad's tool pulls?

                    Comment


                    • #15
                      For example, the image attached shows where my course is - just barely outside of the aerial photo. Below is the text I get when I process lidar.

                      Processing: CanyonClass.las
                      EPSG is not map projection, skipping: 4269
                      Could not find metadata for CanyonClass.las.
                      File is likely in Geographic Coordinates (Lat/Lon Degrees). You probably want to find alternate files, but we will try to project this for you.
                      For center coordinates: (46.12499999999979, -119.12500000000286):
                      Overwriting projection with EPSG:32611
                      Unit in use is 1.0
                      Proj4 : pyproj.Proj('+units=m +proj=latlong +datum=WGS84 ', preserve_units=True)
                      Warning: Data has different projection, re-projecting coordinates. This may take some time.
                      Generating lidar intensity image
                      No lidar intensity found, using elevation instead
                      5.03% visualizing lidar
                      9.91% visualizing lidar
                      14.8% visualizing lidar
                      19.64% visualizing lidar
                      24.55% visualizing lidar
                      29.54% visualizing lidar
                      34.6% visualizing lidar
                      39.59% visualizing lidar
                      44.68% visualizing lidar
                      49.75% visualizing lidar
                      54.87% visualizing lidar
                      59.61% visualizing lidar
                      64.7% visualizing lidar
                      69.69% visualizing lidar
                      74.7% visualizing lidar
                      79.76% visualizing lidar
                      84.68% visualizing lidar
                      89.65% visualizing lidar
                      94.7% visualizing lidar
                      99.38% visualizing lidar
                      Adding golf features to lidar data
                      OpenStreetMap Overpass query: (node(45.99997882436863,-119.2511562627481,46.25015032332811,-118.99132445067389);way(45.99997882436863,-119.2511562627481,46.25015032332811,-118.99132445067389);out;
                      Mapquest Image URL Request: https://open.mapquestapi.com/staticm...ue&format=png& center=46.12514478798985,-119.12094371204233&type=hyb&zoom=14&size=1023,1500
                      Attached Files

                      Comment

                      Working...
                      X