Skip to content

update ionex GIM filename convention #1337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

yuankailiu
Copy link
Contributor

@yuankailiu yuankailiu commented Mar 30, 2025

Description of proposed changes

The filename convention of JPL GIM has changed since 2023-11-27 (since GPS week 2238).

See issue: #1331

Here we use the date_str to create the file base name in the new convention is the date is in and after GPS week 2238. If the date is before GPS week 2238, we use the old file base name convention.

Reminders

  • [ v] Fix ionex to download GIM from JPL #1331
  • Pass Pre-commit check (green)
  • Pass Codacy code review (green)
  • Pass Circle CI test (green)
  • Make sure that your code follows our style. Use the other functions/files as a basis.
  • If modifying functionality, describe changes to function behavior and arguments in a comment below the function declaration.
  • If adding new functionality, add a detailed description to the documentation and/or an example.

Summary by Sourcery

Update IONEX GIM filename convention to support the new IGS filename format introduced in GPS week 2238

New Features:

  • Add support for the new IGS long product filename convention for IONEX files after November 27, 2022

Bug Fixes:

  • Fix IONEX file download and naming to match the updated JPL GIM filename convention

Enhancements:

  • Modify get_ionex_filename function to handle both old and new filename conventions
  • Add datetime_to_gps_week helper function to calculate GPS week number

Copy link

sourcery-ai bot commented Mar 30, 2025

Reviewer's Guide by Sourcery

This pull request updates the IONEX filename convention to align with the changes implemented by JPL GIM since GPS week 2238. It includes a new function to calculate the GPS week number and modifies the filename generation logic to handle both the old and new conventions. It also modifies the wget command to specify the output filename.

Sequence diagram for downloading IONEX file

sequenceDiagram
  participant User
  participant dload_ionex
  participant get_ionex_filename
  participant WebServer

  User->>dload_ionex: Calls dload_ionex(date_str, tec_dir, sol_code)
  dload_ionex->>get_ionex_filename: Calls get_ionex_filename(date_str, tec_dir, sol_code)
  get_ionex_filename-->>dload_ionex: Returns fname_src (URL of IONEX file)
  dload_ionex->>WebServer: Sends HTTP request to download fname_src
  WebServer-->>dload_ionex: Returns IONEX file
  dload_ionex-->>User: Returns
Loading

Updated class diagram for ionex filename handling

classDiagram
    class ionex {
        +dload_ionex(date_str, tec_dir, sol_code, date_fmt, print_msg)
        +get_ionex_filename(date_str, tec_dir, sol_code, date_fmt)
        +datetime_to_gps_week(dd)
    }

    class datetime {
        +strptime(date_str, date_fmt)
        +timetuple()
        +year
        +hour
        +minute
    }

    ionex -- datetime : uses
    note for ionex "Filename convention updated for GPS week >= 2238"
Loading

File-Level Changes

Change Details Files
Added a function to calculate the GPS week number from a datetime object.
  • Implemented the datetime_to_gps_week function to convert a datetime object to a GPS week number.
src/mintpy/objects/ionex.py
Updated the IONEX filename generation logic to accommodate the new naming convention introduced after GPS week 2238.
  • Modified get_ionex_filename to use the new IGS long product filename convention for dates on or after GPS week 2238.
  • Implemented conditional logic to switch between the old and new filename conventions based on the date.
  • Adjusted the URL subdirectory structure to match the new file organization.
  • Added the url_subdir variable to handle the change in directory structure.
  • Modified the local file name to remove the '.Z' suffix.
src/mintpy/objects/ionex.py
Modified the wget command to specify the output filename.
  • Added -O "{fname_dst}" to the wget command to specify the output filename.
src/mintpy/objects/ionex.py

Assessment against linked issues

Issue Objective Addressed Explanation
#1331 Fix the 404 error when downloading JPL GIM data due to changes in the filename convention.

Possibly linked issues

  • ionex to download GIM from JPL #1331: The PR fixes the issue by updating the JPL GIM filename convention to follow the IGS long product filename convention, which changed since 2023-11-27.

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!
  • Generate a plan of action for an issue: Comment @sourcery-ai plan on
    an issue to generate a plan of action for it.

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @yuankailiu - I've reviewed your changes - here's some feedback:

Overall Comments:

  • Consider extracting the filename generation logic into separate functions for the old and new naming conventions to improve readability.
  • It might be clearer to use datetime.date(2022, 11, 27) instead of dt.datetime when you only care about the date.
Here's what I looked at during the review
  • 🟢 General issues: all looks good
  • 🟢 Security: all looks good
  • 🟢 Testing: all looks good
  • 🟡 Complexity: 1 issue found
  • 🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@yunjunz
Copy link
Member

yunjunz commented Apr 1, 2025

Thank you @yuankailiu for the fix!

I am trying to test the new code, but it gives me the following authentication error:

(insar) yunjunz:~/data/S1_D062/mintpy_aoi>$ iono_tec.py timeseries.h5 -g inputs/geometryRadar.h5 

------------------------------------------------------------------------------
downloading GNSS-based TEC products in IONEX format from NASA/CDDIS ...
https://cddis.nasa.gov/Data_and_Derived_Products/GNSS/atmospheric_products.html
number of TEC files to download: 251
local TEC file directory: /Users/yunjunz/data/aux/IONEX
--------------------
DATE 1/251: 20141009
wget --continue --auth-no-challenge "https://cddis.nasa.gov/archive/gnss/products/ionex/2014/282/jplg2820.14i" -O "/Users/yunjunz/data/aux/IONEX/jplg2820.14i.Z"
--2025-04-01 15:56:31--  https://cddis.nasa.gov/archive/gnss/products/ionex/2014/282/jplg2820.14i
Resolving cddis.nasa.gov (cddis.nasa.gov)... 198.118.199.52, 2001:4d0:241a:442::52
Connecting to cddis.nasa.gov (cddis.nasa.gov)|198.118.199.52|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=gDQnv1IO0j9O2xXdwS8KMQ&response_type=code&redirect_uri=https%3A%2F%2Fcddis.nasa.gov%2Fproxyauth&state=aHR0cDovL2NkZGlzLm5hc2EuZ292L2FyY2hpdmUvZ25zcy9wcm9kdWN0cy9pb25leC8yMDE0LzI4Mi9qcGxnMjgyMC4xNGk [following]
--2025-04-01 15:56:36--  https://urs.earthdata.nasa.gov/oauth/authorize?client_id=gDQnv1IO0j9O2xXdwS8KMQ&response_type=code&redirect_uri=https%3A%2F%2Fcddis.nasa.gov%2Fproxyauth&state=aHR0cDovL2NkZGlzLm5hc2EuZ292L2FyY2hpdmUvZ25zcy9wcm9kdWN0cy9pb25leC8yMDE0LzI4Mi9qcGxnMjgyMC4xNGk
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Unknown authentication scheme.

Username/Password Authentication Failed.
gzip --force --keep --decompress /Users/yunjunz/data/aux/IONEX/jplg2820.14i.Z

I tried to add the user/password info in the wget command, but it does not seem to work. I am not sure if this is related to the CDDIS website notice below:

NASA will be performing network maintenance on Tuesday, April 1, 2025, between 9:00 AM EDT and 12:00 PM EDT (1300 to 1600 UTC). There will be a few short periods of disruption. This will result in the unavailability of the CDDIS NTRIP Caster service. If you experience problems outside this window, contact [email protected].

@yuankailiu
Copy link
Contributor Author

I just test the code again with you inquired files. It is downloading okay

Input

(insar) ykliu@shikoku:~/marmot-nobak/aqaba/topsStack/d021/hpc_isce_stack/mintpy_1/inputs$ wget --continue --auth-no-challenge "https://cddis.nasa.gov/archive/gnss/products/ionex/2014/282/jplg2820.14i" -O "./jplg2820.14i.Z"

Result is here

--2025-04-01 14:55:31--  https://cddis.nasa.gov/archive/gnss/products/ionex/2014/282/jplg2820.14i
Resolving cddis.nasa.gov (cddis.nasa.gov)... 198.118.199.52, 2001:4d0:241a:442::52
Connecting to cddis.nasa.gov (cddis.nasa.gov)|198.118.199.52|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=gDQnv1IO0j9O2xXdwS8KMQ&response_type=code&redirect_uri=https%3A%2F%2Fcddis.nasa.gov%2Fproxyauth&state=aHR0cDovL2NkZGlzLm5hc2EuZ292L2FyY2hpdmUvZ25zcy9wcm9kdWN0cy9pb25leC8yMDE0LzI4Mi9qcGxnMjgyMC4xNGk [following]
--2025-04-01 14:55:31--  https://urs.earthdata.nasa.gov/oauth/authorize?client_id=gDQnv1IO0j9O2xXdwS8KMQ&response_type=code&redirect_uri=https%3A%2F%2Fcddis.nasa.gov%2Fproxyauth&state=aHR0cDovL2NkZGlzLm5hc2EuZ292L2FyY2hpdmUvZ25zcy9wcm9kdWN0cy9pb25leC8yMDE0LzI4Mi9qcGxnMjgyMC4xNGk
Resolving urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)... 198.118.243.33, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|198.118.243.33|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://cddis.nasa.gov/proxyauth?code=uGEjvPm2QvkB_OosYCXd7HPlDVgtadWwXiEvQi-23pNJd17Gnqx4aNuf9LD-LTDNaMlhcA5xgSDvLRjS7RKuaJr2Q6VZotkjRdKYb8B0R4BqmA&state=aHR0cDovL2NkZGlzLm5hc2EuZ292L2FyY2hpdmUvZ25zcy9wcm9kdWN0cy9pb25leC8yMDE0LzI4Mi9qcGxnMjgyMC4xNGk [following]
--2025-04-01 14:55:31--  https://cddis.nasa.gov/proxyauth?code=uGEjvPm2QvkB_OosYCXd7HPlDVgtadWwXiEvQi-23pNJd17Gnqx4aNuf9LD-LTDNaMlhcA5xgSDvLRjS7RKuaJr2Q6VZotkjRdKYb8B0R4BqmA&state=aHR0cDovL2NkZGlzLm5hc2EuZ292L2FyY2hpdmUvZ25zcy9wcm9kdWN0cy9pb25leC8yMDE0LzI4Mi9qcGxnMjgyMC4xNGk
Connecting to cddis.nasa.gov (cddis.nasa.gov)|198.118.199.52|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://cddis.nasa.gov/archive/gnss/products/ionex/2014/282/jplg2820.14i [following]
URL transformed to HTTPS due to an HSTS policy
--2025-04-01 14:55:32--  https://cddis.nasa.gov/archive/gnss/products/ionex/2014/282/jplg2820.14i
Connecting to cddis.nasa.gov (cddis.nasa.gov)|198.118.199.52|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 207607 (203K) [application/x-compress]
Saving to: ‘./jplg2820.14i.Z’

./jplg2820.14i.Z                  100%[==========================================================>] 202.74K  1.07MB/s    in 0.2s

2025-04-01 14:55:32 (1.07 MB/s) - ‘./jplg2820.14i.Z’ saved [207607/207607]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ionex to download GIM from JPL
2 participants