L2 matchups with PACE data -- RRS
- Create a plan for files to use
pc.plan() - Print the plan to check it
print(plan.summary()) - Do the plan and get matchups
pc.matchup(plan, geometry="swath")
Prerequisite -- Login to EarthData
The examples here use NASA EarthData and you need to have an account with EarthData. Make sure you can login.
<earthaccess.auth.Auth at 0x7fd3d7713d10>
Here are the level 2 datasets
import earthaccess
results = earthaccess.search_datasets(instrument="oci")
short_names = [
item.summary()["short-name"]
for item in results
if "L2" in item.summary()["short-name"]
]
print(short_names)
['PACE_OCI_L2_UVAI_UAA_NRT', 'PACE_OCI_L2_UVAI_UAA', 'PACE_OCI_L2_AER_UAA_NRT', 'PACE_OCI_L2_AER_UAA', 'PACE_OCI_L2_AOP_NRT', 'PACE_OCI_L2_AOP', 'PACE_OCI_L2_CLOUD_MASK_NRT', 'PACE_OCI_L2_CLOUD_MASK', 'PACE_OCI_L2_CLOUD_NRT', 'PACE_OCI_L2_CLOUD', 'PACE_OCI_L2_LANDVI_NRT', 'PACE_OCI_L2_LANDVI', 'PACE_OCI_L2_BGC_NRT', 'PACE_OCI_L2_BGC', 'PACE_OCI_L2_IOP_NRT', 'PACE_OCI_L2_IOP', 'PACE_OCI_L2_PAR_NRT', 'PACE_OCI_L2_PAR', 'PACE_OCI_L2_SFREFL_NRT', 'PACE_OCI_L2_SFREFL', 'PACE_OCI_L2_TRGAS_NRT', 'PACE_OCI_L2_TRGAS']
Load some points
import pandas as pd
url = (
"https://raw.githubusercontent.com/"
"fish-pace/point-collocation/main/"
"examples/fixtures/points.csv"
)
df_points = pd.read_csv(url)
df_points.head()
| lat | lon | date | |
|---|---|---|---|
| 0 | 27.3835 | -82.7375 | 2024-06-13 |
| 1 | 27.1190 | -82.7125 | 2024-06-14 |
| 2 | 26.9435 | -82.8170 | 2024-06-14 |
| 3 | 26.6875 | -82.8065 | 2024-06-14 |
| 4 | 26.6675 | -82.6455 | 2024-06-14 |
Get a plan for matchups for 1st 100 points from PACE data
%%time
import point_collocation as pc
plan = pc.plan(
df_points[0:100], # -82.7375, 27.3835
data_source="earthaccess",
source_kwargs={
"short_name": "PACE_OCI_L2_AOP",
},
time_buffer="12h"
)
CPU times: user 610 ms, sys: 89.3 ms, total: 700 ms
Wall time: 3.25 s
Plan: 100 points → 24 unique granule(s)
Points with 0 matches : 0
Points with >1 matches: 20
Time buffer: 0 days 12:00:00
First 5 point(s):
[0] lat=27.3835, lon=-82.7375, time=2024-06-13 12:00:00: 2 match(es)
→ https://obdaac-tea.earthdatacloud.nasa.gov/ob-cumulus-prod-public/PACE_OCI.20240613T171620.L2.OC_AOP.V3_1.nc
→ https://obdaac-tea.earthdatacloud.nasa.gov/ob-cumulus-prod-public/PACE_OCI.20240613T184939.L2.OC_AOP.V3_1.nc
[1] lat=27.1190, lon=-82.7125, time=2024-06-14 12:00:00: 1 match(es)
→ https://obdaac-tea.earthdatacloud.nasa.gov/ob-cumulus-prod-public/PACE_OCI.20240614T175104.L2.OC_AOP.V3_1.nc
[2] lat=26.9435, lon=-82.8170, time=2024-06-14 12:00:00: 1 match(es)
→ https://obdaac-tea.earthdatacloud.nasa.gov/ob-cumulus-prod-public/PACE_OCI.20240614T175104.L2.OC_AOP.V3_1.nc
[3] lat=26.6875, lon=-82.8065, time=2024-06-14 12:00:00: 1 match(es)
→ https://obdaac-tea.earthdatacloud.nasa.gov/ob-cumulus-prod-public/PACE_OCI.20240614T175104.L2.OC_AOP.V3_1.nc
[4] lat=26.6675, lon=-82.6455, time=2024-06-14 12:00:00: 1 match(es)
→ https://obdaac-tea.earthdatacloud.nasa.gov/ob-cumulus-prod-public/PACE_OCI.20240614T175104.L2.OC_AOP.V3_1.nc
geometry : 'swath'
open_method : 'datatree-merge'
Dimensions : {'number_of_bands': 286, 'number_of_reflective_bands': 286, 'wavelength_3d': 172, 'number_of_lines': 1710, 'pixels_per_line': 1272}
Variables : ['wavelength', 'vcal_gain', 'vcal_offset', 'F0', 'aw', 'bbw', 'k_oz', 'k_no2', 'Tau_r', 'year', 'day', 'msec', 'time', 'detnum', 'mside', 'slon', 'clon', 'elon', 'slat', 'clat', 'elat', 'csol_z', 'Rrs', 'Rrs_unc', 'aot_865', 'angstrom', 'avw', 'nflh', 'l2_flags', 'longitude', 'latitude', 'tilt']
Geolocation: ('longitude', 'latitude') — lon dims=('number_of_lines', 'pixels_per_line'), lat dims=('number_of_lines', 'pixels_per_line')
DataTree groups (detail):
/
Dimensions : {}
Variables : []
/sensor_band_parameters
Dimensions : {'number_of_bands': 286, 'number_of_reflective_bands': 286, 'wavelength_3d': 172}
Variables : ['wavelength', 'vcal_gain', 'vcal_offset', 'F0', 'aw', 'bbw', 'k_oz', 'k_no2', 'Tau_r']
/scan_line_attributes
Dimensions : {'number_of_lines': 1710}
Variables : ['year', 'day', 'msec', 'time', 'detnum', 'mside', 'slon', 'clon', 'elon', 'slat', 'clat', 'elat', 'csol_z']
/geophysical_data
Dimensions : {'number_of_lines': 1710, 'pixels_per_line': 1272, 'wavelength_3d': 172}
Variables : ['Rrs', 'Rrs_unc', 'aot_865', 'angstrom', 'avw', 'nflh', 'l2_flags']
/navigation_data
Dimensions : {'number_of_lines': 1710, 'pixels_per_line': 1272}
Variables : ['longitude', 'latitude', 'tilt']
/processing_control
Dimensions : {}
Variables : []
/processing_control/input_parameters
Dimensions : {}
Variables : []
/processing_control/flag_percentages
Dimensions : {}
Variables : []
Get the matchups using that plan
pc.matchup() needs to know that you are working with swath not gridded data. It has no reliable way to know that and instead of guessing, you need to tell it you are working with swath (level 2) data. batch_size is the number of files that pc.matchup() works with at once. A smaller batch size uses less RAM, and may or may not be slower. This example uses about 2Gb of RAM and takes 3 minutes.
granules 1-1 of 24 processed, 1 points matched, 00:00:07
granules 2-2 of 24 processed, 1 points matched, 00:00:10
granules 3-3 of 24 processed, 9 points matched, 00:00:24
granules 4-4 of 24 processed, 8 points matched, 00:00:37
granules 5-5 of 24 processed, 8 points matched, 00:00:51
granules 6-6 of 24 processed, 5 points matched, 00:00:58
granules 7-7 of 24 processed, 3 points matched, 00:01:03
granules 8-8 of 24 processed, 10 points matched, 00:01:18
granules 9-9 of 24 processed, 1 points matched, 00:01:23
granules 10-10 of 24 processed, 1 points matched, 00:01:25
granules 11-11 of 24 processed, 2 points matched, 00:01:29
granules 12-12 of 24 processed, 1 points matched, 00:01:33
granules 13-13 of 24 processed, 1 points matched, 00:01:35
granules 14-14 of 24 processed, 1 points matched, 00:01:39
granules 15-15 of 24 processed, 9 points matched, 00:01:51
granules 16-16 of 24 processed, 8 points matched, 00:02:06
granules 17-17 of 24 processed, 8 points matched, 00:02:17
granules 18-18 of 24 processed, 6 points matched, 00:02:27
granules 19-19 of 24 processed, 3 points matched, 00:02:35
granules 20-20 of 24 processed, 9 points matched, 00:02:49
granules 21-21 of 24 processed, 9 points matched, 00:03:03
granules 22-22 of 24 processed, 9 points matched, 00:03:18
granules 23-23 of 24 processed, 6 points matched, 00:03:30
granules 24-24 of 24 processed, 1 points matched, 00:03:36
CPU times: user 2min 43s, sys: 6.29 s, total: 2min 49s
Wall time: 3min 37s
| lat | lon | time | granule_id | Rrs_346 | Rrs_348 | Rrs_351 | Rrs_353 | Rrs_356 | Rrs_358 | ... | Rrs_706 | Rrs_707 | Rrs_708 | Rrs_709 | Rrs_711 | Rrs_712 | Rrs_713 | Rrs_714 | Rrs_717 | Rrs_719 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 27.3835 | -82.7375 | 2024-06-13 12:00:00 | https://obdaac-tea.earthdatacloud.nasa.gov/ob-... | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 1 | 27.3835 | -82.7375 | 2024-06-13 12:00:00 | https://obdaac-tea.earthdatacloud.nasa.gov/ob-... | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 2 | 27.1190 | -82.7125 | 2024-06-14 12:00:00 | https://obdaac-tea.earthdatacloud.nasa.gov/ob-... | 0.012990 | 0.012946 | 0.013148 | 0.013172 | 0.012918 | 0.012968 | ... | 0.000238 | 0.000228 | 0.000198 | 0.000194 | 0.000186 | 0.000172 | 0.000152 | 0.000122 | 0.000108 | 0.000094 |
| 3 | 26.9435 | -82.8170 | 2024-06-14 12:00:00 | https://obdaac-tea.earthdatacloud.nasa.gov/ob-... | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 4 | 26.6875 | -82.8065 | 2024-06-14 12:00:00 | https://obdaac-tea.earthdatacloud.nasa.gov/ob-... | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 115 | 25.0900 | -82.2570 | 2024-10-23 12:00:00 | https://obdaac-tea.earthdatacloud.nasa.gov/ob-... | 0.004512 | 0.004754 | 0.004986 | 0.005262 | 0.005426 | 0.005730 | ... | 0.000362 | 0.000344 | 0.000314 | 0.000298 | 0.000272 | 0.000244 | 0.000218 | 0.000192 | 0.000162 | 0.000142 |
| 116 | 25.1450 | -82.4125 | 2024-10-23 12:00:00 | https://obdaac-tea.earthdatacloud.nasa.gov/ob-... | 0.007960 | 0.008424 | 0.008790 | 0.009128 | 0.009304 | 0.009436 | ... | 0.000852 | 0.000830 | 0.000792 | 0.000764 | 0.000732 | 0.000690 | 0.000650 | 0.000616 | 0.000506 | 0.000408 |
| 117 | 25.3800 | -82.4740 | 2024-10-23 12:00:00 | https://obdaac-tea.earthdatacloud.nasa.gov/ob-... | 0.005936 | 0.006560 | 0.007292 | 0.007608 | 0.007666 | 0.007882 | ... | 0.000618 | 0.000596 | 0.000572 | 0.000544 | 0.000514 | 0.000470 | 0.000440 | 0.000394 | 0.000294 | 0.000222 |
| 118 | 25.4045 | -82.5885 | 2024-10-23 12:00:00 | https://obdaac-tea.earthdatacloud.nasa.gov/ob-... | 0.005908 | 0.006522 | 0.006776 | 0.006764 | 0.006640 | 0.006914 | ... | 0.000526 | 0.000496 | 0.000460 | 0.000430 | 0.000404 | 0.000380 | 0.000364 | 0.000336 | 0.000282 | 0.000224 |
| 119 | 25.4910 | -82.7660 | 2024-10-24 12:00:00 | https://obdaac-tea.earthdatacloud.nasa.gov/ob-... | 0.007694 | 0.008076 | 0.008246 | 0.008106 | 0.008110 | 0.008530 | ... | 0.000516 | 0.000486 | 0.000466 | 0.000440 | 0.000416 | 0.000408 | 0.000378 | 0.000348 | 0.000264 | 0.000194 |
120 rows × 176 columns