AssertionError when running ACwater/Polymer on recent EnMAP dataset
When I apply ACwater/Polymer to a recent EnMAP dataset (ENMAP01-____L1B-DT0000005210_20221108T071035Z_005_V010110_20221121T092646Z), ACwater crashes with an AssertionError
. However, the traceback also contains an OSError: [Errno 9] Bad file descriptor
. This needs to be fixed with highest priority, otherwise ACwater is not usable within EnPT. Here is the complete log:
test_controller.py::Test_EnPT_Controller_DLR_testdata::test_run_all_processors EnPT Controller received the following configuration:
{'general_opts': {'CPUs': 1,
'average_elevation': 0,
'create_logfile': True,
'disable_progress_bars': False,
'drop_bad_bands': True,
'is_dummy_dataformat': False,
'log_level': 'DEBUG',
'n_lines_to_append': 50,
'path_dem': '',
'path_l1b_enmap_image': '/home/gfz-fe/scheffler/python/EnPT/tests/data/EnMAP_Level_1B/destriped_ENMAP01-____L1B-DT0000005210_20221108T071035Z_005_V010110_20221121T092646Z.zip',
'path_l1b_enmap_image_gapfill': '',
'path_l1b_snr_model': '/home/gfz-fe/scheffler/python/EnPT/enpt/resources/EnMAP_Sensor/EnMAP_Level_1B_SNR.zip',
'working_dir': None},
'output': {'output_dir': '/home/gfz-fe/scheffler/python/EnPT/enpt/../tests/data/test_outputs',
'output_format': 'ENVI',
'output_interleave': 'pixel'},
'processors': {'atmospheric_correction': {'auto_download_ecmwf': False,
'blocksize': 100,
'enable_ac': True,
'mode_ac': 'water',
'polymer_root': '',
'scale_factor_boa_ref': 10000,
'threads': 1},
'dead_pixel': {'algorithm': 'spectral',
'interp_method_spatial': 'linear',
'interp_method_spectral': 'linear',
'run_processor': True},
'geometry': {'enable_absolute_coreg': False,
'enable_keystone_correction': False,
'enable_vnir_swir_coreg': False,
'path_reference_image': ''},
'orthorectification': {'resamp_alg': 'nearest',
'target_coord_grid': {'x': array([ 0, 30]),
'y': array([ 0, 30])},
'target_epsg': None,
'target_projection_type': 'UTM',
'vswir_overlap_algorithm': 'swir_only'},
'smile': {'run_processor': False},
'toa_ref': {'path_earthSunDist': '/home/gfz-fe/scheffler/python/EnPT/enpt/resources/earth_sun_distance/Earth_Sun_distances_per_day_edited__1980_2030.csv',
'path_solar_irr': '/home/gfz-fe/scheffler/python/EnPT/enpt/resources/solar_irradiance/SUNp1fontenla__350-2500nm_@0.1nm_converted.txt',
'scale_factor_toa_ref': 10000}}}
2022/12/08 17:50:23: Reading metadata for VNIR detector...
2022/12/08 17:50:24: Calculating solar irradiance...
2022/12/08 17:50:24: Reading metadata for SWIR detector...
2022/12/08 17:50:24: Calculating solar irradiance...
2022/12/08 17:50:24: Reading image masks in VNIR sensor geometry.
2022/12/08 17:50:27: Converting DN values to radiance [mW/m^2/sr/nm] for VNIR detector...
2022/12/08 17:50:29: Converting DN values to radiance [mW/m^2/sr/nm] for SWIR detector...
2022/12/08 17:50:33: Computing SNR from VNIR TOA radiance.
2022/12/08 17:50:41: Computing SNR from SWIR TOA radiance.
2022/12/08 17:50:46: Computing VNIR geolayer...
2022/12/08 17:50:49: Computing SWIR geolayer...
2022/12/08 17:52:23: Correcting dead pixels of VNIR detector...
Used algorithm: linear interpolation in the spectral domain
2022/12/08 17:52:26: Percentage of defective pixels: 0.15
2022/12/08 17:52:26: Percentage of pixels with a defect in any band: 12.60
2022/12/08 17:52:28: Correcting dead pixels of SWIR detector...
Used algorithm: linear interpolation in the spectral domain
2022/12/08 17:52:33: Percentage of defective pixels: 1.14
2022/12/08 17:52:33: Percentage of pixels with a defect in any band: 64.60
2022/12/08 17:52:41: No DEM for the VNIR detector provided. Falling back to an average elevation of 0 meters.
2022/12/08 17:52:41: No DEM for the SWIR detector provided. Falling back to an average elevation of 0 meters.
2022/12/08 17:52:45: Transforming the 'mask_landwater' attribute from VNIR to SWIR sensor geometry.
2022/12/08 17:52:51: Starting atmospheric correction for VNIR and SWIR detector in 'water' mode. Source radiometric unit code is 'TOARad'.
2022/12/08 17:52:51: Running atmospheric correction in 'water' mode, i.e., ACWater/Polymer is applied to water surfaces only. Note that land surfaces will NOT be included in the EnMAP L2A product.
merge
2022/12/08 17:52:51: Running polymer...
2022/12/08 17:52:51: Currently only tested with ERA5 data, i.e. set True for "auto_download_ecmwf"
Initializing ENMAP product of size (1024, 1000)
2022/12/08 17:53:12: Landmask is EnMAP L1B
Starting processing at 2022-12-08 17:53:12.782496
2022/12/08 17:53:15: The atmospheric correction for water surfaces based on ACwater/Polymer failed (issue tracker at https://gitlab.awi.de/phytooptics/acwater/-/issues).
Alternatively, you may run EnPT in the 'land' atmospheric correction mode based on SICOR.
The error message is now raised:
FAILED
tests/test_controller.py:67 (Test_EnPT_Controller_DLR_testdata.test_run_all_processors)
self = <tests.test_controller.Test_EnPT_Controller_DLR_testdata testMethod=test_run_all_processors>
def test_run_all_processors(self):
> self.CTR.run_all_processors()
test_controller.py:69:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../enpt/execution/controller.py:184: in run_all_processors
self.run_atmospheric_correction()
../enpt/execution/controller.py:152: in run_atmospheric_correction
self.L1_obj.run_AC()
../enpt/model/images/images_sensorgeo.py:858: in run_AC
AC.run_ac(self)
../enpt/processors/atmospheric_correction/atmospheric_correction.py:264: in run_ac
self._run_AC__water_mode(enmap_ImageL1)
../enpt/processors/atmospheric_correction/atmospheric_correction.py:153: in _run_AC__water_mode
polymer_ac_enmap(enmap_l1b=enmap_ImageL1,
../../../../mambaforge/envs/enpt_full/lib/python3.9/site-packages/acwater/acwater.py:214: in polymer_ac_enmap
product = run_atm_corr(l1_enmap,
../../polymer/polymer/main.py:517: in run_atm_corr
for block in block_iter:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <multiprocessing.pool.IMapUnorderedIterator object at 0x7f79b66b5400>
timeout = None
def next(self, timeout=None):
with self._cond:
try:
item = self._items.popleft()
except IndexError:
if self._index == self._length:
self._pool = None
raise StopIteration from None
self._cond.wait(timeout)
try:
item = self._items.popleft()
except IndexError:
if self._index == self._length:
self._pool = None
raise StopIteration from None
raise TimeoutError from None
success, value = item
if success:
return value
> raise value
E AssertionError
../../../../mambaforge/envs/enpt_full/lib/python3.9/multiprocessing/pool.py:870: AssertionError
=================================== FAILURES ===================================
__________ Test_EnPT_Controller_DLR_testdata.test_run_all_processors ___________
self = <tests.test_controller.Test_EnPT_Controller_DLR_testdata testMethod=test_run_all_processors>
def test_run_all_processors(self):
> self.CTR.run_all_processors()
test_controller.py:69:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../enpt/execution/controller.py:184: in run_all_processors
self.run_atmospheric_correction()
../enpt/execution/controller.py:152: in run_atmospheric_correction
self.L1_obj.run_AC()
../enpt/model/images/images_sensorgeo.py:858: in run_AC
AC.run_ac(self)
../enpt/processors/atmospheric_correction/atmospheric_correction.py:264: in run_ac
self._run_AC__water_mode(enmap_ImageL1)
../enpt/processors/atmospheric_correction/atmospheric_correction.py:153: in _run_AC__water_mode
polymer_ac_enmap(enmap_l1b=enmap_ImageL1,
../../../../mambaforge/envs/enpt_full/lib/python3.9/site-packages/acwater/acwater.py:214: in polymer_ac_enmap
product = run_atm_corr(l1_enmap,
../../polymer/polymer/main.py:517: in run_atm_corr
for block in block_iter:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <multiprocessing.pool.IMapUnorderedIterator object at 0x7f79b66b5400>
timeout = None
def next(self, timeout=None):
with self._cond:
try:
item = self._items.popleft()
except IndexError:
if self._index == self._length:
self._pool = None
raise StopIteration from None
self._cond.wait(timeout)
try:
item = self._items.popleft()
except IndexError:
if self._index == self._length:
self._pool = None
raise StopIteration from None
raise TimeoutError from None
success, value = item
if success:
return value
> raise value
E AssertionError
../../../../mambaforge/envs/enpt_full/lib/python3.9/multiprocessing/pool.py:870: AssertionError
=============================== warnings summary ===============================
tests/test_controller.py::Test_EnPT_Controller_DLR_testdata::test_run_all_processors
/home/gfz-fe/scheffler/python/EnPT/enpt/execution/controller.py:61: RuntimeWarning: No digital elevation model provided. Note that this may cause uncertainties, e.g., in the atmospheric correction and the orthorectification.
self.cfg: EnPTConfig = config or EnPTConfig(**config_kwargs)
tests/test_controller.py::Test_EnPT_Controller_DLR_testdata::test_run_all_processors
tests/test_controller.py::Test_EnPT_Controller_DLR_testdata::test_run_all_processors
tests/test_controller.py::Test_EnPT_Controller_DLR_testdata::test_run_all_processors
/home/gfz-fe/scheffler/python/polymer/polymer/luts.py:187: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
if self.data.dtype in [np.float, np.float32, np.float64]:
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED test_controller.py::Test_EnPT_Controller_DLR_testdata::test_run_all_processors
================== 1 failed, 4 warnings in 256.51s (0:04:16) ===================
Exception ignored in: <function Pool.__del__ at 0x7f78a156a700>
Traceback (most recent call last):
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.9/multiprocessing/pool.py", line 268, in __del__
self._change_notifier.put(None)
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.9/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.9/multiprocessing/connection.py", line 205, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.9/multiprocessing/connection.py", line 416, in _send_bytes
self._send(header + buf)
File "/home/gfz-fe/mambaforge/envs/enpt_full/lib/python3.9/multiprocessing/connection.py", line 373, in _send
n = write(self._handle, buf)
OSError: [Errno 9] Bad file descriptor
Process finished with exit code 1