SDSS get_spectra() fail when data_release=12

[joacoh]

Version of packages:
Python 3.9.12
astroquery 0.4.7
astropy 6.0.1
numpy 1.26.4

The problem happens when I try to query a list of coordinates and when I try to get the spectra using plate, fiberID and mjd.

I'm quering real sources from the Portsmouth VAC of galaxy properties in the latest version available, which uses BOSS DR12 spectra. The pipeline I built used to work, but now it doesn't.

For example, using:
PLATE MJD FIBERID
7057 56593 293

SDSS.get_spectra(plate=7057, fiberID=293, mjd=56593, data_release=12)[0]

It gives me the following error:

Long error!

--------------------------------------------------------------------------- InconsistentTableError Traceback (most recent call last) Input In [19], in () ----> 1 SDSS.get_spectra(plate=7057, fiberID=293, mjd=56593, data_release=12)[0]

File ~/.local/lib/python3.9/site-packages/astroquery/sdss/core.py:766, in SDSSClass.get_spectra(self, coordinates, radius, matches, plate, fiberID, mjd, timeout, get_query_payload, data_release, cache, show_progress)
    753 @prepend_docstr_nosections(get_spectra_async.__doc__)
    754 def get_spectra(self, *, coordinates=None, radius=2. * u.arcsec,
    755                 matches=None, plate=None, fiberID=None, mjd=None,
    756                 timeout=TIMEOUT, get_query_payload=False,
    757                 data_release=conf.default_release, cache=True,
    758                 show_progress=True):
    759     """
    760     Returns
    761     -------
    762     list : List of `~astropy.io.fits.HDUList` objects.
    763 
    764     """
--> 766     readable_objs = self.get_spectra_async(coordinates=coordinates,
    767                                            radius=radius, matches=matches,
    768                                            plate=plate, fiberID=fiberID,
    769                                            mjd=mjd, timeout=timeout,
    770                                            get_query_payload=get_query_payload,
    771                                            data_release=data_release,
    772                                            cache=cache,
    773                                            show_progress=show_progress)
    775     if get_query_payload:
    776         return readable_objs

File ~/.local/lib/python3.9/site-packages/astroquery/sdss/core.py:706, in SDSSClass.get_spectra_async(self, coordinates, radius, matches, plate, fiberID, mjd, timeout, get_query_payload, data_release, cache, show_progress)
    704 if not matches:
    705     if coordinates is None:
--> 706         matches = self.query_specobj(plate=plate, mjd=mjd, fiberID=fiberID,
    707                                      fields=['run2d', 'plate', 'mjd', 'fiberID'],
    708                                      timeout=timeout, get_query_payload=get_query_payload,
    709                                      data_release=data_release, cache=cache)
    710     else:
    711         matches = self.query_crossid(coordinates, radius=radius, timeout=timeout,
    712                                      specobj_fields=['run2d', 'plate', 'mjd', 'fiberID'],
    713                                      spectro=True, get_query_payload=get_query_payload,
    714                                      data_release=data_release, cache=cache)

File ~/.local/lib/python3.9/site-packages/astroquery/utils/class_or_instance.py:25, in class_or_instance.__get__.<locals>.f(*args, **kwds)
     23 def f(*args, **kwds):
     24     if obj is not None:
---> 25         return self.fn(obj, *args, **kwds)
     26     else:
     27         return self.fn(cls, *args, **kwds)

File ~/.local/lib/python3.9/site-packages/astroquery/utils/process_asyncs.py:29, in async_to_sync.<locals>.create_method.<locals>.newmethod(self, *args, **kwargs)
     27 if kwargs.get('get_query_payload') or kwargs.get('field_help'):
     28     return response
---> 29 result = self._parse_result(response, verbose=verbose)
     30 self.table = result
     31 return result

File ~/.local/lib/python3.9/site-packages/astroquery/sdss/core.py:1049, in SDSSClass._parse_result(self, response, verbose)
   1046 if sys.platform.startswith('win'):
   1047     warnings.filterwarnings("ignore", category=AstropyWarning,
   1048                             message=r'OverflowError converting to IntType in column.*')
-> 1049 arr = Table.read(response.text, format='ascii.csv', comment="#")
   1050 for id_column in ('objid', 'specobjid', 'objID', 'specobjID', 'specObjID'):
   1051     if id_column in arr.columns:

File ~/.local/lib/python3.9/site-packages/astropy/table/connect.py:62, in TableRead.__call__(self, *args, **kwargs)
     59 units = kwargs.pop("units", None)
     60 descriptions = kwargs.pop("descriptions", None)
---> 62 out = self.registry.read(cls, *args, **kwargs)
     64 # For some readers (e.g., ascii.ecsv), the returned `out` class is not
     65 # guaranteed to be the same as the desired output `cls`.  If so,
     66 # try coercing to desired class without copying (io.registry.read
     67 # would normally do a copy).  The normal case here is swapping
     68 # Table <=> QTable.
     69 if cls is not out.__class__:

File ~/.local/lib/python3.9/site-packages/astropy/io/registry/core.py:221, in UnifiedInputRegistry.read(self, cls, format, cache, *args, **kwargs)
    218         kwargs.update({"filename": path})
    220 reader = self.get_reader(format, cls)
--> 221 data = reader(*args, **kwargs)
    223 if not isinstance(data, cls):
    224     # User has read with a subclass where only the parent class is
    225     # registered.  This returns the parent class, so try coercing
    226     # to desired subclass.
    227     try:

File ~/.local/lib/python3.9/site-packages/astropy/io/ascii/connect.py:19, in io_read(format, filename, **kwargs)
     17     format = re.sub(r"^ascii\.", "", format)
     18     kwargs["format"] = format
---> 19 return read(filename, **kwargs)

File ~/.local/lib/python3.9/site-packages/astropy/utils/decorators.py:604, in deprecated_renamed_argument.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    601             msg += f"\n        Use {alternative} instead."
    602         warnings.warn(msg, warning_type, stacklevel=2)
--> 604 return function(*args, **kwargs)

File ~/.local/lib/python3.9/site-packages/astropy/utils/decorators.py:604, in deprecated_renamed_argument.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    601             msg += f"\n        Use {alternative} instead."
    602         warnings.warn(msg, warning_type, stacklevel=2)
--> 604 return function(*args, **kwargs)

    [... skipping similar frames: deprecated_renamed_argument.<locals>.decorator.<locals>.wrapper at line 604 (2 times)]

File ~/.local/lib/python3.9/site-packages/astropy/utils/decorators.py:604, in deprecated_renamed_argument.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    601             msg += f"\n        Use {alternative} instead."
    602         warnings.warn(msg, warning_type, stacklevel=2)
--> 604 return function(*args, **kwargs)

File ~/.local/lib/python3.9/site-packages/astropy/io/ascii/ui.py:430, in read(table, guess, **kwargs)
    428 fast_reader_rdr = get_reader(**fast_kwargs)
    429 try:
--> 430     dat = fast_reader_rdr.read(table)
    431     _read_trace.append(
    432         {
    433             "kwargs": copy.deepcopy(fast_kwargs),
   (...)
    436         }
    437     )
    438 except (
    439     core.ParameterError,
    440     cparser.CParserError,
    441     UnicodeEncodeError,
    442 ) as err:
    443     # special testing value to avoid falling back on the slow reader

File ~/.local/lib/python3.9/site-packages/astropy/io/ascii/fastbasic.py:153, in FastBasic.read(self, table)
    150     try_string = {}
    152 with _set_locale("C"):
--> 153     data, comments = self.engine.read(try_int, try_float, try_string)
    154 out = self.make_table(data, comments)
    156 if self.return_header_chars:

File ~/.local/lib/python3.9/site-packages/astropy/io/ascii/cparser.pyx:417, in astropy.io.ascii.cparser.CParser.read()

InconsistentTableError: Number of header columns (1) inconsistent with data columns in data line 47

Now, this happens for every source in my sample (around 1k sources), it works for both DR14 and DR17, but I need DR12 spectras. The last time I ran it, I had astroquery 0.4.6, but now even forcing the use of that version, it doesn't work.

edit: forgot to mention that the spectra does indeed exist in every case, for example:

https://data.sdss.org/sas/dr12/boss/spectro/redux/v5_7_0/spectra/7057/spec-7057-56593-0293.fits

is the spectra for the plate, fiberid and mjd of the example for DR12

[bsipocz]

Thanks for reporting this. I can reproduce this and indeed I see a couple of test failures regarding this data release, so I suspect some upstream changes may have been made.

cc @weaverba137 in case he has some ideas straight away without diving into full debug mode

[weaverba137]

I agree that this is likely an unexpected upstream API change. It may take me a while to find time to investigate. In the meantime, do you get the expected results when you interact directly with the SDSS data servers?

[joacoh]

I only use astroquery, so I can only confirm that when using DR14 and DR17 all my functions and custom queries works as usual, is only when I use DR12 when everything breaks.

I can also add that other queries from NED and SIMBAD don't present any problem at all, so the problem must come directly from SDSS or the utilities in between. Just to point out, the same InconsistentTableError was pointed in astropy/astropy#9797 , but no solution was given, so maybe is the same problem? Just a different scenario. It seems kinda odd tho, since I know this function worked somewhere around July, 2024, and that issue is from 2019.

The AstroML implementation of a fetch_sdss_spectrum() function works like charm, but they uses the DAS (kinda outdated, DR7), while astroquery seems to use SAS as far as I know, so the implementations are different.

Hopefully this details are useful.

[weaverba137]

I have confirmed that this is a server-side issue. In fact the DR12 service is throwing an exception, so I don't think this is an API change. I have reached out to some SDSS colleagues to investigate further.

[bsipocz]

Thank you!

[weaverba137]

I checked today, and DR12 seems to be working again. Please confirm and close.

[bsipocz]

Thank you! Indeed the example code in the OP is now working.