read_parquet Does Not Throw Error for Missing Column

[knoam]

Describe the bug

If you call pandas read_parquet and specify a missing column, you get an error as expected. But in corresponding awswrangler command, you do not.

How to Reproduce

Pandas code raises an exception:

pd.DataFrame({'a':[1],'b':[2]}).to_parquet('test.parquet')
pd.read_parquet('test.parquet', columns=['a','b','c'])

ArrowInvalid: No match for FieldRef.Name(c) in a: int64 b: int64

But awswrangler does not:

wr.s3.to_parquet(pd.DataFrame({'a':[1],'b':[2]}), f's3://{bucket}/test.parquet')
wr.s3.read_parquet(f's3://{bucket}/test.parquet', columns=['a','b','c'])

Expected behavior

I expect awswrangler to throw an error like the pandas one.

Your project

No response

Screenshots

No response

OS

Linux

Python version

3.6.13

AWS DataWrangler version

2.14.0

Additional context

No response

[malachi-constant]

The library is using pyarrow.parquet.ParquetFile.read() under the hood which allows for this behavior.

Example:

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table(
  {
    'a': [0],
    'b': [1]
  }
)

pq.write_table(table, 'example.parquet')
parquet_file = pq.ParquetFile('example.parquet')

parquet_file.read(columns=["a", "b", "c"])

Result:

pyarrow.Table
a: int64
b: int64
----
a: [[0]]
b: [[1]]

I'm not sure we want to deviate wrangler from the underlying pyarrow behavior. I'll let @jaidisido and @kukushking weigh in if they have thoughts otherwise.

[kukushking]

I agree since it's backward-incompatible change we should be careful. Perhaps, we could check the columns if validate_schema=True?

[malachi-constant]

Sure, I like that idea, I'll relabel this issue and add to the roadmap.