ValueError when using opensearch.index_df with documents with an array field

[oeN]

Describe the bug

When I try to index a data frame that contains documents with array fields I receive this error:

.../awswrangler/opensearch/_write.py", line 82, in <dictcomp>
    yield {k: _deserialize(v) for k, v in document.items() if notna(v)}
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

How to Reproduce

Code that triggers the error:

df = pd.DataFrame([
    {"_id": "1", "name": "John", "tags": ['foo', 'bar']}, 
    {"_id": "2", "name": "George", "tags": ['bar', 'baz']}, 
    {"_id": "3", "name": "Julia", "tags": ['bar']}
])
wr.opensearch.index_df(client=client,df=df,index='test_index_for_array')

Expected behavior

No response

Your project

No response

Screenshots

No response

OS

Mac

Python version

3.9.12

AWS DataWrangler version

2.16.0

Additional context

No response

[malachi-constant]

Thanks for the issue @oeN , I was able to replicate and not sure what exactly is causing this at the moment.

Looks like we have a problem here with items of type array and a length >= 1.

@kukushking Any ideas on this one before I investigate further?

[maxdavidson91]

It appears that if notna(v) is causing this error when a list object is passed to it.

This is untested, and I'm not sure what impact this would have on the rest of the code base, but the following would fix this specific error:
yield {k: _deserialize(v) for k, v in document.items() if (type(v)==list and notna(v).any()) or notna(v)}

[malachi-constant]

That didn't quite work in my testing as it tries to evaluate notna(v).any()) for all values, but the same thinking gave me an idea. Let me test a new change, i'll open a PR.

[malachi-constant]

@oeN Ok I put a potential fix in, take a look and the other reviewers will weigh in as well.

[maxdavidson91]

@malachi-constant casting all values to a numpy array seems like a good idea.
My only concern is that using notna(v).all() over notna(v).any() may result in data loss.

• notna(v).all() will only evaluate to True if all values in the array are not NA. So if there is one NA in the array, the whole array will be dropped
• notna(v).any() will evaluate to True as long as there is at least one non-NA value

[oeN]

@malachi-constant thank you very much!