Skip to content

fix: concat with union categories #3127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Apr 7, 2025
Merged

fix: concat with union categories #3127

merged 2 commits into from
Apr 7, 2025

Conversation

kukushking
Copy link
Contributor

Feature or Bugfix

  • Bugfix

Detail

  • When executing chunked reads over multiple files that contain different categories, category information is lost due to pd.concat() not preserving unmatched categories. Luckily, we already have a utility that concatenates data frames and unions categories.

Relates

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@kukushking kukushking requested review from a team and Copilot April 7, 2025 15:17
@kukushking kukushking self-assigned this Apr 7, 2025
Copilot
Copilot AI reviewed Apr 7, 2025
View reviewed changes
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

@kukushking kukushking linked an issue Apr 7, 2025 that may be closed by this pull request
Why does s3.read_parquet() returns different data type depending on chunk-size? #3123
Closed
@jaidisido
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 43161f9
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jaidisido
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
  • Commit ID: 40d9ce4
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jaidisido
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
  • Commit ID: 43161f9
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@jaidisido
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
  • Commit ID: 40d9ce4
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@kukushking kukushking merged commit a08e7e9 into main Apr 7, 2025
22 checks passed
@kukushking kukushking deleted the fix/concat-union-categories branch April 7, 2025 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

@jaidisido jaidisido jaidisido approved these changes

Assignees

@kukushking kukushking

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Why does s3.read_parquet() returns different data type depending on chunk-size?
2 participants
@kukushking @jaidisido