Skip to content

feat: pipeline checkpoints crash allow to resume from snapshot #9743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 53 commits into from
Sep 1, 2025
Merged

feat: pipeline checkpoints crash allow to resume from snapshot #9743

merged 53 commits into from
Sep 1, 2025

Conversation

davidsbatista
Copy link
Contributor

@davidsbatista davidsbatista commented Aug 26, 2025 •
edited
Loading

Related Issues

Proposed Changes:

  • When an error occurs during the pipeline execution, the full pipeline state before the crash is saved to a file
  • This allows resuming pipeline execution from the last successful state rather than starting over

How did you test it?

  • Extended the test that simulates a pipeline crash and ensured that the file was created

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added unit tests and updated the docstrings
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I documented my code
  • I ran pre-commit hooks and fixed any issue

@davidsbatista
initial PoC idea running
ef96b9f
@davidsbatista
removing test code
f00919b
@davidsbatista
cleaning up
443eb39
@davidsbatista
wip
9ed9e09
@davidsbatista
Merge branch 'main' into poc/resume-state-for-crashes
cdb2718
@davidsbatista
cleaning up demos
f1c01fc
@davidsbatista
adding more pipelines to test persistence saving
f249e22
@davidsbatista
wip
85dd0f1
@davidsbatista
wip
f3abfc7
@davidsbatista
working example for logging components inputs in run time
e17166b
@davidsbatista
reverting to a simpler solution for intermediate results
87cbb00
@davidsbatista
cleaning up
db7fca5
@davidsbatista
testing that in a crash components outputs/inputs up to the crash poi…
b0caafc 
…nt are returned
@davidsbatista
adding tests for state persistance in a RAG pipeline
30176c5
@davidsbatista
updataing tests for state persistance in a RAG pipeline
b93b221
@davidsbatista
Merge branch 'main' into poc/resume-state-for-crashes
f79a9cd
@davidsbatista
removing use cases of agent tests
7b75803
@davidsbatista
adding LICENSE header
d741d55
@davidsbatista
adding LICENSE header
b1bdda5
@davidsbatista
adding release notes
89aa58f
@davidsbatista
updating tests for mocked components only
a276bc4
@davidsbatista
Merge branch 'main' into poc/resume-state-for-crashes
2f0ed63
@davidsbatista
updating release notes
44d2faf
@davidsbatista
adapting PipelineRuntimeError
d6a0919
@davidsbatista
cleaning up tests
98b8b3c
@davidsbatista
fixing test pipeline crash components inputs/outputs are saved
f0165b3
@davidsbatista
fixing tests for state persistance
2f3db39
sjrl
sjrl reviewed Aug 28, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 28, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 28, 2025
View reviewed changes
@github-actions github-actions bot added the type:documentation Improvements on the docs label Aug 29, 2025
@davidsbatista
Copy link
Contributor Author

@sjrl thanks for the comments - I've added something to handle the saving of the snapshot file in a more safe way

@davidsbatista davidsbatista requested a review from sjrl August 29, 2025 11:24
sjrl
sjrl reviewed Aug 29, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 29, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 29, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 29, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 29, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 29, 2025
View reviewed changes
davidsbatista and others added 2 commits August 29, 2025 14:57
sjrl
sjrl reviewed Aug 29, 2025
View reviewed changes
sjrl
sjrl reviewed Aug 29, 2025
View reviewed changes
@davidsbatista davidsbatista removed the type:documentation Improvements on the docs label Aug 29, 2025
@davidsbatista davidsbatista requested a review from sjrl August 29, 2025 15:18
sjrl
sjrl approved these changes Sep 1, 2025
View reviewed changes
Copy link
Contributor

@sjrl sjrl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@davidsbatista davidsbatista merged commit 6c7ae8f into main Sep 1, 2025
23 checks passed
@davidsbatista davidsbatista deleted the feat/pipeline-checkpoints-crash-resume branch September 1, 2025 09:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sjrl sjrl sjrl approved these changes

Assignees
No one assigned
Labels
topic:core topic:tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Allow for a regular Pipeline to be resumed from any execution point
3 participants
@davidsbatista @coveralls @sjrl