[Bug]: Rspack persistent cache doesn't work for code-generated files

[slorber]

System Info

Rspack 1.2.5

Details

When an app file to bundle is code-generated like that:

await fs.promises.writeFile(
  "./src/dynamicFile.js",
  `export const dynamic = 42`,
);

Then Rspack persistent cache will not work as expected: the JS loader will run again on subsequent app compilations.

This also happens when using this.addDependency("./dynamicFile.js")" in a loader.

This differs from Webpack 5 behavior which will cache the file and skip reloading it.

Reproduce link

slorber/rspack-cache-repro#1

Reproduce Steps

Check code on repro/PR: slorber/rspack-cache-repro#1

You can run it with yarn install && yarn test

Here's the console output printing the extra unwanted loader runs

###############################################
# USING WEBPACK
#############

##########
Bundle App start
- Loading src/entry.js
- Loading src/file.js
- Loading src/dynamicFile.js
Bundle App: 2.146s

##########
Bundle App start
Bundle App: 5.507ms

##########
Bundle App start
Bundle App: 4.528ms

###############################################
# USING RSPACK
#############

##########
Bundle App start
- Loading src/entry.js
- Loading src/file.js
- Loading src/dynamicFile.js
Bundle App: 2.032s

##########
Bundle App start
- Loading src/dynamicFile.js ===========> BAD
Bundle App: 1.009s

##########
Bundle App start
- Loading src/dynamicFile.js ===========> BAD
Bundle App: 1.006s

cc @hardfist @jerrykingxyz as requested on Discord

[tats-u]

Possibly is it fixed in 1.2.7?

[jerrykingxyz]

@slorber @tats-u Yes, This bug should be fixed in 1.2.7.
By the way, I noticed that the repro you provided used compiler.run more than once. You should pay attention to providing modifiedFiles and removeFiles when calling compiler.run. https://rspack.dev/api/javascript-api/compiler#run

[slorber]

Thanks

To be honest I don't really understand how I am supposed to use these options.

Webpack doesn't document these options, and it works well out of the box when calling compiler.run() multiple times
https://webpack.js.org/api/node/#run

If possible, I'd like Rspack to work as well as Webpack does, without having to provide any options.

[jerrykingxyz]

@slorber
In webpack, the main purpose of modifiedFiles & removeFiles is to provide information for plugins. Webpack will scan files during make stage to determine whether the cache is invalid (usually the modification time + hash of the file).
For performance reasons, rspack uses modifiedFiles & removeFiles as the condition for cache invalidation, which saves additional file reading, hash calculation and other overhead.

For your questions

1. How to use modifiedFiles & removeFiles?
   Generally, we trigger a rebuild when user modifies a file, so we can get the file from watcher and pass them to modifedFiles, and the same applies to deleted files

2. Is it possible not to pass it like webpack?
   Assuming that we don't consider the performance overhead caused by recalculation, this can be done. Let's go back to this issue. The reason why the loader is run every time when a file is updated is because the previous persistent cache strategy is to automatically calculate if ModifiedFiles and RemovedFiles are not provided, but this calculation is based on the modification time of the file but does not include hash, so this problem occurs.

In summary, I think it is acceptable to provide ModifiedFiles and removedFiles for performance reasons. Of course, we can also provide a configuration option to automatically calculate them when ModifiedFiles and removedFiles are not provided, but I guess users will be more inclined to pass them when they find the performance degradation.

[slorber]

@jerrykingxyz thanks for the explanations.

Unfortunately, I'm not sure how to leverage modifiedFiles for my use case. It would require me to somehow implement my own persistent cache on top of Rspack one to be able to provide that value.

I've created a second repro here so that my problem is clearer: slorber/rspack-cache-repro#2

The first repro was misleading because it calls compiler.run() multiple times.

In practice, what I try to optimize is prod builds. There's no file watcher to get a list of modifiedFiles in that context. We generate a ton of files at the start of the build process, that happens in a new CI job.

Let me know if you need more information to understand the problem

[slorber]

And here's another repro showing a similar problem we have when using this.addDependency("./dynamicFile.js")

slorber/rspack-cache-repro#3

To me, v2 and v3 repros highlight the same problem: relying on last modified time instead of hash(fileContent)