`assets:clean` task is not cleaning predigested assets with `.digest` in the name

[tsrivishnu]

Whenever rake assets:clean is run, the assets in public/assets with .digest in their name are not cleaned at all. This causes assets to accumulate on production and drastic increase in disk usage over time.

I realised that this is because of the following:

propshaft/lib/propshaft/output_path.rb

Lines 55 to 60 in 69c285f

	def extract_path_and_digest(digested_path)
	digest = digested_path.to_s[/-([0-9a-f]{7,128})\.(?!digested)[^.]+\z/, 1]
	path = digest ? digested_path.sub("-#{digest}", "") : digested_path
	[path, digest]
	end

The regex ignores all files that have a . digest in them.

Is this on purpose? or is this a bug?

[tsrivishnu]

Looks like there a brief mention of this in #110 but no (clear) resolution.

[brenogazzola]

It's a bug. Are you using esbuild and allowing chunking?

[tsrivishnu]

We allow chunking. Not with esbuild but with Webpack. I suppose the bundler doesn’t matter. Are there any attempts at fixing it that I can look at? Or known workarounds? For us, the assets directory grows to 40GB in a month or two and resolving that requires clobbering and precompile with a downtime of couple of minutes — a heavy JavaScript app that is going through a lot of iterations basically.  I’m going to test a script tomorrow that would look at the `.manifest.json` and deletes all the files with `.digested` in their name and are not mentioned in the manifest. On 29. May 2023, at 20:10, Breno Gazzola ***@***.***> wrote: It's a bug. Are you using esbuild and allowing chunking? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[brenogazzola]

I think the problem is caused by #100 which I'm considering reverting as it's causing multiple problems, I just need confirmation. Last question: Do all chunks have the same base name and only differ in the digest?

I'm asking because clean is actually working for me, keeping only the 3 most recent versions of each file

[brenogazzola]

Actually, I just figured it out. Clean is not going to work because each file name is unique due to the digest, and therefore Propshaft cannot group them into "versions" to delete the old ones.

I'll have to think a bit about how to handle this (I would also welcome a PR)

[tsrivishnu]

Examples of pre-digested files:

public/assets/Vue3Lottie-1ea0f3b6ddc59bef7019.digested.js
public/assets/Vue3Lottie-23af1d5aa7baee8b1ca8.digested.js
public/assets/Vue3Lottie-26a7aeceee4e261a29b5.digested.js
public/assets/Vue3Lottie-337a4df42c71a547bccd.digested.js
public/assets/Vue3Lottie-3d0d69a396646d61f938.digested.js
public/assets/Vue3Lottie-41282d7f6fa2d6b4d891.digested.js

public/assets/Vue3Lottie-1ea0f3b6ddc59bef7019.digested.js-261e2968c4f774cb50ba.digested.map
public/assets/Vue3Lottie-23af1d5aa7baee8b1ca8.digested.js-c3e14423760666ecd16d.digested.map
public/assets/Vue3Lottie-26a7aeceee4e261a29b5.digested.js-c9d39559c1296b3e0dab.digested.map
public/assets/Vue3Lottie-337a4df42c71a547bccd.digested.js-27763e2e7bdcf11da3ff.digested.map
public/assets/Vue3Lottie-3d0d69a396646d61f938.digested.js-32f969d5062f50ff3a60.digested.map
public/assets/Vue3Lottie-41282d7f6fa2d6b4d891.digested.js-abed1ffa6c0eaad60d5a.digested.map

public/assets/Vue3Lottie-1ea0f3b6ddc59bef7019.digested.js.LICENSE.txt
public/assets/Vue3Lottie-23af1d5aa7baee8b1ca8.digested.js.LICENSE.txt
public/assets/Vue3Lottie-26a7aeceee4e261a29b5.digested.js.LICENSE.txt
public/assets/Vue3Lottie-337a4df42c71a547bccd.digested.js.LICENSE.txt
public/assets/Vue3Lottie-3d0d69a396646d61f938.digested.js.LICENSE.txt
public/assets/Vue3Lottie-41282d7f6fa2d6b4d891.digested.js.LICENSE.txt

I can look into it sometime to see if I can make a PR. At least, I will try to give a workaround script using the manifest to clear unused assets.

[brenogazzola]

I'd love to say we could just group them by the name before the digest, but esbuild gives all chunks the same name and that would break it.

[schmijos]

I've got a naive solution here: #231
It boils down to a different Regex than for a Propshaft::Asset while still listing them pre-digested files in Propshaft::OutputPath#files.