AssetManager refactoring

[razvanphp]

hello,

We are planning the deployment of a new Yii2 app to production and from past experience, the most painful thing is related to DevOps, docker containers or multi server setup: the AssetManager needs to be refactored in a way to make stateless deployments possible, since asset publishing mechanism does not work in this new era of cloud native apps.

To explain this better, imagine you pack your yii2 app in a docker container and deploy it in a service with 10 replicated pods.

1. First request will hit container number 1, which will publish it's own assets locally (which is bad anyway, those containers should not write to disk)
2. Then the browser tries to request the asset and the load balancer uses container number 2 to serve that request.
3. Since this new container did not publish yet his assets, it will result in a 404.
4. Also, a configuration scenario with assets hosted in a CDN (preferred in a modern deployment), there is no way to publish all static assets to the CDN/S3 BEFORE deploying the new app version, so that when deploying there is no race condition which could cause assets not to be found.
5. The current recommended way to change the assets links to CDN hosted versions via the bundle or assetMap class config is not the best, since yii\web\YiiAsset internal assets for example are not publicly published to any CDN.
6. The whole logic of asset hashing in the path is based on the assumption that it will run on the same server, with the same modified timestamp, so if unexperienced developers try to use this, it will end up in duplicated assets being published (one must need to know they have to change the hashCallback function -- maybe we should change it by default with this new release).

Now, I'm not sure what the best way to improve/fix this is, but:

1. one idea I tried is mikk150/yii2-asset-manager-flysystem, which unfortunately introduces new dependancies and make the response time worse (maybe because of caching)
2. maybe we should include a command that scans the codebase and publishes all assets from CLI.
3. remove all JS dependancies from core framework, publish YiiAsset as a NPM package so popular CDNs can pick it up with release versions and then developers must include and configure the bundles to use it... some can even be deprecated, like PJAX in favour of HTMX, inputmask is also very old and unmaintained, etc.
4. better, more elegant suggestions?

I'm willing to work on this after we brainstorm ideas.

Thank you & I'm really excited for this new release!
R

[schmunk42]

The whole logic of asset hashing in the path is based on the assumption that it will run on the same server,

You can adjust hashCallback from assetManager to fit your needs, i.e. generate a hash from your current app version.

You also have yii asset/compress to pre-bundle assets, but it's a pain to identify all asset-bundles and moreover there are asset bundles with logic, which always will be published on demand.

IMHO the current asset-manager can be configured to fit "state-of-the-art cloud needs", but anyway, if you want to reimplement the publish logic, just create an extension.

[mikk150]

the fact that there is asset-bundles with logic is actually a big problem.. this pattern should be discouraged somehow

[razvanphp]

what do you mean by logic? Something like this? Or worse?

public $js = [
    YII_ENV_DEV ? 'htmx.js' : 'htmx.min.js',
];

[mikk150]

yes

[razvanphp]

You can adjust hashCallback from assetManager to fit your needs, i.e. generate a hash from your current app version.

I think one does not want to break all the asset cache on each app version bump, but rather only when the version of the asset itself was changed. This is where the logic gets pretty complicated and would love to put this logic into the framework itself, rather than expect each developer to implement this on their own, the right way.

IMHO the current asset-manager can be configured to fit "state-of-the-art cloud needs", but anyway, if you want to reimplement the publish logic, just create an extension.

I would love to learn that this is a skill issue on my part, so please let me know how you would do it then: how do you make the docker image stateless with read-only disk and how do you publish the YiiAsset to a CDN, before deployment, in a build pipeline?

[schmunk42]

but rather only when the version of the asset itself was changed. This is where the logic gets pretty complicated and would love to put this logic into the framework itself, rather than expect each developer to implement this on their own, the right way.

How would you detect this in the right way?
Current default is to look after the mtime of the directory the file is in and hash this. Which works good in standard single server deployments, but tends to be problematic in development.

how do you make the docker image stateless with read-only disk and how do you publish the YiiAsset to a CDN, before deployment, in a build pipeline?

This depends heavily on your setup, ie. we have a project where it is possible to share a network volume between containers, but in this case it's tightly connected to the ISP and K8S.
You could also use a different baseUrl for published assets, which always points to one specific container, this can be handled in a reverse proxy.
Or you could publish to something like S3, which usually needs some management with lock-files, because it's slow and tends to have race-conditions and so on...

I think there's not the one solution when it comes to asset publishing on scaled systems.

[mikk150]

This depends heavily on your setup, ie. we have a project where it is possible to share a network volume between containers, but in this case it's tightly connected to the ISP and K8S.

I wouldn't do that for performance reasons, especially when you can just compile and compress all assets into container itself using Yii2 internal already-existing tools.. file paths using this method won't change any more.
Yes, it does require more effort from developers to track what assets are needed(and preferrably where) but performance wise this is the best solution

[razvanphp]

but rather only when the version of the asset itself was changed. This is where the logic gets pretty complicated and would love to put this logic into the framework itself, rather than expect each developer to implement this on their own, the right way.

How would you detect this in the right way? Current default is to look after the mtime of the directory the file is in and hash this. Which works good in standard single server deployments, but tends to be problematic in development.

We can make use of composer Runtime Utilities:

\Composer\InstalledVersions::getVersion('my/package')

or even ::getReference() and substr the original git hash.

Is there anybody NOT using composer these days?

[mikk150]

but rather only when the version of the asset itself was changed. This is where the logic gets pretty complicated and would love to put this logic into the framework itself, rather than expect each developer to implement this on their own, the right way.

How would you detect this in the right way? Current default is to look after the mtime of the directory the file is in and hash this. Which works good in standard single server deployments, but tends to be problematic in development.

We can make use of composer Runtime Utilities:

\Composer\InstalledVersions::getVersion('my/package')

or even ::getReference() and substr the original git hash.

Is there anybody NOT using composer these days?

but what if I publish assets that are not installed from composer (application in-built assets like icons and custom css/JS)
What can I use then? because composer has no idea bout them

[razvanphp]

We can still use the current hashing mechanism and function with override possibility as fallback then.

So this change would actually be an improvement with backwards compatibility, so that hash does not change if package is installed with composer and version did not change.

[razvanphp]

CC @mikk150

[mikk150]

Best way I have found how to do it is using asset manager

1. define asset compilation config as shown
2. compress assets in your build step as shown

IF you do not want java in your application in any point of time, you do need to change yii2 assetconverter to My own implementation

I haven't see any reason to deploy assets to CDN if you can simply use cloudfare

[mikk150]

also, do not use my flysystem thingomabob... I think probably it can be used to publish those minified and compiled assets to CDN.. but..... still, why?

[razvanphp]

Ideally, static file requests should not be served by the PHP containers, but be hosted on dedicated assets server. Reason is, when putting cloudflare or any other CDN in front, you still get a lot of requests from all the edges of the CDN network, up until they get warm.

oh, and java dependency is not cool at all... no even in my CI container 🙂 yuicompressor is abandoned since 5 years, maybe we can switch to bun or parcel instead.

[My6UoT9]

I imagine an AssetManager which allows to define assets to be either bundled into a bundle, or standalone using it directly with registerAsset in a view.

Then a a cli command, which creates the effective bundle configuration, which is then passed to webpack, rollupjs or similar (maybe even vite, but vite is based on rollupjs, I would prefer rollupjs).

Then the JS bundler, creates the minified versions and everything necessary so it can be published to /assets.
It will also compile the standalone assets.

Lastly use the existing JS oecosystem, that is package.json and one of its package managers. Yii can either publish there assets to npm, or use the AssetManager.
What would be nice, some way to safely add dependencies to the package.json from composer.