Fitting a LassoModel with a particular choice of lambda

[barankarakus]

It seems to me that the Lasso model-fitting API is not designed for the user to simply pick a particular choice of lambda and fit a model with that particular choice. Having looked at the source code I can see that the way to do this would be to fit a LassoPath and specify the chosen lambda value as a vector containing only that value. This is a bit clunky: I only want to fit one model yet I'm using the LassoPath API.

I suggest having a fit(LassoModel, some stuff; \lambda) method which allows to do this. It should fit a LassoPath only at the chosen lambda value, then construct the LassoModel object from this. We could also give it a default value of 1.0, like sklearn, so that the user can just run fit(LassoModel, X, y) to quickly fit a 'default' lasso model.

Maintainers of this project: If you give me the green light, I can go ahead and submit a PR.

[AsafManela]

This sounds reasonable from the user's perspective, but I'm not sure the coordinate descent algorithm will work properly in that case. Does glmnet in R or any other package provide this option so you could test against?

[barankarakus]

sklearn in Python does. Why would co-ordinate descent not work properly? Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Asaf Manela <notifications@github.com> Sent: Tuesday, August 18, 2020 3:02:44 PM To: JuliaStats/Lasso.jl <Lasso.jl@noreply.github.com> Cc: barankarakus <barankarakus@hotmail.co.uk>; Author <author@noreply.github.com> Subject: Re: [JuliaStats/Lasso.jl] Fitting a LassoModel with a particular choice of lambda (#49) This sounds reasonable from the user's perspective, but I'm not sure the coordinate descent algorithm will work properly in that case. Does glmnet in R or any other package provide this option so you could test against? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#49 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIFDPKU2F64XBMGVGAAL3WTSBKCYJANCNFSM4QCDAYGQ>.

[AsafManela]

It works by starting from a null model (high lambda) then gradually decreasing it with small changes to coefficients. I'm just not sure how it would work if you solved directly for a specific lambda, though the API currently seems to support it.
I suggest you first experiment a bit with providing a vector with a single lambda and see what it does and how it compares with what sklearn gives in that case. It's been a while since I looked at this code carefully, so it may just work.

[barankarakus]

Ah, I see. I’m guessing that, given a sequence of lambdas, the algorithm solves the problem for the largest lambda (more precisely, for lambda_max, the smallest lambda giving rise to the null model), and then solves for the next largest lambda, using the solution of the current problem as an initial iterate... until we’ve solved the problem for all the lambdas. If this is accurate, then I’m guessing that feeding it just one lambda value would simply apply coordinate descent for that one problem, with initial iterate the zero vector. This is what sklearn does too. Are you happy with me running some tests and, if all looks good, submit a PR? I imagine an API like this: fit(LassoModel, X, y; lambda=1.0, alpha=1.0, <possibly some other options like randomising coefficient selection or not>) Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Asaf Manela <notifications@github.com> Sent: Tuesday, August 18, 2020 3:16:09 PM To: JuliaStats/Lasso.jl <Lasso.jl@noreply.github.com> Cc: barankarakus <barankarakus@hotmail.co.uk>; Author <author@noreply.github.com> Subject: Re: [JuliaStats/Lasso.jl] Fitting a LassoModel with a particular choice of lambda (#49) It works by starting from a null model (high lambda) then gradually decreasing it with small changes to coefficients. I'm just not sure how it would work if you solved directly for a specific lambda, though the API currently seems to support it. I suggest you first experiment a bit with providing a vector with a single lambda and see what it does and how it compares with what sklearn gives in that case. It's been a while since I looked at this code carefully, so it may just work. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#49 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIFDPKU4RFJPDBUYM2C46BLSBKEKTANCNFSM4QCDAYGQ>.

[AsafManela]

Sounds good.
There may also be a way to use the select option to get what you want too, by implementing a segment selector (SegSelect) for that single lambda. Its segselect() would just give 1 always. This may or may not be simpler.

[barankarakus]

Sure, but we don’t want to be fitting an entire path then selecting the particular segment when we just need the problem solved for that particular segment. That said, even in this scenario it may be faster to fit a path: choosing a sequence of lambdas lambda_0 = lambda < lambda_1 < ... < lambda_n = lambda_max (where lambda is the value we’re interested in solving the problem for), then solving the problem for lambda_n, lambda_{n-1}, ... in sequence. Though I think I’ll just keep it simple and solve the problem for lambda only. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Asaf Manela <notifications@github.com> Sent: Tuesday, August 18, 2020 3:40:38 PM To: JuliaStats/Lasso.jl <Lasso.jl@noreply.github.com> Cc: barankarakus <barankarakus@hotmail.co.uk>; Author <author@noreply.github.com> Subject: Re: [JuliaStats/Lasso.jl] Fitting a LassoModel with a particular choice of lambda (#49) Sounds good. There may also be a way to use the select option to get what you want too, by implementing a segment selector (SegSelect) for that single lambda. Its segselect() would just give 1 always. This may or may not be simpler. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#49 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIFDPKX4TWHSEURKD43CIEDSBKHGNANCNFSM4QCDAYGQ>.

[BigBen-L]

Hey guys, have you considered about using LARS to solve Lasso problems? barankarakus <notifications@github.com> 于2020年8月18日周二 上午5:34写道：

…

It seems to me that the Lasso model-fitting API is not designed for the user to simply pick a particular choice of lambda and fit a model with that particular choice. Having looked at the source code I can see that the way to do this would be to fit a LassoPath and specify the chosen lambda value as a vector containing only that value. This is a bit clunky: I only want to fit one model yet I'm using the LassoPath API. I suggest having a fit(LassoModel, *some stuff*; \lambda) method which allows to do this. It should fit a LassoPath only at the chosen lambda value, then construct the LassoModel object from this. We could also give it a default value of 1.0. Maintainers of this project: If you give me the green light, I can go ahead and submit a PR. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#49>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMRJPWYD3HTS7XANYX2TM23SBGA57ANCNFSM4QCDAYGQ> .

[barankarakus]

I’ve never used it and don’t know the algorithm in detail, but to the best of my knowledge the time complexity does not scale well with the number of features and the precision required of the solutions. Moreover, I don’t think the LARS algorithm can take advantage of so-called ‘screening rules’ which (I believe) the glmnet algorithm utilises; these can provide significant speed-ups. Get Outlook for iOS<https://aka.ms/o0ukef>

________________________________ From: BigBen-L <notifications@github.com> Sent: Wednesday, August 19, 2020 12:26:20 AM To: JuliaStats/Lasso.jl <Lasso.jl@noreply.github.com> Cc: barankarakus <barankarakus@hotmail.co.uk>; Author <author@noreply.github.com> Subject: Re: [JuliaStats/Lasso.jl] Fitting a LassoModel with a particular choice of lambda (#49) Hey guys, have you considered about using LARS to solve Lasso problems? barankarakus <notifications@github.com> 于2020年8月18日周二 上午5:34写道：

It seems to me that the Lasso model-fitting API is not designed for the user to simply pick a particular choice of lambda and fit a model with that particular choice. Having looked at the source code I can see that the way to do this would be to fit a LassoPath and specify the chosen lambda value as a vector containing only that value. This is a bit clunky: I only want to fit one model yet I'm using the LassoPath API. I suggest having a fit(LassoModel, *some stuff*; \lambda) method which allows to do this. It should fit a LassoPath only at the chosen lambda value, then construct the LassoModel object from this. We could also give it a default value of 1.0. Maintainers of this project: If you give me the green light, I can go ahead and submit a PR. ― You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#49>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMRJPWYD3HTS7XANYX2TM23SBGA57ANCNFSM4QCDAYGQ> .

― You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#49 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIFDPKVOBBI7JC3VQ23ULBTSBMEZZANCNFSM4QCDAYGQ>.

[GabrielPonte]

do you have any updates related to this? thanks!