Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial implementation of likelihood profiling #37

Merged
merged 28 commits into from
Sep 18, 2024
Merged
Changes from 1 commit
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
4ca65f4
Initial implementation of likelihood profiling (fix #1)
billdenney Dec 13, 2022
6c540b8
Additional feature updates
billdenney Dec 13, 2022
f4503f4
Intermediate work
billdenney Dec 19, 2022
612ca73
Merge branch 'main' into fix-1
billdenney Jan 21, 2023
fc6e1b5
Update documentation
billdenney Jan 21, 2023
b516473
Merge branch 'main' into fix-1
billdenney Aug 8, 2024
c7a3a63
Update documentation
billdenney Aug 8, 2024
725830e
Fix check issues
billdenney Aug 8, 2024
56ac86e
A much-improved work-in-progress. `profile.nlmixr2FitCore()` now wor…
billdenney Aug 18, 2024
f8d87f5
Stop estimation based on number of significant digits in estimated pa…
billdenney Aug 19, 2024
589a622
Use the default RSE when it is not available
billdenney Aug 19, 2024
d3c5272
Rename rse_theta to rseTheta
billdenney Aug 19, 2024
114ccf2
Update CI
billdenney Aug 19, 2024
1a7b123
Remove `quiet` argument
billdenney Aug 19, 2024
26b965a
Add more development version packages for CI
billdenney Aug 19, 2024
c482d93
correct dparser repo location
billdenney Aug 19, 2024
4c1891d
Remove unused `ignoreBounds` argument
billdenney Aug 19, 2024
5159186
Fix check issues
billdenney Aug 19, 2024
98805e1
Remove unused `maxpts` argument
billdenney Aug 19, 2024
3bfb925
Switch to a generalized profile method with `method` and `control` ar…
billdenney Aug 20, 2024
8e5f79f
Ensure that initial estimates are within the limits for the parameter…
billdenney Aug 20, 2024
cd720f6
Test residual error profiling
billdenney Aug 20, 2024
8e42770
Update documentation
billdenney Aug 20, 2024
101d16a
Documentation fix
billdenney Aug 20, 2024
1b705ca
Add omega values and covariances to profiling output
billdenney Aug 20, 2024
509db7b
Add rxUiDeparse method for llpControl
mattfidler Sep 17, 2024
64a4f43
Version bump
mattfidler Sep 17, 2024
b17bbe1
Update check config
mattfidler Sep 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 149 additions & 0 deletions R/profile.R

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Bill (@billdenney),
Thanks for the initial implementation. Very interesting! I have a question to you: Your code only allows the profiling of fixed effect parameters. Why not profiling e.g. the variance-covariances?
Thanks in advance!
Simon

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no specific reason for that right now. Generally, profiling fixed effects is easier than random effects because random effects can be correlated. So, I think that you would need to detect correlation and only allow profiling of the random effects without correlations (or remove the correlation). @mattfidler, can you fix a random effect or its correlation when it is correlated without fixing the other parts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have to fix the whole block

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was referring the omegas not the individual etas, since they are part of the marginal likelihood.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@simonbeyer1, yes, I understood that you were referring to omegas. The idea I was getting to is that if you specify an eta like etacl + etavc ~ c(0.2, 0.1, 0.2), we would have to fix or not fix the entirety of that, all or none of the c(0.2, 0.1, 0.2) part. I don't immediately have a good idea of how to profile correlated BSV, so that will remain out of scope for this initial version of profiling.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh okay! Now I got it! Thanks!

Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
profile.nlmixr2FitData <- function(fitted, ..., which = NULL, maxpts = 10, ofvIncrease = 3.84, normQuantile = 1.96, rse_theta, tol = 0.001, ignoreBounds = FALSE, quiet = TRUE) {
browser()
if (is.null(which)) {
which <- names(fixef(fitted))
} else {
checkmate::assert_subset(x = which, choices = names(fixef(fitted)), empty.ok = FALSE)
}
if (missing(rse_theta)) {
fittedVcov <- vcov(fitted)
if (is.null(fittedVcov)) {
# Handle when vcov() is an error
rse_theta <- 30
cli::cli_alert_info(paste("covariance is unavailable, using default rse_theta of", rse_theta))
} else {
sd_theta <- sqrt(diag(fittedVcov))
rse_theta <- 100*abs(sd_theta/fixef(fitted)[names(sd_theta)])
}
}
if (length(rse_theta) == 1 & is.null(names(rse_theta))) {
rse_theta <- setNames(rep(rse_theta, length(which)), which)
} else {
checkmate::assert_names(names(rse_theta), subset.of = names(fixef(fitted)))
checkmate::assert_numeric(rse_theta, lower = 0, any.missing = FALSE, min.len = 1)
}
if (quiet) {
# TODO: set print = 0 for all the control arguments
}
if (length(which) > 1) {
ret <- data.frame()
# Make the general case be a concatenation of single cases
for (currentWhich in which) {
ret <-
rbind(
ret,
profile.nlmixr2FitData(fitted = fitted, ..., which = currentWhich, maxpts = maxpts, ofvIncrease = ofvIncrease, normQuantile = normQuantile, rse_theta = rse_theta, tol = tol, quiet = quiet)
)
}
} else {
ret <- profileNlmixr2FitDataRet(fitted = fitted, which = which)
for (direction in c(-1, 1)) {
found <- FALSE
boundCol <- ifelse(direction < 0, "lower", "upper")
if (ignoreBounds) {
bound <- Inf*direction
} else {
bound <- setNames(fitted$ui$iniDf[[boundCol]], fitted$ui$iniDf$name)[which]
}
if (abs(bound - ret[[which]][1]/ret[[which]][1]) < tol) {
found <- TRUE
cli::cli_warn(
"parameter %s is close to the %s boundary and likelihood will not be profiled in that direction",
which, boundCol
)
}
currentEst <- ret[[which]][1] + direction*normQuantile*rse_theta[which]/100*abs(ret[[which]][1])
# Confirm that the initial estimate is within the bounds
if (direction < 0) {
currentEst <- max(bound, currentEst)
} else {
currentEst <- min(bound, currentEst)
}
currentMaxpts <- maxpts
# Prepare the current parameter to be fixed for each estimation
fittedFix <- do.call(ini, append(list(x=fitted), setNames(list(as.name("fix")), which)))
while (currentMaxpts > 0 & !found) {
currentMaxpts <- currentMaxpts - 1
newEst <- setNames(list(currentEst), which)
# TODO: How can I detect the current estimation method from the fitted
# object? Can you even do likelihood profiling with something other
# than focei?
newFit <- try(nlmixr(ini(fittedFix, newEst), est = "focei", control = list(print = 0)))
if (inherits(newFit, "try-error")) {
# TODO: Work around failed fits
stop("Fit failed")
} else {
ret <- rbind(ret, profileNlmixr2FitDataRet(fitted = newFit, which = which, rowname = nrow(ret) + 1))
ret <- ret[order(ret[[which]]), ]
currentEstList <- profileNlmixr2FitDataNewEst(estimates = ret, which = which, direction = direction, bound = bound, ofvIncrease = ofvIncrease, tol = tol)
if (is.character(currentEstList$found)) {
found <- TRUE
cli::cli_alert_danger(sprintf("Stopping search for %s in %s direction because %s", which, boundCol, currentEstList$found))
} else {
currentEst <- currentEstList$newEst
found <- currentEstList$found
}
}
}
}
}
ret
}

#' Give the output data.frame for a single model for profile.nlmixr2FitData
#' @inheritParams profile.nlmixr2FitData
#' @return A data.frame with columns for Parameter (the parameter name), OFV
#' (the objective function value), and the current estimate for each of the
#' parameters
#' @noRd
profileNlmixr2FitDataRet <- function(fitted, which, rowname = 0) {
if (any(names(fixef(fitted)) %in% c("Parameter", "OFV"))) {
cli::cli_abort("Cannot profile a model with a parameter named either 'Parameter' or 'OFV'")
}
ret <-
cbind(
data.frame(Parameter = which, OFV = fitted$objDf$OBJF),
data.frame(t(fixef(fitted)))
)
rownames(ret) <- as.character(rowname)
ret
}

#' Generate a new estimate, and return a list indicating if the new estimate
#' should not be run (because the current data find the value within the
#' tolerance)
#'
#' @param estimates A data.frame with the parameter estimates
profileNlmixr2FitDataNewEst <- function(estimates, which, direction, bound, ofvIncrease, tol, method = "linapprox") {
minOFV <- min(estimates$OFV, na.rm = TRUE)
minRow <- which(estimates$OFV %in% minOFV)
maskDirection <- !is.na(estimates$OFV) & estimates[[which]] <= estimates[[which]][minRow]
dOFV <- estimates$OFV[maskDirection] - minOFV
# Estimates in the correct direction
estDir <- estimates[[which]][maskDirection]
if (all(dOFV < ofvIncrease)) {
# Expand the search
newEst <- estimates[[which]][minRow] + direction*2*abs(diff(range(estDir)))
found <- FALSE
} else {
# Check that the OFV is monotonic
monotonic <- length(unique(sign(diff(dOFV)/diff(estDir)))) == 1
if (!monotonic) {
newEst <- NA
found <- "OFV is not monotonic"
} else if (method == "linapprox") {
newEst <- approx(x = dOFV, y = estDir, xout = ofvIncrease)$y
} else {
cli::cli_abort(sprintf("method %s is not implemented", method))
}
if (!is.na(newEst)) {
estAbove <- min(estimates[[which]][estimates[[which]] > newEst])
estBelow <- min(estimates[[which]][estimates[[which]] < newEst])
found <- abs((estAbove - estBelow)/estBelow) < tol
}
}
list(
newEst = newEst,
found = found
)
}