2022 07 25 report

Report 2022 07 25

Last week I was separating partially enumerated methods from fully enumerated ones, and updating reporting accordingly. The results are sort of mixed:

Module                 Methods  Stable  Partial  Unstable  Any  Vararg  Generic  TcFail  NoFuel
DifferentialEquations  1448     54      567      3         706  37      77       1       3
Flux                   577      29      86       1         206  103     150      0       2
Gadfly                 454      233     44       24        60   44      47       0       2
Gen                    1000     286     202      23        293  3       191      0       2
Genie                  55       35      1        4         8    1       6        0       0
IJulia                 174      50      51       8         54   6       0        0       5
JuMP                   705      250     126      37        168  11      106      3       4
Knet                   160      13      26       2         66   24      29       0       0
Plots                  1745     464     234      143       485  232     164      2       21
Pluto                  287      137     24       29        68   16      9        0       4

Partial is the new column; the first column (Methods) should be the sum of all the rest.

So, for some packages (e.g. Gadfly, Genie, JuMP, Pluto) fully enumerated instances dominate both partially enumerated and Any (when we see Any in the signature, we just bail out). But for some others, the picture is the opposite (esp. DifferentialEquations, Flux, Knet), and some are in between (Gen, IJulia, Plots).

I’m not sure what more to do with it at this point. Maybe it is a sign that we need, after all, some strategy for Any/Existentials (by the way, Any is essentially a trivial case of an existential: it’s equivalent to T where T). I’m thinking of starting to match dynamic analysis and see if there’s any interesting patterns there.

1. First, a sanity check is good: all stable methods according to the enumeration-based approach should show as stable in dynamic.
2. Next, it’s interesting if Unstable according to enumeration show up as unstable during dynamic analysis and if not why: whether it’s a the test suite that is overly optimistic and doesn’t try the method on an “unfortunate” input, or that enumeration goes into weird corner cases (from examples, I remember that Bool arithmetic is weird, and clients can rightly desire to not bother about it).
3. Speaking of a better strategy for Any: one idea is to only take the types that were used in tests and warn about it… This is, of course, very limiting, but perhaps better than nothing. From users perspective, they could choose whether to make the annotation more precise or to settle on the set of cases important to them. Worth remembering that our procedure is not sound under open world: we only enumerate types that we know about at the current point.