Releases: CenterForCollectiveLearning/DIVE-backend
DIVE Update (4/2/16 - 4/10/16)
The latest development version on staging (please use this to test):
Staging: staging.usedive.com. (password: macro)
For access to all projects, login with the credentials (username: admin, password: password)
For any bug reports or feature requests, please e-mail us personally or at dive@media.mit.edu.
[Design] Landing Page Redesign + Flow
After we redesigning the within-project experience, our landing page seemed a mile behind. We approached the redesign with three goals: 1) to make the workflow clearer to users, and 2) to remove unnecessary cruft, and 3) to make the entire app more unified while still showing a distinction between top-level and within-project pages.
[Feature] Progressive loading of visualization recommendations
Last week we introduced the four different types of visualization recommendations. This allowed us to have a more principled approach to presenting our recommendations to a user. However, we got rid of the UI allowing user control over recommendation types, which was unclear. Instead, we implemented progressive loading of different visualization types, which is more obvious and efficient.
Bug fixes and small features
We dedicated a lot of time this week to squashing bugs, including:
- Expanding the set of viable datasets by being stricter about field coercion
- Handling N/A values better
- Handling more concurrent connections to our database
- Allowing username or e-mail sign-in
Next Up
- Finish regression builder
- Finish analysis export
- Finally include a shareable link to exported visualizations
Other items on the docket include:
- Designing and implementing a workflow for detecting and fixing dirty datasets
- Speeding up field selection on the visualization page
- Including more models on the regression page
- Allowing users to export visualizations with configuration and conditional changes
- Allow users to select the maximum number of points shown on scatterplots
DIVE Update (3/18/16 - 4/1/16)
The latest development version on staging (please use this to test):
Staging: staging.usedive.com. (password: macro)
For access to all projects, login with the credentials (username: admin, password: password)
For any bug reports or feature requests, please e-mail us personally or at dive@media.mit.edu.
[Feature] Modular Visualization Recommendation / Enumeration
Up until now, our visualization recommendations were not well organized on the front-end. That caused three problems for a user: lack of transparency about how DIVE works, difficulty sorting through visualizations, and long wait times for fetching visualizations.
In this latest version, we've broken down visualization recommendation into four cases. Given a user's field selections, we have visualizations that are either 1) exact matches including every selected field, 2) close matches including 2+ selected fields, 3) individual matches including only one field, and 4) expanded matches including one or more selected fields with non-selected fields.
For now, the first three cases are selected by default, and are fully under a user's control. Going forward we're going to think of more performant (and more understandable) user control.
Exact matches:
Close, individual, and expanded matches:
Next up:
- Encoding recommendation type selection in URL
- Progressive loading of visualizations in different cases
- Redesigning user control
- Modularization of calls (almost done)
[Feature] Project editing + deletion + navigation.
Now that projects are tied to users, we ran into the problem of users having tons of projects named the same thing. Now in the top left hand corner of projects, users can edit a project's properties (name and description for now, with more to come), and also navigate between projects.
[Design] Visualization Aesthetics
We've changed the palette assigned to visual elements within a visualizations such that visualizations involving similar fields are the same color. This increases usability on the gallery page.
Also, we've unified the design of most visualizations, with respect to labels, fonts, and other attributes.
[Feature] Histograms (for real)
Histograms were previously mis-used bar charts, but now are their own entity given that they are conceptually distinct. On the surface, this means that axis ticks are now in between bars. But a user can also now filter for histograms and bar charts separately.
TODO:
- Fix sorting for histograms
[WIP] Analysis export
We're 80% of the way done with being able to export regressions, correlations, and summaries, such that they can be incorporated into composed narratives.
Next Up
In preparation for beta user testing in the coming weeks, and member's week, we aim in the next few weeks to:
- Allow exporting of analyses
- Redesign the landing page flow
- Finally include a shareable link to exported visualizations
- Begin working on a visualization builder
Other items on the docket include:
- Designing and implementing a workflow for detecting and fixing dirty datasets
- Speeding up field selection on the visualization page
- Including more models on the regression page
- Allowing users to export visualizations with configuration and conditional changes
- Allow users to select the maximum number of points shown on scatterplots
DIVE Update (3/10/16 - 3/17/16)
The latest development version on staging (please use this to test):
Staging: staging.usedive.com. (password: macro)
For access to all projects, login with the credentials (username: admin, password: password)
For any bug reports or feature requests, please e-mail us personally or at dive@media.mit.edu.
[Redesign] Within-project design and structure
DIVE has looked the same for several months now. While it was good to focus on infrastructure and feature development, it's time for a new face. We aimed for a redesign that:
- Makes the work flow and structure of DIVE more apparent for new users
- Visually separates different concerns (e.g. changing configuration vs. changing entire modes)
- Lends itself to an intuitive walkthrough system
- Is more visually appealing
We've spent a while thinking through a design that meets these criteria. The structure is: all project-level navigation is on the left (what are you doing), all content-level options are on the top (what do you see), and all configuration is on the right (how are you seeing it). It's not radically different or novel, but we had to change a lot under the hood, which gives us a better base to build from. While there's still more work to do, we're happy to say that this is now what DIVE looks like:
Full project flow
Dataset Inspection
Visualization Exploration
Correlation
Narrative Composition
Next up is to continue iterating on the project pages, and to redesign the landing page.
[Feature] Authentication and Accounts
We've put off user accounts and authentication for a while, but it's finally here! It's a thankless job, and prone to mistakes, but we bit the bullet and now have the base of a secure system implemented. We are associating projects to users with varying levels of access. Users can create private projects and also have access to specific preloaded projects. Preloaded projects themselves are either public or scoped to users.
Full login flow
Login Page
Registration Page
There's still a lot to do here:
- Implementing access groups
- Implementing anonymous users that can create projects but must register after a certain point
- Allowing users themselves to allow access to different parts of their projects
- Integrating OAuth
Next Up
In preparation for beta user testing in the coming weeks, we aim in the next few weeks to:
- Allow exporting of analyses
- Redesign the landing page flow
- Allow editing projects (name, description, access, etc)
- Finally include a shareable link to exported visualizations
- Begin working on a visualization builder
Other items on the docket include:
- Designing and implementing a workflow for detecting and fixing dirty datasets
- Speeding up field selection on the visualization page
- Including more models on the regression page
- Fixing bar chart axis sorting bug
- Allowing users to export visualizations with configuration and conditional changes
- Allow users to select the maximum number of points shown on scatterplots
In terms of use cases, we're really looking forward to working with the full clinical dataset, and possibly build bespoke visualizations.
DIVE Update (2/19/16 - 2/26/16)
DIVE is located at two password-protected sites, with the latest development version on staging (please use this to test) and the weekly stable release on the stable site:
Staging: staging.usedive.com. (password: macro)
Stable: usedive.com. (password: macro. Limited datasets)
For any bug reports or feature requests, please e-mail us personally or at dive@media.mit.edu.
[Feature] Correlation
Four functionalities form the core of DIVE's analysis capabilities: summary statistics, correlation, comparison, and regression. Each mode allows the user to explore different statistical features in their dataset. In the past week, we completed the majority of the correlation functionality. So for any selection of quantitative fields, a user receives a correlation matrix and corresponding scatterplots with fitted lines.
Testing link: http://staging.usedive.com/projects/1/datasets/1/analyze/correlation
We select all fields by default to show the largest possible correlation matrix, given that correlation is a relatively cheap and fast operation. The remaining tasks include:
- Choosing a better color palette
- Speeding up scatterplots (e.g. not recalculating every time)
- Thinking through how to best present the scatterplots
- Allows users to export correlation matrices or scatterplots
[Feature] Compose Export
The obvious question after completing a document on the Compose page is: what now? So for every document, there is a corresponding narrative page, which is a pared-down, optimized, shareable form of that document. We don't have an easy way to access that link from the compose page yet (which will be easy), but they're located at /narrative/:documentID.
Testing link: http://staging.usedive.com/narrative/1
Bug Fixes / Small Features (a lot)
Most of our time this past week was actually focused on squashing bugs:
- Starring on visualization gallery view
- Proper visualization type filters on gallery page
- Proper visualization type filters on single-visualization page
- Smarter histogram decimal formatting
- Proper histogram sorting
- Correct axis labels for nested visualizations
- Joining working
- Pivot button working
- Showing summary table column totals
Next Up
- Correlating ad spend and sales data (ensuring the features meet the usecase)
- Full redesign
- Walkthrough features
- Back-end optimization
DIVE Update (2/10/16 - 2/18/16)
DIVE is located at two password-protected sites, with the latest development version on staging (please use this to test) and the weekly stable release on the stable site:
Staging: staging.usedive.com. (password: macro)
Stable: usedive.com. (password: macro. Limited datasets)
For any bug reports or feature requests, please e-mail us personally or at dive@media.mit.edu.
[Feature] Compose
Since we've worked on DIVE, we've wondered about the best way for users to share their results. We already support sharing individual visualizations, but what if a user wants to share multiple visualizations and include context? So on the new Compose page, we let users create long scrolling narratives like the Observatory of Economic Complexity country profiles:
The workflow is: 1) star visualizations (and, in the future, analyses), 2) go to Compose and click starred visualizations to insert them as blocks on a document, and 3) change the content of blocks or the format and size of visualizations:
There's a lot more work to do, but we've finished the first 80%:
- Allowing visualization saving
- Creating and deleting documents
- Basic content + visualization formatting on blocks
- Auto-saving and loading documents
We'd love to have everyone's feedback on the interaction and features! Next up are mostly sharing and aesthetic features, but also more interactions:
- Shareable links to documents (mostly done)
- Exporting to PDF
- Allowing users to choose different styles/themes for their documents
- Rearranging blocks
- More text formatting options
Bugs we're aware of:
- Title truncation
- Inability to navigate from compose back to visualization or analysis without first returning to datasets page
[Compatibility] IE11
While we develop on Chrome for Mac, DIVE nows works smoothly and almost completely on IE11+ (tested on Windows 7). There are a few remaining styling differences that we will address.
Next Up
- Compose export, styling, interaction features
- Correlating ad spend and sales data
- Full redesign
DIVE Update (2/1/16 - 2/9/16)
DIVE is located at two password-protected sites, with the latest development version on staging (please use this to test) and the weekly stable release on the stable site:
Staging: staging.usedive.com. (password: macro)
Stable: usedive.com. (password: macro. Limited datasets)
For any bug reports or feature requests, please e-mail us personally or at dive@media.mit.edu.
[Feature] Conditionals / Filters
We implemented categorical and quantitative filters on single visualizations, as well as the ability to combine conditionals with AND/OR. Previously we only allowed single categorical conditionals (e.g. only sales in DIVISION = ASIA).
Testing link: http://staging.usedive.com/projects/6/datasets/17/visualize/builder/2243
[Feature] Binning
We implemented four more procedural binning rules in addition to the previous, default Freedman binning rule. Different rules work better for certain distributions (e.g. Doane's formula for non-normal data), while others (like Square-root) are faster.
We also allow users to manually specify the number of equal-sized bins to use.
Testing link: http://staging.usedive.com/projects/1/datasets/1/visualize/builder/1016
[Feature] Summary Tables + Marginal Values
We've formatted field summaries into a grid. Upon selecting a field, marginal value tables are shown.
Testing link: http://staging.usedive.com/projects/3/datasets/3/analyze/summary
Testing link: http://staging.usedive.com/projects/4/datasets/60/analyze/summary
[Feature] Error messaging
For every asynchronous task (dataset upload, transformation, visualization), if an error occurs we now return server-side stack traces that are logged to the console. This reduces hanging for the user, and makes debugging easier for us.
[Performance] Asynchronous dataset transformation
We moved converted transformations (pivoting, reducing, and joining) from synchronous to asynchronous processes. This allows us to offload computation onto worker processes, and prevents request time-out for the user.
[Performance] Optimization
Serialization
We've reduced serialization time (converting server-side returns into a browser-usable format) up to 50%, using less type checks and casts.
Caching intermediate data frames
If multiple visualizations using the same intermediate grouped data frames, we cache the intermediate data frames to eliminate the biggest bottleneck in mapping visualizations specs to visualization data.
Bug fixes
- Fixed decimal formatting, so no more ridiculously long numbers.
- Fixing hanging on large dataset transformations
Next Up
- Visualization configuration
- Minimum compose
- Correlating ad spend and sales data