Haskell Relational Records (HRR) - initial sprint #50

mgmeier · 2016-11-04T23:35:04Z

This sprint's focus is the evaluation of HRR as a DB abstraction. The result should implement the domain API as per specs for Tenant, User and Role data types, encompassing the following:

CreateRole, DeleteRole, AssignRole, RemoveRole, GetRole
CreateUser, UpdateUser, ActivateUser, DeactivateUser, GetUser
CreateTenant, UpdateTenant, ActivateTenant, DeactivateTenant, GetTenant

Furthermore, the result should serve as an example how to implement and/or deal with the following topics in HRR:

DB model setup and migrations in HRR
Mapping to and from Haskell data types
Mapping of special types to and from Haskell: PostgreSQL enums and arrays, JSONB
Expressive power of JOINs in HRR (e.g. fetch a list of Users along with their Roles)
Treatment of housekeeping data in HRR (timestamp columns, transaction log)

Time estimate: 20 hours

mgmeier · 2016-11-12T01:25:04Z

@saurabhnanda The domain API is now up and running as described in the PR, minus the topics still open (enums, arrays, jsonb).

This would be a good time for you to review the code, comment some design choices, look at the overall resulting workflow etc. etc. so that I can adjust the code to your feedback, which will be highly appreciated.

saurabhnanda · 2016-11-13T14:42:47Z

@mgmeier ran into the very first (expected) issue. The HRR TemplateHaskell needs to connect to PG in the compilation step itself. How do I configure the PG credentials?

saurabhnanda · 2016-11-13T15:13:20Z

@mgmeier never mind. Figured it out. Btw, hitting a new compile error. Let's discuss it on Gitter.

saurabhnanda

Does the PG text or varchar always map to Haskell's String? What's the easiest way to make it map to Data.Text?
What does Relation x y really mean? You can use relationalQuery to convert Relation x y -> Query x y where Query x y = Query {untypeQuery :: String}. The type parameters x y in the entire chain seem to be phantom types. So, is Relation x y basically some form of SQL represented in a Haskell-friendly AST (or DSL)?
Consequently, what's the difference between SimpleQuery x y and Relation x y?
Has audit logging (transaction logging) been completed?

saurabhnanda · 2016-11-15T17:46:30Z

RelationalRecord/src/Relations/Tenant.hs

+    , uEmail        :: Maybe String
+    , uBOD          :: Maybe String
+    , uStatus       :: Maybe Int32
+    , uOwnerId      :: Maybe (Maybe Int32)


Is it a good idea to conflate a no-op with an update to NULL? eg. say this column were initially non-nullable, and thus, had the type Maybe Int32. And we had some call-sites where this field was being set as Nothing, assuming that it would map to an sql NULL. Then we made this column nullable, thus changing its type to Maybe (Maybe Int32). The call sites would still type-check, but they would result in completely unexpected behaviour.

What are your thoughts on the following: Nullable (Maybe Int32)

Maybe a is the HRR derivation of a nullable type a on the db.

The conflation problem you adressed can actually not occur: The outer Maybe signals the presence vs. absence of a value for the update. So, no value assignment whatsoever takes place in the update query for a value of Nothing. The inner Maybe can then be set to Just Nothing to update with sql NULL. Thus, the call-sites that never update some field (Nothing) stay correct, the ones updating a column that has been made nullable fail to type-check, as it should be.

You're right however that the mechanism is not quite clear at first glance. I've changed that by replacing the outer Maybe with a different, more expressive type.

saurabhnanda · 2016-11-15T17:56:30Z

RelationalRecord/src/Relations/User.hs

+    }
+$(makeRecordPersistableDefault ''UserInsert)
+
+piUser :: Pi Users UserInsert


Just wondering if it is possible to avoid this boilerplate if we're fine with having the same data-structure/type for, both, reading from the DB and writing to the DB?

HRR derived types have all strict fields; an "identity projection" of such a type would require us to assign a (non-existent) primary key value where present to construct the record -- which will then clash on the DB when we run the insert. For tables without PK, this is absolutely possible; I've done so for the join-through table users_roles, it looks like this:

assignRole :: Insert UsersRoles assignRole = typedInsert (tableOf usersRoles) (defineDirectPi [0, 1])

saurabhnanda · 2016-11-15T17:58:33Z

RelationalRecord/src/Relations/User.hs

+
+userUpdate :: UserUpdate
+userUpdate = UserUpdate
+    Nothing Nothing Nothing Nothing Nothing Nothing


Can be replaced with Data.Default?

saurabhnanda · 2016-11-15T18:07:14Z

RelationalRecord/src/Relations/User.hs

+
+-- UPDATES
+
+data UserUpdate = UserUpdate


how do we handle the case where certain columns need to be protected from "blind" updates? eg. password column? While it can be inserted initially, it should not be updatable later via a variadic update? One should be forced to call a special function for changing the password, which would also trigger a side-effect of notifying the user by email?

the solution would be to remove that field from the variadic update type completely.
for those updates which require special attention / a special code-path (you mentioned side effects), a specialized, explicit Update relation and corresponding domain API function would be in order.

saurabhnanda · 2016-11-15T18:08:19Z

RelationalRecord/src/Relations/User.hs

+    User.lastName'  <-#? uLastName
+    User.status'    <-#? uStatus
+
+    (phTStamp, _)   <- placeholder (\tStamp -> User.updatedAt' <-# tStamp)


Need some comments on what exactly is happening here? How is the current timestamp actually getting assigned?

Also, how is createdAt being handled?

The helper type for this update is type TimestampedUpdate = Update (DBTime, PKey) . This means, the generated query will have two placeholders, one for the timestamp, one for the primary key to update. The other updated values are rendered in the query as literals. So when rendering the query DSL, we get something like:
UPDATE users SET first_name='Foo', last_name='Bar', updatedAt=? WHERE id=?, where substitution of these placeholders happens one step later.

In HRR, these type parameters for e.g. an Update a have to be realized via the placeholder syntax. It's also possible to apply some datatype argument first, and use the literal inside the query, e.g. DBTime -> PKey -> Update (). One can decide on the context which is preferable.

Either way, getting a current timestamp from the system clock is done in the DBInterface, which in turn then runs the update query.

createdAt timestamps are always left to the DB, where they have default value current_timestamp in the schema.

Possible to discuss this on gitter chat?

saurabhnanda · 2016-11-15T18:20:52Z

RelationalRecord/src/Relations/Role.hs

+    return  a
+
+-- given a user, get all his/her roles (inner join)
+getRoles :: Relation Users Roles


Can you please give a usage example of this function? Is this just a DSL/AST representation of an inner-join between roles and users via the user_roles join-through table, with an unapplied user argument? What is the type of the unapplied user argument? Is it the PKey of the User table or the actual User record? How can you tell which arguments are unapplied from the function signature?

When this query is executed, what does it evaluate to? A list of Roles?

How would one write the following -- get all users along with each user's role(s), where the user has been created after a given timestamp AND the user's email ID belongs to a given domain (say, gmail.com or vacationlabs.com)?

Can I compose this Relation with another wheres clause specified outside this function? Can I compose this with another JOIN, say with tenants table?

mgmeier · 2016-11-15T23:15:15Z

I've changed HRR default mapping of PostgreSQL type text to Haskell's Data.Text. Such a custom mapping can easily be specified with HRR's db driver and is done in the module DefineTable.hs.

…jection

mgmeier · 2016-11-18T16:20:51Z

JSONB and ENUM: investigation results

This is the relation used by the HRR driver to get postgres type info; those will be the types considered for generating a corresponding attribute in the derived Haskell type (in pseudo-code, taken from module Database.Relational.Schema.PostgreSQL):

(select * from pg_type)
    wheres $ att ! Attr.atttypid'    .=. typ ! Type.oid'
    wheres $ typ ! Type.typtype'     .=. value 'b'  -- 'b': base type only

    wheres $ typ ! Type.typcategory' `in'` values [ 'B' -- Boolean types
                                                  , 'D' -- Date/time types
                                                  , 'I' -- Network Address types
                                                  , 'N' -- Numeric types
                                                  , 'S' -- String types
                                                  , 'T' -- typespan types
                                                  ]

We can see, JSONB is not amongst them as it is of category 'U' (user-defined).
Also, enums are not considered for mapping, they're of category 'E'; also, they're not a base-type, but an 'e' - enum type.

Conclusion: The HRR library would have to be patched accordingly for HRR to even consider
generating a datatype derivation for those categories in Haskell.

saurabhnanda · 2016-11-18T17:47:20Z

Do you think this is easy to patch in HRR (JSONB and enums)?

On 18 Nov 2016 9:50 pm, "M. G. Meier" notifications@github.com wrote:

JSONB and ENUM: investigation results

This is the relation used by the HRR driver to get postgres type info;
those will be the types considered for a generating a corresponding
attribute in the derived Haskell type (in pseudo-code, taken from module
Database.Relational.Schema.PostgreSQL):

(select * from pg_type)
wheres $ att ! Attr.atttypid' .=. typ ! Type.oid'
wheres $ typ ! Type.typtype' .=. value 'b' -- 'b': base type only
wheres $ typ ! Type.typcategory' `in'` values [ 'B' -- Boolean types
                                              , 'D' -- Date/time types
                                              , 'I' -- Network Address types
                                              , 'N' -- Numeric types
                                              , 'S' -- String types
                                              , 'T' -- typespan types
                                              ]
We can see, JSONB is not amongst them as it is of category 'U'
(user-defined).
Also, enums are not considered for mapping, they're of category 'E'.

Conlusion: The HRR library has to be patched accordingly for HRR to even
consider
generating a datatype derivation for those categories in Haskell.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#50 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABu0bDV2S0BeVoeuTFW4sCZIpviWHE0ks5q_dBjgaJpZM4KqFnU
.

mgmeier · 2016-11-19T20:25:17Z

Audit logging works fine for now; the logic for creating the JSON diff is in the application layer (Haskell), not on the DB.
NB. for the time being, log entries are texts. When I patch HRR for a derivation of jsonb, that'll change. Patching shouldn't be too complicated, as I see it.

mgmeier · 2016-11-20T00:24:51Z

Don't forget to create DB from scratch (schema.sql) and possibly completely eliminate .stack-work/, since I've replaced the source repo for HRR with the fork where my patch is applied.

initial commit; minimal working example

a00f803

mgmeier mentioned this pull request Nov 4, 2016

Relational Records + wai: initial sprint #48

Closed

mgmeier added 7 commits November 5, 2016 18:25

complete db schema and types; start adding relations

f56b8fc

deletes, updates; updatedAt housekeeping

3e5a9ea

inserts; auditlog; refactor

ebd9467

more inserts; WIP auditlog; start domainAPI

e2ea126

expand role API; inner join queries

6f69c56

variadic updates

5ff5b3b

finish insert/update boilerplate; complete domain API

4f9470c

changed json to jsonb in default value

28fb780

saurabhnanda reviewed Nov 15, 2016

View reviewed changes

mgmeier added 3 commits November 15, 2016 22:08

add wrapper type for variadic args; use Data.Default

a51527a

change mapping to use Data.Text as default text type

4eb48e9

fixup!

59f564b

mgmeier added 2 commits November 18, 2016 15:58

jsonDiff helper for auditing; trying direct inserts with identity pro…

e6b88dd

…jection

investigated jsonb and enums

63249c7

mgmeier added 3 commits November 19, 2016 00:59

audit_log entries; insert into ... returning *;

e27c383

multiway-join

3da0f73

refactor DBInterface + DomainAPI; simplify updates; better audit logic

ae634d9

jsonb mapping working w/ patched relational-schemas

8cd48a9

mgmeier added 2 commits November 21, 2016 18:50

mapping of Postgres enum type to Haskell works!

151e03b

retrieve enum values from DB at compile time

fee8b20

mgmeier added 3 commits November 25, 2016 19:06

ENUM deriving working (proof-of-concept)

fc2184c

Merge branch 'master' into feature/RelationalRecord

a505a94

ENUM derivation now working beautifully

c88be35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Haskell Relational Records (HRR) - initial sprint #50

Haskell Relational Records (HRR) - initial sprint #50

mgmeier commented Nov 4, 2016

mgmeier commented Nov 12, 2016

saurabhnanda commented Nov 13, 2016

saurabhnanda commented Nov 13, 2016

saurabhnanda left a comment

saurabhnanda Nov 15, 2016

mgmeier Nov 15, 2016 •

edited

Loading

saurabhnanda Nov 15, 2016

mgmeier Nov 19, 2016

saurabhnanda Nov 15, 2016

mgmeier Nov 15, 2016

saurabhnanda Nov 15, 2016

mgmeier Nov 15, 2016 •

edited

Loading

saurabhnanda Nov 15, 2016

saurabhnanda Nov 15, 2016

mgmeier Nov 18, 2016

mgmeier Nov 18, 2016

saurabhnanda Nov 18, 2016

saurabhnanda Nov 15, 2016

mgmeier commented Nov 15, 2016 •

edited

Loading

mgmeier commented Nov 18, 2016 •

edited

Loading

saurabhnanda commented Nov 18, 2016

mgmeier commented Nov 19, 2016 •

edited

Loading

mgmeier commented Nov 20, 2016

Haskell Relational Records (HRR) - initial sprint #50

Are you sure you want to change the base?

Haskell Relational Records (HRR) - initial sprint #50

Conversation

mgmeier commented Nov 4, 2016

mgmeier commented Nov 12, 2016

saurabhnanda commented Nov 13, 2016

saurabhnanda commented Nov 13, 2016

saurabhnanda left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgmeier Nov 15, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgmeier Nov 15, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mgmeier commented Nov 15, 2016 • edited Loading

mgmeier commented Nov 18, 2016 • edited Loading

JSONB and ENUM: investigation results

saurabhnanda commented Nov 18, 2016

mgmeier commented Nov 19, 2016 • edited Loading

mgmeier commented Nov 20, 2016

mgmeier Nov 15, 2016 •

edited

Loading

mgmeier Nov 15, 2016 •

edited

Loading

mgmeier commented Nov 15, 2016 •

edited

Loading

mgmeier commented Nov 18, 2016 •

edited

Loading

mgmeier commented Nov 19, 2016 •

edited

Loading