Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start of Sprint 2 #49

Open
wants to merge 69 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
830d2ec
Formatting
sras Nov 4, 2016
a43ee57
Insert records directily with out having to convert to PGW types
sras Nov 4, 2016
45c7a2d
Use the records directly without converting to thier PG counterparts …
sras Nov 4, 2016
19b697b
Fix issue when dealing with enum field with default value
sras Nov 5, 2016
f6a52f0
Add housekeeping fields to Tenant
sras Nov 7, 2016
46ba90a
Add housekeeping fields to remaining models
sras Nov 7, 2016
e7fef1e
Update housekeeping columns without lens
sras Nov 7, 2016
bc4b2df
Fix postgresql/haskell conversion issue and start using Lens
sras Nov 7, 2016
4e25c7c
Made role api to use create_item function that uses lenses to deal wi…
sras Nov 7, 2016
f11a655
Convert more api function to use lenses
sras Nov 8, 2016
bb234be
Fix json generation
sras Nov 8, 2016
785c7d7
Implementing generic update function
sras Nov 8, 2016
f286b54
Generic update function type checks now
sras Nov 8, 2016
4781ca8
Refactor
sras Nov 8, 2016
e16f7a0
Convert everything to use lens
sras Nov 8, 2016
df887ed
Split the generic insert/update function in a separate module
sras Nov 9, 2016
8308879
Fixed issue
sras Nov 9, 2016
74e8743
Refactor
sras Nov 9, 2016
a8c94fd
Make created fields optional and updated field required for writing
sras Nov 9, 2016
74eff26
Removed unused AllowAmbiguousTypes extension
sras Nov 9, 2016
290cc61
Reenable create functions
sras Nov 9, 2016
945ef75
Merge OpaleyeTypes with OpaleyeDef
sras Nov 10, 2016
e770e4c
Conver names to camel case and drop id from update functions
sras Nov 10, 2016
04fc93d
Make id fields read only and remove unused Default instances
sras Nov 10, 2016
93b6037
Fix warnings
sras Nov 10, 2016
3734c41
Stylize
sras Nov 10, 2016
07304e8
Refactor to use listToMaybe functions
sras Nov 10, 2016
1a298cb
Start of Audtable api implementation
sras Nov 12, 2016
dd2d971
Convert all api function to use AuditM
sras Nov 12, 2016
bd209c5
Add wrapper to opaleye functions that runs in AuditM
sras Nov 12, 2016
1004ca1
Add functions that does raw db updates in AuditM
sras Nov 14, 2016
4b32bdd
Rename AuditM to AppM
sras Nov 15, 2016
ea708d7
Added TH functions to create auditable lenses
sras Nov 15, 2016
6f80c26
Generate typeclasses for the auditable wrapper using TH
sras Nov 16, 2016
f940de5
Clean up warnings and start conversion of audit log to json Value type
sras Nov 16, 2016
18e2dfe
Change the audit log to use the json Value object
sras Nov 16, 2016
c6f7d14
Added type syns for AuditM wrapper
sras Nov 16, 2016
03a33cc
Convert all the api function to run in AppM
sras Nov 17, 2016
7b4ba29
Use Auditable models in api functions
sras Nov 17, 2016
6ebf760
Insert log to db on update calls
sras Nov 17, 2016
79d5114
Test case for audit logs
sras Nov 17, 2016
f02c93a
Fix build issues
sras Nov 18, 2016
a995b2d
Fix typo in TH function name
sras Nov 20, 2016
4a38268
Adding comments
sras Nov 21, 2016
14e2d8a
Ammende TH functions to take types that are not sysnonyms into account
sras Nov 25, 2016
4ce8966
Added schema files
sras Nov 29, 2016
4b3fca4
Added list tenants endpoints for load testing
sras Nov 30, 2016
82d2d78
Send mail via sendgrid
sras Dec 5, 2016
69f172f
Added email templates and linked mail sending from create tenant json…
sras Dec 7, 2016
aa38a20
Add inline attachment embedding using cid method
sras Dec 7, 2016
95f65e2
Fix double quoting of links and add hardcoded key
sras Dec 7, 2016
27cf3b3
Fix double quote issue with quasiquoter and add a hardcoded key to th…
sras Dec 7, 2016
480119f
Send mail from another thread and code refactor
sras Dec 7, 2016
741ff7d
Add user services
sras Dec 8, 2016
87b9674
Review modifications
sras Dec 9, 2016
81f6fab
Cleaned up stack.yaml
sras Dec 9, 2016
0a8c997
Use the updated mime-email library
sras Dec 12, 2016
29a8a80
Wrapped the app monad around ExceptT to enable handling of uncaught e…
sras Dec 12, 2016
65c5ccc
Trying to add request body to the error logs
sras Dec 13, 2016
52c102b
Use a helper function to fetch env info and log it in case of an exce…
sras Dec 13, 2016
0ce8cbf
Add request info to the log
sras Dec 13, 2016
cd991a1
Added airbrake source
sras Dec 14, 2016
e788e65
cabale file update
sras Dec 14, 2016
6ec820b
Wrap the auditable types
sras Dec 15, 2016
f56c9c5
Got the refactor to compile
sras Dec 15, 2016
caa39c0
Rename files
sras Dec 16, 2016
117f8db
Moved code to a separate package AppCore
sras Dec 16, 2016
54dcd2e
Added removeAuditableRow function to api base
sras Dec 16, 2016
39dbab3
Hide stuff in AppCore module
sras Dec 16, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions SpockOpaleye/SpockOpaleye.cabal
Original file line number Diff line number Diff line change
Expand Up @@ -15,26 +15,34 @@ cabal-version: >=1.10

library
hs-source-dirs: src
exposed-modules: Lib,
exposed-modules: DataTypes,
ApiBase,
TenantApi,
UserApi,
RoleApi,
OpaleyeDef,
CryptoDef,
JsonInstances,
Validations,
DataTypes
TH
build-depends: base >= 4.7 && < 5
,product-profunctors
,profunctors
,bytestring
,opaleye
,time
,postgresql-simple
,bcrypt
,text
,lens
,mtl
,transformers
,time
,vector
,Spock >=0.11
,aeson
,unordered-containers
,template-haskell
default-language: Haskell2010

executable SpockOpaleye-exe
Expand All @@ -46,10 +54,14 @@ executable SpockOpaleye-exe
, SpockOpaleye
, Spock >=0.11
, mtl
, transformers
, lens
, text
, time
, bcrypt
, vector
, aeson
, unordered-containers
default-language: Haskell2010

test-suite SpockOpaleye-test
Expand Down
67 changes: 52 additions & 15 deletions SpockOpaleye/app/Main.hs
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE TemplateHaskell #-}

module Main where

import Database.PostgreSQL.Simple
import DataTypes
import JsonInstances ()
Expand All @@ -11,7 +13,13 @@ import Validations
import Web.Spock
import Web.Spock.Config

import Control.Monad.Reader
import Control.Monad.Writer
import qualified Data.Text as T
import Data.Time
import Prelude hiding (id)
import Control.Lens
import CryptoDef

data MySession =
EmptySession
Expand All @@ -30,19 +38,48 @@ main = do
DummyAppState
runSpock 8080 (spock spockCfg app)

runAppM :: AppM a -> Connection -> IO a
runAppM x conn = do
user <- getTestUser
(item, lg) <- runReaderT (runWriterT x) (conn, Just $ auditable getTestTenant, Just $ auditable user)
putStrLn lg
return item

getTestTenant :: Tenant
getTestTenant = Tenant (TenantId 44) tz tz "tjhon" "John" "Jacob" "john@gmail.com" "2342424" TenantStatusNew Nothing "Bo domain"
where
tz = UTCTime {
utctDay = ModifiedJulianDay {
toModifiedJulianDay = 0
}
, utctDayTime = secondsToDiffTime 0
}

getTestUser :: IO User
getTestUser = do
Just password <- bcryptPassword "adsasda"
return $ User (UserId 27) tz tz (TenantId 44) "John" password (Just "2342424") (Just "asdada") UserStatusActive
where
tz = UTCTime {
utctDay = ModifiedJulianDay {
toModifiedJulianDay = 0
}
, utctDayTime = secondsToDiffTime 0
}


app :: SpockM Connection MySession MyAppState ()
app = do
post ("tenants/new") $
do maybe_tenant_incoming <- jsonBody
maybe_newtenant <-
case maybe_tenant_incoming of
Just incoming_tenant -> do
result <-
runQuery (\conn -> validateIncomingTenant conn incoming_tenant)
case result of
Valid -> runQuery (\conn -> create_tenant conn incoming_tenant)
_ -> return Nothing
Nothing -> return Nothing
case maybe_newtenant of
Just tenant -> json tenant
_ -> json $ T.pack "Tenant not created"
do maybeTenantIncoming <- jsonBody
case maybeTenantIncoming of
Just incomingTenant -> do
result <- runQuery $ runAppM $ validateIncomingTenant incomingTenant
case result of
Valid -> do
newTenant <- runQuery $ runAppM $ createTenant incomingTenant
let updatedTenant = newTenant & firstname .~ "Jake"
runQuery $ runAppM $ updateTenant updatedTenant
json newTenant
_ -> json $ T.pack "Validation fail"
Nothing -> json $ T.pack "Unrecognized input"
157 changes: 157 additions & 0 deletions SpockOpaleye/src/ApiBase.hs
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
{-# LANGUAGE FlexibleContexts #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE FunctionalDependencies #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE TemplateHaskell #-}

module ApiBase where

import Control.Lens
import Control.Monad.IO.Class
import Control.Monad.Reader
import Control.Monad.Writer
import qualified Data.Profunctor.Product.Default as D
import Data.Time (UTCTime, getCurrentTime)
import DataTypes
import Opaleye
import OpaleyeDef
import qualified Data.Text as T
import GHC.Int
import Prelude hiding (id)
import TH
import Data.Aeson (Value(..))
import JsonInstances ()

makeAudtableLenses ''Role
makeAudtableLenses ''Tenant
makeAudtableLenses ''User

auditLog :: String -> AppM ()
auditLog = tell

removeRawDbRows :: Table columnsW columnsR -> (columnsR -> Column PGBool) -> AppM GHC.Int.Int64
removeRawDbRows table matchFunc = do
conn <- getConnection
liftIO $ runDelete conn table matchFunc

createDbRows :: (Show columnsW, D.Default QueryRunner columnsR haskells)
=> Table columnsW columnsR -> [columnsW] -> AppM [haskells]
createDbRows table pgrows = do
auditLog $ "Create : " ++ (show pgrows)
conn <- getConnection
liftIO $ runInsertManyReturning conn table pgrows (\x -> x)

updateDbRow :: (Show columnsW, HasId columnsR (Column PGInt4)) => Table columnsW columnsR -> Column PGInt4 -> columnsW -> AppM columnsW
updateDbRow table row_id item = do
auditLog $ "Update :" ++ (show item)
conn <- getConnection
_ <- liftIO $ runUpdate conn table (\_ -> item) (matchFunc row_id)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we be using runUpdateReturning here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. That would be ideal. But since we already hold the record in its updated state, I thought, why do the additional fetching. If there was a failure with update, we will get an exception anyway, right?

I am not sure, but I think the use case of "update returning" would be if we are doing an operation on existing value, like "update product set quantity = quantity + 1", and in that case, returning the new values can save an additional select.

But here we are not doing anything like that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel kts better to use updateReturning because it allows us to represent the latest state of the row in the DB, which might be different than what is in the memory, possibly due to DB triggers.

return item
where
matchFunc :: (HasId cmR (Column PGInt4)) => (Column PGInt4 -> cmR -> Column PGBool)
matchFunc itId item' = (item' ^. id) .== itId

createRow ::(
Show incoming,
Show columnsW,
HasCreatedat columnsW (Maybe (Column PGTimestamptz)),
HasUpdatedat columnsW (Column PGTimestamptz),
D.Default Constant incoming columnsW, D.Default QueryRunner returned row)
=> Table columnsW returned -> incoming -> AppM row
createRow table item = do
auditLog $ "Create : " ++ (show item)
currentTime <- liftIO $ fmap pgUTCTime getCurrentTime
let itemPg = (constant item) & createdat .~ (Just currentTime) & updatedat .~ (currentTime)
fmap (head) $ createDbRows table [itemPg]

updateRow :: (
Show columnsW
, Show haskells
, HasUpdatedat haskells UTCTime
, D.Default Constant haskells columnsW
, D.Default Constant itemId (Column PGInt4)
, HasId haskells itemId
, HasId columnsR (Column PGInt4)
)
=> Table columnsW columnsR -> haskells -> AppM haskells
updateRow table item = do
auditLog $ "Update : " ++ (show item)
let itId = item ^. id
currentTime <- liftIO getCurrentTime
let updatedItem = (putUpdatedTimestamp currentTime) item
_ <- updateDbRow table (constant itId) (constant updatedItem)
return updatedItem
where
putUpdatedTimestamp :: (HasUpdatedat item (UTCTime)) => UTCTime -> item -> item
putUpdatedTimestamp timestamp = updatedat .~ timestamp

updateAuditableRow :: (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be in a DB transaction? Have we solved for DB transaction yets?

Show columnsW
, HasUpdatedat (Auditable haskells) UTCTime
, D.Default Constant haskells columnsW
, D.Default Constant itemId (Column PGInt4)
, HasId (Auditable haskells) itemId
, HasId columnsR (Column PGInt4)
)
=> Table columnsW columnsR -> Auditable haskells -> AppM (Auditable haskells)
updateAuditableRow table audti = do
--auditLog $ "Update : " ++ (show item)
let itId = audti ^. id
currentTime <- liftIO getCurrentTime
let updatedItem = (putUpdatedTimestamp currentTime) audti
let Auditable { _data = item, _log = _log} = updatedItem
_ <- updateDbRow table (constant itId) (constant item)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we really be ignoring the result? Isn't it better to use UPDATE... RETURNING * and returning the updated record wrapped in a fresh auditable wrapper?

insertIntoLog table itId "" _log
return audti
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the in-memory audit log being blanked out after a successful update or insert?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. I completely missed that. Will do.

where
putUpdatedTimestamp :: (HasUpdatedat item (UTCTime)) => UTCTime -> item -> item
putUpdatedTimestamp timestamp = updatedat .~ timestamp

insertIntoLog :: (
D.Default Constant item_id (Column PGInt4)
) => Table a b -> item_id -> T.Text -> Value -> AppM ()
insertIntoLog table auditable_id summary changes = do
case table of
Table table_name _ -> do
Just tenant <- getCurrentTenant
Just user <- getCurrentUser
conn <- getConnection
let tenant_id = tenant ^. id
let user_id = user ^. id
_ <- liftIO $ runInsertMany conn auditTable [(
(),
constant tenant_id,
Just $ constant user_id,
Just $ pgBool False,
constant auditable_id,
pgStrictText $ T.pack table_name,
pgStrictText $ summary,
constant changes,
Nothing)]
return ()
_ -> error "Unsupported Table constructor"

removeRow :: (
Show haskells
, D.Default Constant itemId (Column PGInt4)
, HasId haskells itemId
, HasId columnsR (Column PGInt4)
) => Table columnsW columnsR -> haskells -> AppM GHC.Int.Int64
removeRow table item = do
auditLog $ "Remove : " ++ (show item)
conn <- getConnection
liftIO $ do
runDelete conn table $ matchFunc $ item ^. id
where
matchFunc :: (
HasId columnsR (Column PGInt4),
D.Default Constant itemId (Column PGInt4)
) => (itemId -> columnsR -> Column PGBool)
matchFunc itId item' = (item' ^. id) .== (constant itId)

readRow :: (D.Default QueryRunner columnsR haskells) =>
Opaleye.Query columnsR -> AppM [haskells]
readRow query' = do
conn <- getConnection
liftIO $ runQuery conn query'
4 changes: 4 additions & 0 deletions SpockOpaleye/src/CryptoDef.hs
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import Data.Text
import Data.Text.Encoding
import Database.PostgreSQL.Simple.FromField
import Opaleye
import Data.Aeson (ToJSON(..), Value(..))

newtype BcryptPassword =
BcryptPassword ByteString
Expand All @@ -38,3 +39,6 @@ instance FromField BcryptPassword where

instance QueryRunnerColumnDefault PGBytea BcryptPassword where
queryRunnerColumnDefault = fieldQueryRunnerColumn

instance ToJSON BcryptPassword where
toJSON _ = String ""
Loading