DeepOculos: Generative AI For Eye Retina Disease Tracking
by Timothy Gao, Henry Hong, Marcus Koh, Ohm Rajpal, and Henry Lee
Original Retinal Eye Motion Video (data provided by C. Light Technologies):
512x512
Our DCGAN-generated Synthetic Eye Motion Video (with embedded pupil trace statistics):
64x64 (upscaled with cubic interpolation)
DCGAN Training Progress:
- First 1000 Epochs
![Screenshot 2023-11-12 at 10 23 30 PM](https://private-user-images.githubusercontent.com/35588167/282379451-7a328986-45de-4766-bfaa-f50f040e3161.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5NjM1MjksIm5iZiI6MTcyNTk2MzIyOSwicGF0aCI6Ii8zNTU4ODE2Ny8yODIzNzk0NTEtN2EzMjg5ODYtNDVkZS00NzY2LWJmYWEtZjUwZjA0MGUzMTYxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDEwMTM0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWFhMWJjODk3ZWYwZGIwZmExOTI2MDE1MjBmZTcxNzVmNjU5MzljM2U0NWJiY2E5MjBkZGFjMzdmMTVlYTViMzYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.5kfNqv-Ax8TJ4ybBY7UEp1urw9Kyo0cYKznSEVCfEBk)
![Screenshot 2023-11-12 at 10 23 41 PM](https://private-user-images.githubusercontent.com/35588167/282379486-85be834c-2381-4e82-b578-115fc131ffc5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5NjM1MjksIm5iZiI6MTcyNTk2MzIyOSwicGF0aCI6Ii8zNTU4ODE2Ny8yODIzNzk0ODYtODViZTgzNGMtMjM4MS00ZTgyLWI1NzgtMTE1ZmMxMzFmZmM1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDEwMTM0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTg5NGQ3NTYyODFkZmRhN2JhMThiN2U3NmY1ZjExNzA0NjQxYTMxN2RkZDQ3M2QyODE4Mzk0NDE0OGRkNjk0MDAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.pZTlHbYYAiGdr8WqSSC3g_JngNwXZQLTvlVrT4NhrHs)
![Screenshot 2023-11-12 at 10 23 49 PM](https://private-user-images.githubusercontent.com/35588167/282379505-73088109-1453-4845-8cc4-92bad22fd3cd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5NjM1MjksIm5iZiI6MTcyNTk2MzIyOSwicGF0aCI6Ii8zNTU4ODE2Ny8yODIzNzk1MDUtNzMwODgxMDktMTQ1My00ODQ1LThjYzQtOTJiYWQyMmZkM2NkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDEwMTM0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWU2ZGQ2YjM1YzU1NmQ1NWY5MDY4Y2I4OTA4ZDZmMGVlM2JiY2JjMDUyNTkwMDM3MWEwYzYwMTVjNWQ2NWZjZWMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.G_IbiquVmzOgJJ9CgxH9O-wM2Q0d9aSciiheJb8gvGw)
![Screenshot 2023-11-12 at 10 24 00 PM](https://private-user-images.githubusercontent.com/35588167/282379548-d4f136bd-e965-4036-8bb0-831fd7af37e2.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5NjM1MjksIm5iZiI6MTcyNTk2MzIyOSwicGF0aCI6Ii8zNTU4ODE2Ny8yODIzNzk1NDgtZDRmMTM2YmQtZTk2NS00MDM2LThiYjAtODMxZmQ3YWYzN2UyLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDEwMTM0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWNkYjk1MDY4ZTBjYWQxMmI0MGM0MWY3ZDBlZDVhMTZkN2YzYWNhZGM3YmRhMmEwZDAzYjM0OTZkZjcyMmVhZjImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.8xadkdQxKkya21qusOXQHEpwDvyWqbkjgz1hN4sSe1M)
![Screenshot 2023-11-12 at 10 24 19 PM](https://private-user-images.githubusercontent.com/35588167/282379591-5e452afe-e1dd-4b08-81d7-c94e99e28801.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5NjM1MjksIm5iZiI6MTcyNTk2MzIyOSwicGF0aCI6Ii8zNTU4ODE2Ny8yODIzNzk1OTEtNWU0NTJhZmUtZTFkZC00YjA4LTgxZDctYzk0ZTk5ZTI4ODAxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDEwMTM0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWUwNmYxNGQ3ZTA3NmUzODY1YjgyOGY2ODMxNDRjZTBiNDA5MTY0YzM0NDA2NjcxOWZmNTM3MGQ0MTZiZGI4ODAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.nu-fduPcoQ_FxhVPVgbGKfmctq09yj7rm0bp6hwW6Ow)
![Screenshot 2023-11-12 at 10 24 38 PM](https://private-user-images.githubusercontent.com/35588167/282379654-7c31d2ca-b560-40d7-93bc-65989078d1b8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5NjM1MjksIm5iZiI6MTcyNTk2MzIyOSwicGF0aCI6Ii8zNTU4ODE2Ny8yODIzNzk2NTQtN2MzMWQyY2EtYjU2MC00MGQ3LTkzYmMtNjU5ODkwNzhkMWI4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDEwMTM0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThjYzY4NDVkMWY1ODAwZDBhYjgwZWUxYTA3NjMwZTRjZDhjMDQ5MjczMmNmMDhiYzlmZjI5YTQ5YzFkNmU1ODEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.5ygLh5p07sqDxEzLXj1_PD2wwwj9A7tiUVjmlm6FXyc)
- 2000 Epochs
![Screenshot 2023-11-12 at 10 25 10 PM](https://private-user-images.githubusercontent.com/35588167/282379747-7fc79b7d-6d4d-4d7d-a6d5-b5e88a56309e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5NjM1MjksIm5iZiI6MTcyNTk2MzIyOSwicGF0aCI6Ii8zNTU4ODE2Ny8yODIzNzk3NDctN2ZjNzliN2QtNmQ0ZC00ZDdkLWE2ZDUtYjVlODhhNTYzMDllLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDEwMTM0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTlmN2IwZjhmY2UxZDgwZGVjYTNiN2IyZmYyMmU5YjU1Mzc2YjlhYWZiN2QyODRjNmZkODMyYWEwY2E4MWVmODAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.zyauYY5fgSq13h_yj4LHbfQwt3AvB4GD_wdBvJ1KoWo)
- 3000 Epochs
![Screenshot 2023-11-12 at 10 25 20 PM](https://private-user-images.githubusercontent.com/35588167/282379771-e8ee9c6f-8368-4c3e-b6b0-28a0600a37e6.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5NjM1MjksIm5iZiI6MTcyNTk2MzIyOSwicGF0aCI6Ii8zNTU4ODE2Ny8yODIzNzk3NzEtZThlZTljNmYtODM2OC00YzNlLWI2YjAtMjhhMDYwMGEzN2U2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDEwMTM0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTlmZDkwNDBkYmJjNzgzMTg4MWNkYmU4NTFmYWE5MDhmMTVhYmQzNDQ1NjY4MmJlY2EyMDY1ZjdmMTE1NDMzYjUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.-nV2VY6XvzY0PeAYLt9OHoZ09iqKCRPFNDSU7AG_Lkg)
- We view horizontal and vertical location of pupil as independent time series and apply LSTM to generate future prediction for each
- LSTM-generated locations are fed into DCGAN for the pupil trace statistics
![Screenshot 2023-11-12 at 10 11 41 PM](https://private-user-images.githubusercontent.com/35588167/282377689-1846138a-e3a5-4bf3-971e-727fc72057fd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5NjM1MjksIm5iZiI6MTcyNTk2MzIyOSwicGF0aCI6Ii8zNTU4ODE2Ny8yODIzNzc2ODktMTg0NjEzOGEtZTNhNS00YmYzLTk3MWUtNzI3ZmM3MjA1N2ZkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDEwMTM0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTc2NDgyNTIyNGI3NDBmYjQ5Njc1MTI0MjI0OTYxYzM5NWFjYzRlZjg0NjkwZTg0YzcxMzczY2UyMDk1ODZiOGUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ZRM4C_SBP-9NhrAq9WbEfCnHzcpN084U8VfCTMkwIJk)
![Screenshot 2023-11-12 at 10 11 13 PM](https://private-user-images.githubusercontent.com/35588167/282377605-94ab72ba-e1d2-4e5c-b681-bf19164c893a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5NjM1MjksIm5iZiI6MTcyNTk2MzIyOSwicGF0aCI6Ii8zNTU4ODE2Ny8yODIzNzc2MDUtOTRhYjcyYmEtZTFkMi00ZTVjLWI2ODEtYmYxOTE2NGM4OTNhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDEwMTM0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWMxZmFmMDI3MjM4Y2NmZTEyZDIyNmY0ZmUwMWU0NjVkNmQ2ZWY0ZmY4NjhjYTU2NmFmNTg1MjQyYWI4MjY3NzUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.sp7kYABqj1IL4FqSef3ATRwvg2ZtUcZ7to_H0E03yHA)
![Screenshot 2023-11-12 at 10 11 30 PM](https://private-user-images.githubusercontent.com/35588167/282377660-7a400de0-6e11-4825-9d97-f8d76864307d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU5NjM1MjksIm5iZiI6MTcyNTk2MzIyOSwicGF0aCI6Ii8zNTU4ODE2Ny8yODIzNzc2NjAtN2E0MDBkZTAtNmUxMS00ODI1LTlkOTctZjhkNzY4NjQzMDdkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEwVDEwMTM0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU0NWRkMGI4ZDA1NGFjNzAyMzYzOTBlMjg1ZWI5ODJhMmM3ZGI5NWVlMjM3N2Y4YzIwOTNmZGYxMmEyMTQ0ZTImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.PCHNrVrPeQQMePkf81cDxm50k6huaiJIKlGKveoSLRU)
The Challenge: Use generative AI to create artificial retinal video sequences & eye motion traces given a disease state.
Implementation Details:
- Used a four-layer LSTM network with dropout regularization after each LSTM layer and a single dense neuron output layer
- Trained over 50 epochs, with rmsprop optimizer and mean_squared_error as loss function
- Utilized first 80% of each time series as training data, last 20% as testing data
- Used MinMaxScaler to normalize all pixel values to [-1, 1] domain
- Used previous 60 frames as context for each sliding window
Technologies Used:
- Tensorflow: To build LSTMs, GANs
- Scikit-Learn: MinMaxScaler and Train Test Split
- Keras: Deep learning API (Sequential Model and neural network layers)
- Seaborn and Matplotlib: Data Visualization
- NumPy: For calculations and array manipulation
- Pandas: For processing CSV files