Skip to content

Data Maps

Jump to bottom
Jason Hoekstra edited this page Jan 31, 2018 · 8 revisions

Overview

Data Flow uses JSON-based data maps to convert source CSV data to the Ed-Fi API. Each data map mirrors the Ed-Fi API endpoint payload as described by Swagger/OpenAPI metadata and extends additional attributes for data mapping and transformation.

A full set of sample data maps for various learning data providers can be found in the /sample-data/data-maps maps directory within this repository. The /sample-data/bootstrap-data directory contains additional data that needs to be inserted in the ODS before source data is ready to be inserted by these templates.

Sample Data Map

Below is a sample data map for the Students Ed-Fi API endpoint. Notice that the overall body matches /students POST operation and adds additional attributes for processing source data.

{
	"studentUniqueId": {
		"data-type": "string",
		"source": "column",
		"source-column": "ID"
	},
	"firstName": {
		"data-type": "string",
		"source": "column",
		"source-column": "First Name"
	},
	"lastSurname": {
		"data-type": "string",
		"source": "column",
		"source-column": "Last Name"
	},
	"sexType": {
		"data-type": "string",
		"source": "static",
		"value": "Not Selected"
	},
	"birthDate": {
		"data-type": "date-time",
		"source": "column",
		"source-column": "DOB"
	},
	"hispanicLatinoEthnicity": {
		"data-type": "boolean",
                "source": "lookup-table",
                "source-table": "assess-ethnicity",
                "source-column": "Ethnicity",
                "default": "false"
	}
}

Format

Below are possible configurations for each field within a data map:

static

Field Name Value(s) Purpose
data-type "string", "integer", "date-time", or "boolean" Matches Ed-Fi API data type as provided by Swagger
source "static" Specifies a static field for data submission
value [static value] Provides a value for data submission

column

Field Name Value(s) Purpose
data-type "string", "integer", "date-time", or "boolean" Matches Ed-Fi API data type as provided by Swagger
source "column" Specifies field will obtain information directly from source data
source-column [header column name] Specifies column name to retrieve data
default [static value] Provides value if field is blank within row

lookup-table

Field Name Value(s) Purpose
data-type "string", "integer", "date-time", or "boolean" Matches Ed-Fi API data type as provided by Swagger
source "lookup-table" Specifies field will obtain information from source data and transform it according to a lookup table
source-table [group set name within lookup configuration] Specifies group set from dbo.Lookups table to lookup and replace value
source-column [header column name] Specifies column name to retrieve data
default [static value] Provides value if field is blank within row

Required fields

Source data files may provide data in various columns depending on the types of measurements taken. The _required attribute defines column header names that must have data present before processing the block. If data is null or blank in a _required block, the block will not be sent to the Ed-Fi ODS API. This allows for a file with many indicators to process only the present values for that row.

Clone this wiki locally