-
Notifications
You must be signed in to change notification settings - Fork 391
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spark 2.4 support #402
Spark 2.4 support #402
Changes from all commits
77e2229
485fcd5
12f5333
a4ee986
47703fa
ea4b11f
ab81e31
c017869
99ea7e1
daa2672
7d3ebb7
5661640
8852d69
2ab8924
7c8b988
d98c8a9
cce0d8f
8afbae7
e8770f6
afffc56
ed43719
8804acd
a1461c7
b54d0f5
861f862
41f9dc4
cb4cb7b
abca58b
ab8daf8
e58da6c
37614d7
e8a27ea
e205856
f93db0b
505a881
e21d06c
84bb70a
08db1db
3ba95b6
d77515c
d3b53be
1634927
b8a43b5
094be05
cc7de97
4bd5977
46fc60a
0211fde
7147b72
4d686c7
f6a2a4e
adc5b25
c64eb51
e0c164d
b3c0a74
f4ab3fd
50d9dfb
3417972
df38bcc
3af27f3
66afd79
39ff755
09c5f2b
d79bcc3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -69,20 +69,16 @@ task copyTemplates(type: Copy) { | |
fileName.replace(".gradle.template", ".gradle") | ||
} | ||
expand([ | ||
databaseHostname: 'db.company.com', | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. do we absolutely need to change this file ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this was a cleanup, why not? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. because the PR is big enough, and for easier maintenance I hoped to keep it focussed on the Spark upgrade There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure, this has no material difference for the change. You always get a bonus of such with my PRs 😛 |
||
version: scalaVersion, | ||
scalaVersion: scalaVersion, | ||
scalaVersionRevision: scalaVersionRevision, | ||
scalaTestVersion: scalaTestVersion, | ||
junitVersion: junitVersion, | ||
sparkVersion: sparkVersion, | ||
avroVersion: avroVersion, | ||
sparkAvroVersion: sparkAvroVersion, | ||
hadoopVersion: hadoopVersion, | ||
collectionsVersion: collectionsVersion, | ||
transmogrifaiVersion: version, | ||
buildNumber: (int)(Math.random() * 1000), | ||
date: new Date() | ||
transmogrifaiVersion: version | ||
]) | ||
} | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -138,7 +138,7 @@ case class AutomaticSchema(recordClassName: String)(dataFile: File) extends Sche | |
case Some(actualType) => | ||
val newSchema = Schema.create(actualType) | ||
val schemaField = | ||
new Schema.Field(field.name, newSchema, "auto-generated", orgSchemaField.defaultValue) | ||
new Schema.Field(field.name, newSchema, "auto-generated", orgSchemaField.defaultVal()) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. do we need this change? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am keeping the existing functionality as is. We can change the behavior separately. |
||
AvroField.from(schemaField) | ||
} | ||
} else field | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO for a future PR: I really hope we start making strides to automating this stuff. We should have here placeholders that are populated from build. Even if it's some quick and dirty version of
sed
script