Update documentation

CODE-OF-CONDUCT.md

@@ -0,0 +1,51 @@
+# Contributor Code of Conduct
+
+At MIMIRO, trust, respect, collaboration and transparency are core values we believe should live and breathe within our projects. Our community welcomes participants from around the world with different experiences, unique perspectives, and great ideas to share.
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+- Using welcoming and inclusive language
+- Being respectful of differing viewpoints and experiences
+- Gracefully accepting constructive criticism
+- Attempting collaboration before conflict
+- Focusing on what is best for the community
+- Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+- Violence, threats of violence, or inciting others to commit self-harm
+- The use of sexualized language or imagery and unwelcome sexual attention or advances
+- Trolling, intentionally spreading misinformation, insulting/derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others' private information, such as a physical or electronic address, without explicit permission
+- Abuse of the reporting process to intentionally harass or exclude others
+- Advocating for, or encouraging, any of the above behavior
+- Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting us at [MIMIRO](https://en.mimiro.no/#contact). All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+If you are unsure whether an incident is a violation, or whether the space where the incident took place is covered by our Code of Conduct, **we encourage you to still report it**. We would prefer to have a few extra reports where we decide to take no action, than to leave an incident go unnoticed and unresolved that may result in an individual or group to feel like they can no longer participate in the community. Reports deemed as not a violation will also allow us to improve our Code of Conduct and processes surrounding it. If you witness a dangerous situation or someone in distress, we encourage you to report even if you are only an observer.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

CONTRIBUTING.md

@@ -0,0 +1,43 @@
+# Contributing to MIMIRO data hub job testing
+We welcome your input! We want to make contributing to this project as easy and transparent as possible, whether it's:
+
+- Reporting a bug
+- Discussing the current state of the code
+- Submitting a fix
+- Proposing new features
+- Becoming a maintainer
+
+## Code of Conduct
+
+If you haven't already, please read our [code of conduct](CODE-OF-CONDUCT.md)
+
+## We Develop with Github
+We use github to host code, to track issues and feature requests, as well as accept pull requests.
+
+## All Code Changes Happen Through Pull Requests
+We Use [Github Flow](https://guides.github.com/introduction/flow/index.html) to ensure all changes go through pull requests and are reviewed by the maintainers.
+
+Please use the following steps:
+
+1. Fork the repo and create your branch from `master`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. Issue that pull request!
+
+## Contribution License
+When you submit code changes, your submissions are understood to be under the same [Apache 2.0 License](https://choosealicense.com/licenses/apache-2.0/) that covers the project.
+
+## Report bugs using Github Issues
+We use GitHub [issues](https://github.com/mimiro-io/datahub-job-testing/issues) to track public bugs. Report a bug by opening a new issue, but please check to see if this issue has already been reported.
+
+**Great Bug Reports** tend to have:
+
+- A quick summary and/or background
+- Steps to reproduce
+ - Be specific!
+ - Give sample code and data if you can.
+- What you expected would happen
+- What actually happens
+- Notes (possibly including why you think this might be happening, or things you tried that didn't work)

README.md

@@ -6,7 +6,32 @@ This is a simple test runner for datahub jobs. It is designed to be used in a CI
 This tool expects a manifest file to be present inside your datahub config project. The manifest file should contain a list of tests to run. 
 For each test, the runner will spin up a new datahub, upload the defined required datasets, and run the job. It will then compare the output with the expected output.
 
-#### Test configuration
+#### With CLI
+```bash
+djt path/to/manifest.json
+```
+
+#### Import as a module
+```go
+package tests
+
+import (
+ ...
+ djt "github.com/mimiro-io/datahub-job-testing"
+)
+
+func TestMyJob(t *testing.T) {
+ ...
+ manifest := "path/to/manifest.json"
+ tr := djt.NewTestRunner(manifest)
+ if ! tr.RunAllTests() {
+ // tests didn't pass
+ }
+ ...
+}
+```
+
+### Test configuration
 Each test case defined has the following properties:
 ```json
 
@@ -31,28 +56,3 @@ Each test case defined has the following properties:
 Some configuration is common to all tests. To add datasets for all test cases, use the top-level property `common.requiredDatasets`. (See [example manifest](example-manifest.json) for details.)
 
 
-
-
-
-## TODO:
-- [x] Create test manifest
- * Test name
- * jobtitle.js
- * tags
- * sink dataset
- * source dataset
- * required datasets
- * where is the data stored
- * description
- * expected output location
-
-- [x] Load manifest
-- [ ] find all .ts transforms
- * copy into separate ts dir
- * compile them somehow
-- [x] iterate through list of tests
-- [x] spin up datahub for each
-- [x] load required datasets
-- [x] build job with sdk and run it
-- [x] compare output with expected
-- [ ] run egcl on the output (secondary)

testing/datahub.go

@@ -42,4 +42,5 @@ func StartTestDatahub(port string) (*DatahubManager, error) {
 func (dm *DatahubManager) Cleanup() {
  dm.Instance.Stop(context.Background())
  os.RemoveAll(dm.Location)
+ os.Unsetenv("LOG_LEVEL")
 }

testing/manifest.go

@@ -31,20 +31,13 @@ type StoredDataset struct {
  Path string `json:"path"`
 }
 
-type Job struct {
- Title string `json:"title,omitempty"`
- Tags []string `json:"tags,omitempty"`
- SourceDataset string `json:"sourceDataset,omitempty"`
- SinkDataset string `json:"sinkDataset,omitempty"`
- Transform string `json:"transform,omitempty"`
-}
-
 type ManifestManager struct {
  Manifest *Manifest
  ProjectRoot string
  Variables map[string]any
 }
 
+// NewManifestManager creates a new ManifestManager from the given manifest file path
 func NewManifestManager(path string) *ManifestManager {
  projectRoot := getGitRootPath(filepath.Dir(path))
  manifest := parseManifest(path)