Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plugins PoC - Java #234

Closed
6 of 7 tasks
Tracked by #228
f-galland opened this issue May 23, 2024 · 12 comments
Closed
6 of 7 tasks
Tracked by #228

Plugins PoC - Java #234

f-galland opened this issue May 23, 2024 · 12 comments
Assignees
Labels
level/task Task issue request/operational Operational requests type/research Research issue

Comments

@f-galland
Copy link
Member

f-galland commented May 23, 2024

Description

To better understand how plugins work, their lifecylce, how they interact with core functionalities and how to develop them in general, we are going to develop a simple plugin for wazuh-indexer that exposes an API to manage ToDos, as a proof of concept. We'll use Java as the programming language.

As a definition of done, we expect to have an API from which it's possible to:

  • Create ToDos
  • Read ToDos
  • Update ToDos
  • Delete ToDos

ToDo model

The ToDos will be stored using an index, for example: wazuh-todos. Each of the documents in the index could have these properties:

  • id: uuid
  • title: string
  • description: text
  • due_date: date

The index name and its fields must follow the naming convention.

Tasks

  • Create a plugin using the existing template.
  • Rename and set up the plugin accordingly.
  • Create tests.
  • Create routes.
  • Create route handles: request, action, response.
  • Create ToDo model.
  • Create Index manager.
@f-galland
Copy link
Member Author

f-galland commented May 24, 2024

@AlexRuiz7 AlexRuiz7 added level/task Task issue request/operational Operational requests type/research Research issue labels May 27, 2024
@wazuhci wazuhci moved this to In progress in Release 5.0.0 May 27, 2024
@f-galland
Copy link
Member Author

f-galland commented May 30, 2024

I was able to index data to an index using a custom REST endpoint by extending the RestIndexAction class and supplying it with custom routes and overriding the prepareRequest() method with one with hardcoded values:

public class RestHelloWorldAction extends RestIndexAction {

  public static final String HELLO_WORLD_INDEX_NAME = "helloworld";

  @Override
  public String getName() {
    return "rest_handler_hello_world";
  }

  @Override
  public List<Route> routes() {
    return unmodifiableList(asList(
        new Route(POST, "/_plugins/hello_world")));
  }

  @Override
  public RestChannelConsumer prepareRequest(RestRequest request, final NodeClient client) throws IOException {
    assert request.params().get("id") == null : "non-null id: " + request.params().get("id");
    // default to op_type create
    request.params().put("index", HELLO_WORLD_INDEX_NAME );
    request.params().putIfAbsent("op_type", "create");
    return super.prepareRequest(request, client);
  }
}
#### Index a simple json using _plugins/hello_world endpoint
$ curl -XPOST http://localhost:9200/_plugins/hello_world -H "Content-Type: application/json" -d '{"field":"value"}'
{"_index":"helloworld","_id":"Wer2yY8BwC6DtDSTphCq","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

#### Use the Id to retrieve it
curl -XGET "http://localhost:9200/helloworld/_doc/Wer2yY8BwC6DtDSTphCq"
{"_index":"helloworld","_id":"Wer2yY8BwC6DtDSTphCq","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{"field":"value"}}

@f-galland
Copy link
Member Author

I'm working on this PoC in my own repo:

@f-galland
Copy link
Member Author

I was able to implement a very rough CRUD:

$ curl -XPOST http://localhost:9200/_plugins/todo/create -H "Content-Type: application/json" -d '{"field":"value"}'
{"_index":"todo","_id":"9teryo8B79B5eeR91Cfu","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}fede@tyner:~/IdeaProjects/opensearch-plugin-crud-sample (add-read-update-delete *+)

$ curl -XGET "http://localhost:9200/_plugins/todo/read/9teryo8B79B5eeR91Cfu"
{"_index":"todo","_id":"9teryo8B79B5eeR91Cfu","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{"field":"value"}}fede@tyner:~/IdeaProjects/opensearch-plugin-crud-sample (add-read-update-delete *+)

$ curl -XPOST http://localhost:9200/_plugins/todo/update/9teryo8B79B5eeR91Cfu -H "Content-Type: application/json" -d '{"doc":{"field":"value2"}}'
{"_index":"todo","_id":"9teryo8B79B5eeR91Cfu","_version":2,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1}fede@tyner:~/IdeaProjects/opensearch-plugin-crud-sample (add-read-update-delete *+)

$ curl -XGET "http://localhost:9200/_plugins/todo/read/9teryo8B79B5eeR91Cfu"
{"_index":"todo","_id":"9teryo8B79B5eeR91Cfu","_version":2,"_seq_no":1,"_primary_term":1,"found":true,"_source":{"field":"value2"}}fede@tyner:~/IdeaProjects/opensearch-plugin-crud-sample (add-read-update-delete *+)

$ curl -XDELETE "http://localhost:9200/_plugins/todo/delete/9teryo8B79B5eeR91Cfu"
{"_index":"todo","_id":"9teryo8B79B5eeR91Cfu","_version":3,"result":"deleted","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1}fede@tyner:~/IdeaProjects/opensearch-plugin-crud-sample (add-read-update-delete *+)

For the time being I used different paths for each operation as I registered the routes through individual classes.

@f-galland
Copy link
Member Author

I got tests somewhat working by basing my own classes on:

Image

Various warnings pop up, and this is still only testing writing to indices, so I need to dig deeper into the reference code above.

@f-galland
Copy link
Member Author

It seems that we should use yaml based tests for the rest layer:

@f-galland
Copy link
Member Author

I managed to get a simple REST API yaml-based test by following:

Various examples are present under that folder as well:

The following test definition indexes data by hitting the _plugins/todo endpoint with a POST request and {"foo":"bar"} as its content.
It then takes the _id field from the reply and stores it into a document_id "stash" variable.
After that, a GET operation on the /todo index looks for that document's id and checks whether the content is indeed {"foo":"bar"}.

---
"CRUD test":
    - do:
        _plugins.todo:
            body: { foo: bar }

    - set: { _id: document_id }
    - match: { _index: todo }

    - do:
        get:
            index: todo
            id: $document_id
    - match: { _source: { foo: bar } }

However I'm currently stuck at creating a test for a PUT method on the same endpoint (which in my current plugin, updates the content of a particular "todo task").
I think the issue might be related to the plugin's API json definition file under src/yamlRestTest/resources/rest-api-spec/api/, which currently looks like the following:

{
  "_plugins.todo": {
    "stability" : "stable",
    "url": {
      "paths": [
        {
          "path": "/_plugins/todo",
          "methods": [
            "POST",
            "PUT"
          ]
        }
      ]
    },
    "body": {
      "description": "The ToDo plugin's document manipulation endpoint"
    }
  }
}

@f-galland
Copy link
Member Author

f-galland commented Jun 5, 2024

I got the yaml REST API tests working.

In order to do so, the API needs to be properly defined in json files under src/yamlRestTest/resources/rest-api-spec/api/

There needs to be a json file per API endpoint and method:

_plugins.todo_create.json

{
  "_plugins.todo_create": {
    "stability" : "stable",
    "url": {
      "paths": [
        {
          "path": "/_plugins/todo",
          "methods": [
            "POST"
          ]
        }
      ]
    },
    "body": {
      "description": "The document"
    }
  }
}

_plugins.todo_read.json

{
  "_plugins.todo_read": {
    "stability" : "stable",
    "url": {
      "paths": [
        {
          "path": "/_plugins/todo/{id}",
          "methods": [
            "GET"
          ],
          "parts": {
            "id":{
              "type":"string",
              "description":"The id of the document to update"
            }
          }
        }
      ]
    }
  }
}

_plugins.todo_update.json

{
  "_plugins.todo_update": {
    "stability" : "stable",
    "url": {
      "paths": [
        {
          "path": "/_plugins/todo/{id}",
          "methods": [
            "PUT"
          ],
          "parts": {
            "id":{
              "type":"string",
              "description":"The id of the document to update"
            }
          }
        }
      ]
    },
    "body": {
      "description": "The document"
    }
  }
}

_plugins.todo_delete.json

{
  "_plugins.todo_delete": {
    "stability" : "stable",
    "url": {
      "paths": [
        {
          "path": "/_plugins/todo/{id}",
          "methods": [
            "DELETE"
          ],
          "parts": {
            "id":{
              "type":"string",
              "description":"The id of the document to update"
            }
          }
        }
      ]
    }
  }
}

Each of these need to:

  • have a parent object key named the same as the file (it doesn't have to match the endpoint name verbatim).
  • Declare the path of the endpoint and the methods it can accept
    • Placeholders can be used (as examplified in the id fields in the read, update and delete files above) in the path
    • the parts object contains a list of path variables.

It's important to detail the usafe of each of these endpoints correctly, since the yaml tests are based on these definitions.

The yaml tests have their own syntax and need to be placed under src/yamlRestTest/resources/rest-api-spec/test/plugin_name

I've created the following test files (one for accepted API method):

20_create.yml

---
"Create test":
    - do:
        _plugins.todo_create:
            body: { foo: bar }

    - set: { _id: document_id }
    - match: { _index: todo }

    - do:
        get:
            index: todo
            id: $document_id
    - match: { _source: { foo: bar } }

30_read.yml

---
"Read test":
    - do:
          index:
              index:  todo
              id:     1000
              body:
                  foo:    bar

    - do:
          _plugins.todo_read:
              id: 1000
    - match: { _index: todo }
    - match: { _source: { foo: bar } }

40_update.yml

---
"Update test":
    - do:
          index:
              index:  todo
              id:     1000
              body:
                  foo:    bar

    - do:
        _plugins.todo_update:
           id: 1000
           body: { doc: { foo: baz } }

    - do:
          get:
              index: todo
              id: 1000
    - match: { _source: { foo: baz } }

50_delete.yml

---
"Delete test":
    - do:
          index:
              index:  todo
              id:     1000
              body:
                  foo:    bar

    - do:
          _plugins.todo_delete:
              id: 1000
    - match: { _index: todo }
    - match: { result: deleted }

The _plugins.todo_create, _plugins.todo_read, _plugins.todo_update and _plugins.todo_delete sections actually reference the API spec json from before.
The 20_create.yml test starts by calling /_plugins/todo with a POST (as defined in the appropriate API spec file) and creating which creates a document with an automatically generated _id.
The _id field is then read from the reply and stored into a variable called document_id.

The second do: section runs a get query on the todo index with the _id stored in the document_id variable and then checks whether the _source field matches what we've stored in the document.

The rest of the tests start with the indexing of a document by means of the built-in index API.
After that, we call our custom read, update and delete endpoints on the specified document _id. We then cehck if the contents are as expected by means of a match

Image

@f-galland
Copy link
Member Author

It was helpful to browse through the built in yaml tests and the rest api spec files under:

@f-galland
Copy link
Member Author

f-galland commented Jun 5, 2024

All of the above code can be checked out in the repo I've set up for these tests:

I've now moved on to building unit tests for our custom classes.

@wazuhci wazuhci moved this from In progress to On hold in Release 5.0.0 Jun 6, 2024
@wazuhci wazuhci moved this from On hold to In progress in Release 5.0.0 Jun 10, 2024
@f-galland
Copy link
Member Author

Unit tests are going through.
Below is an example of the unit tests for the ToDoRestIndexAction class:

/*
 * SPDX-License-Identifier: Apache-2.0
 *
 * The OpenSearch Contributors require contributions made to
 * this file be licensed under the Apache-2.0 license or a
 * compatible open source license.
 */

/*
 * Licensed to Elasticsearch under one or more contributor
 * license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright
 * ownership. Elasticsearch licenses this file to you under
 * the Apache License, Version 2.0 (the "License"); you may
 * not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

/*
 * Modifications Copyright OpenSearch Contributors. See
 * GitHub history for details.
 */

package org.opensearch.rest.action;

import java.io.IOException;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import org.junit.Before;
import org.mockito.Mockito;
import org.opensearch.client.node.NodeClient;
import org.opensearch.common.xcontent.XContentType;
import org.opensearch.core.common.bytes.BytesArray;
import org.opensearch.rest.RestHandler;
import org.opensearch.rest.RestRequest;
import org.opensearch.test.OpenSearchTestCase;
import org.opensearch.test.rest.FakeRestChannel;
import org.opensearch.test.rest.FakeRestRequest;

public class ToDoRestIndexActionTests extends OpenSearchTestCase {
    private String path;
    private String requestBody;
    private ToDoRestIndexAction toDoRestIndexAction;

    @Before
    public void setUp() throws Exception {
        super.setUp();
        this.path = String.format(Locale.ROOT, "%s", ToDoPlugin.TODO_BASE_URI);
        this.requestBody = "{\"foo\":\"bar\"}";
        this.toDoRestIndexAction = new ToDoRestIndexAction();
    }

    public void testGetNames() {
        String name = toDoRestIndexAction.getName();
        assertEquals("todo_plugin_document_index", name);
    }

    public void testGetRoutes() {
        List<RestHandler.Route> routes = toDoRestIndexAction.routes();
        assertEquals(this.path, routes.get(0).getPath());
    }

    public void testPrepareRequest() throws IOException {
        Map<String, String> params = new HashMap<>();
        FakeRestRequest request = new FakeRestRequest.Builder(xContentRegistry()).withMethod(RestRequest.Method.POST)
            .withPath(this.path)
            .withParams(params)
            .withContent(new BytesArray(this.requestBody), XContentType.JSON)
            .build();

        final FakeRestChannel channel = new FakeRestChannel(request, true, 0);

        this.toDoRestIndexAction.prepareRequest(request, Mockito.mock(NodeClient.class));
        assertEquals(channel.responses().get(), 0);
        assertEquals(channel.errors().get(), 0);
    }
}

These tests run automatically as part of the check gradle task.

Below is a screenshot of the tests' results:

Image

@wazuhci wazuhci moved this from In progress to On hold in Release 5.0.0 Jun 11, 2024
@AlexRuiz7
Copy link
Member

AlexRuiz7 commented Jun 13, 2024

Conclusions

Before closing this issue, we'll be reviewing each of the tasks.

  • We have been able to create a plugin using the OpenSearch plugin template repo. We have renamed and configured the plugin is described in the README.
  • We created basic unit and integration tests.
  • We implemented a REST HTTP API to manage Tasks.

However, we did code how to create indices on start, but we have examples about how to do that.

@wazuhci wazuhci moved this from On hold to Done in Release 5.0.0 Jun 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
level/task Task issue request/operational Operational requests type/research Research issue
Projects
Status: Done
Development

No branches or pull requests

2 participants