Skip to content

Commit

Permalink
update-bearer: update-bearer-token-handling-impl
Browse files Browse the repository at this point in the history
  • Loading branch information
@vishaalcharpe
vishaalcharpe committed Dec 16, 2024
1 parent d048b5a commit e489739
Show file tree
Hide file tree
Showing 9 changed files with 80 additions and 67 deletions.
30 changes: 15 additions & 15 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,10 @@ Create a custom `fluent-bit.conf` or edit the existing one to specify which logs
Workers 1
accessKey <access_key>
accessID <access_ID>
bearerToken <bearer_token>
bearerToken Bearer <bearer_token>
resourceMapping {"<event_key>": "<lm_property>"}
include_metadata <boolean_value>
lm_debug <boolean_value>
includeMetadata <boolean_value>
lmDebug <boolean_value>
```

For more configuration examples, please refer to the examples folder, or see the [Fluentbit configuration documentation](https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file)
Expand Down Expand Up @@ -59,17 +59,17 @@ Produces this event:

## LogicMonitor properties

| Property | Description |
|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `Name` | Name of the input plugin. |
| `lmCompanyName` | LogicMonitor account name with domain. For example, test.logicmonitor.com . |
| `Match` | A pattern to match against the tags of incoming records. For example, * will match everything. |
| `Workers` | Number of workers to operate. |
| `accessID` | LM API Token access ID. |
| `accessKey` | LM API Token access key. |
| `bearerToken` | LM API Bearer Token. Either specify `access_id` and `access_key` both or `bearer_token`. If all specified, LMv1 token(`access_id` and `access_key`) will be used for authentication with Logicmonitor. |
| `resourceMapping` | The mapping that defines the source of the log event to the LM resource. In this case, the `<event_key>` in the incoming event is mapped to the value of `<lm_property>`. |
| `include_metadata` | When `true`, appends additional metadata to the log. default `false`. |
| `lm_debug` | When `true`, logs more information to the fluent-bit console. |
| Property | Description |
|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `Name` | Name of the input plugin. |
| `lmCompanyName` | LogicMonitor account name with domain. For example, test.logicmonitor.com . |
| `Match` | A pattern to match against the tags of incoming records. For example, * will match everything. |
| `Workers` | Number of workers to operate. |
| `accessID` | LM API Token access ID. If not provided, omit setting the key entirely. |
| `accessKey` | LM API Token access key. If not provided, omit setting the key entirely. |
| `bearerToken` | LM API Bearer Token. Either specify `access_id` and `access_key` both or `bearer_token`. If all specified, LMv1 token(`access_id` and `access_key`) will be used for authentication with Logicmonitor. |
| `resourceMapping` | The mapping that defines the source of the log event to the LM resource. In this case, the `<event_key>` in the incoming event is mapped to the value of `<lm_property>`. |
| `includeMetadata` | When `true`, appends additional metadata to the log. default `false`. |
| `lmDebug` | When `true`, logs more information to the fluent-bit console. |


6 changes: 3 additions & 3 deletions examples/fluent-bit-http.conf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
Workers 1
accessKey <accessKey>
accessID <accessID>
bearerToken <bearerToken>
bearerToken Bearer <bearerToken>
resourceMapping {"host":"system.ips"}
include_metadata <boolean_value>
includeMetadata <boolean_value>
id <output_id>
lm_debug <boolean_value>
lmDebug <boolean_value>
6 changes: 3 additions & 3 deletions examples/fluent-bit-multiple-config.conf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
Workers 1
accessKey <accessKey>
accessID <accessID>
bearerToken <bearerToken>
include_metadata <boolean_value>
bearerToken Bearer <bearerToken>
includeMetadata <boolean_value>
id <output_id>
lm_debug <boolean_value>
lmDebug <boolean_value>
6 changes: 3 additions & 3 deletions examples/fluent-bit-tcp.conf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@
Workers 1
accessKey <accessKey>
accessID <accessID>
bearerToken <bearerToken>
bearerToken Bearer <bearerToken>
resourceMapping {"host":"system.ips"}
include_metadata <boolean_value>
includeMetadata <boolean_value>
id <output_id>
lm_debug <boolean_value>
lmDebug <boolean_value>
6 changes: 3 additions & 3 deletions examples/fluent-bit.conf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
Workers 1
accessKey <accessKey>
accessID <accessID>
bearerToken <bearerToken>
bearerToken Bearer <bearerToken>
resourceMapping {"host":"system.ips"}
include_metadata <boolean_value>
lm_debug <boolean_value>
includeMetadata <boolean_value>
lmDebug <boolean_value>
7 changes: 3 additions & 4 deletions examples/fluent-bit_multiline.conf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,8 @@
Workers 1
accessKey <accessKey>
accessID <accessID>
bearerToken <bearerToken>
bearerToken Bearer <bearerToken>
resourceMapping {"host":"system.ips"}
include_metadata true
include_metadata <boolean_value>
includeMetadata <boolean_value>
id <output_id>
lm_debug <boolean_value>
lmDebug <boolean_value>
24 changes: 14 additions & 10 deletions output/client.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ type LogicmonitorClient struct {
resourceMapping string
bulk []model.LogInput
includeMetadata bool
logSource string
versionId string
logger *Logger
sizeThresholdInBytes int
}
Expand All @@ -38,8 +40,8 @@ type LogicmonitorClient struct {
type ClientOptionFunc func(*LogicmonitorClient) error

// NewClient is a constructor for Logicmonitor http client
func NewClient( lmCompanyName string, accessID string, accessKey string, bearerToken string, useBearerTokenforAuth bool, resourceMapping string, includeMetadata bool, logger *Logger ) (*LogicmonitorClient) {
func NewClient( lmCompanyName string, accessID string, accessKey string, bearerToken string, useBearerTokenforAuth bool, resourceMapping string, includeMetadata bool, logSource string, versionId string, logger *Logger ) (*LogicmonitorClient) {

logicmonitorClient := &LogicmonitorClient{
lmCompanyName: lmCompanyName,
accessID: accessID,
Expand All @@ -49,13 +51,15 @@ func NewClient( lmCompanyName string, accessID string, accessKey string, bearerT
resourceMapping: resourceMapping,
bulk: nil,
includeMetadata: includeMetadata,
logSource: logSource,
versionId: versionId,
logger: logger,
sizeThresholdInBytes: maxRequestBodySizeInBytes,
}

return logicmonitorClient


}

func NewLogIngester(logicmonitorClient *LogicmonitorClient) (*logs.LMLogIngest){
Expand All @@ -64,11 +68,12 @@ func NewLogIngester(logicmonitorClient *LogicmonitorClient) (*logs.LMLogIngest){
BearerToken: logicmonitorClient.bearerToken}

companyName := logicmonitorClient.lmCompanyName
userAgent := fmt.Sprintf("%s/%s", logicmonitorClient.logSource, logicmonitorClient.versionId)
options := []logs.Option{
logs.WithLogBatchingDisabled(),
logs.WithAuthentication(auth),
logs.WithEndpoint("https://"+companyName+"/rest"),
logs.WithUserAgent("lm-logs-fluentbit"),
logs.WithUserAgent(userAgent),
}

lmLog, err := logs.NewLMLogIngest(context.Background(), options...)
Expand Down Expand Up @@ -128,15 +133,14 @@ func (logicmonitorClient *LogicmonitorClient) Send(log []byte, logIngestor *logs

for k, v := range resourceMapReceived {
if(k != ""){
resourceMap[v] = jsonMap[k]
resourceMap[k] = v
}

}
if (jsonMap["host"].(string) != "" && len(resourceMap)==0){
resourceMap["system.hostname"] = jsonMap["host"]
}

if(logicmonitorClient.includeMetadata){
logger.Debug(fmt.Sprintf("include metadata: %s",logicmonitorClient.includeMetadata))
if(logicmonitorClient.includeMetadata){
metadata = getMetadata(jsonMap)
}

Expand Down
60 changes: 35 additions & 25 deletions output/out_lm.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ package main
import (
"C"
"fmt"
"github.com/fluent/fluent-bit-go/output"
jsoniter "github.com/json-iterator/go"
"github.com/logicmonitor/lm-data-sdk-go/api/logs"
"os"
"reflect"
"strconv"
"time"
"unsafe"

"github.com/fluent/fluent-bit-go/output"
jsoniter "github.com/json-iterator/go"
"github.com/logicmonitor/lm-data-sdk-go/api/logs"
)

const (
Expand All @@ -28,11 +28,11 @@ var (
)

type LogicmonitorOutput struct {
plugin Plugin
logger *Logger
client *LogicmonitorClient
logIngestor *logs.LMLogIngest
id string
plugin Plugin
logger *Logger
client *LogicmonitorClient
logIngestor *logs.LMLogIngest
id string
}

var (
Expand Down Expand Up @@ -145,7 +145,7 @@ func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int
break
}

log, err := serializeRecord(ts, C.GoString(tag), record,id)
log, err := serializeRecord(ts, C.GoString(tag), record, id)
if err != nil {
continue
}
Expand All @@ -160,12 +160,12 @@ func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int
//
//export FLBPluginExit
func FLBPluginExit() int {
plugin.Flush(nil,nil)
plugin.Flush(nil, nil)
return output.FLB_OK
}

func initConfigParams(ctx unsafe.Pointer) error {
debug, err := strconv.ParseBool(output.FLBPluginConfigKey(ctx, "lm_debug"))
debug, err := strconv.ParseBool(output.FLBPluginConfigKey(ctx, "lmDebug"))
if err != nil {
debug = false
}
Expand All @@ -186,38 +186,47 @@ func initConfigParams(ctx unsafe.Pointer) error {
}

logger = NewLogger(outputName+"_"+outputId, debug)


lmCompanyName := output.FLBPluginConfigKey(ctx, "lmCompanyName")
if lmCompanyName == "" {
return fmt.Errorf("LM Company name is not specified. Please specify the company name in the configuration")
return fmt.Errorf("LM Company name is not specified. Please specify the company name in the configuration")
}

accessKey := output.FLBPluginConfigKey(ctx, "accessKey")
accessID := output.FLBPluginConfigKey(ctx, "accessID")

useBearerTokenforAuth := true
if (accessID == "" || accessKey == "") {
useBearerTokenforAuth := false
if accessID == "" || accessKey == "" {
logger.Log("accessID or accessKey is empty. Using bearer Token for authentication")
useBearerTokenforAuth = false
useBearerTokenforAuth = true
}
bearerToken := output.FLBPluginConfigKey(ctx, "bearerToken")

if(bearerToken == "" && useBearerTokenforAuth){
return fmt.Errorf("Bearer token not specified. Either access_id and access_key both or bearer_token must be specified for authentication with Logicmonitor.")
if accessID == "" || accessKey == "" {
if bearerToken == "" {
return fmt.Errorf("Bearer token not specified. Either access_id and access_key both or bearer_token must be specified for authentication with Logicmonitor.")
}
}

resourceMapping := output.FLBPluginConfigKey(ctx, "resourceMapping")

includeMetadata, err := strconv.ParseBool(output.FLBPluginConfigKey(ctx, "include_metadata"))

client := NewClient(lmCompanyName ,accessID , accessKey, bearerToken, useBearerTokenforAuth, resourceMapping, includeMetadata, logger)
includeMetadata, err := strconv.ParseBool(output.FLBPluginConfigKey(ctx, "includeMetadata"))
logSource := output.FLBPluginConfigKey(ctx, "logSource")
if logSource == "" {
logSource = "lm-logs-fluentbit"
}
versionId := output.FLBPluginConfigKey(ctx, "versionId")
if versionId == "" {
versionId = "1.0.0"
}

client := NewClient(lmCompanyName, accessID, accessKey, bearerToken, useBearerTokenforAuth, resourceMapping, includeMetadata, logSource, versionId, logger)
logIngestor := NewLogIngester(client)
outputs[outputId] = LogicmonitorOutput{
logger: logger,
client: client,
logIngestor: logIngestor,
id: outputId,
logger: logger,
client: client,
logIngestor: logIngestor,
id: outputId,
}

return nil
Expand All @@ -238,6 +247,7 @@ func serializeRecord(ts interface{}, tag string, record map[interface{}]interfac

body["timestamp"] = formatTimestamp(ts)
body["fluentbit_tag"] = tag
body["_resource.type"] = "Fluentbit"

serialized, err := jsoniter.Marshal(body)
if err != nil {
Expand Down
2 changes: 1 addition & 1 deletion output/out_lm_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ func (p *TestPlugin) Environment(ctx unsafe.Pointer, key string) string {
return p.token
case "lm_url":
return p.url
case "lm_debug":
case "lmDebug":
return p.debug
}
return "not found"
Expand Down

0 comments on commit e489739

Please sign in to comment.