Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support tls ? #3

Open
zljohn-ux opened this issue Apr 18, 2018 · 3 comments
Open

support tls ? #3

zljohn-ux opened this issue Apr 18, 2018 · 3 comments

Comments

@zljohn-ux
Copy link

if use tls.Listen(), netpoll.must Will fail,The reason for the failure is "could not get file descriptor", so support tls?

@acgreek
Copy link

acgreek commented Mar 12, 2021

I was able to get TLS to work using netpoll.NewDesc(fd, netpoll.EventRead)

to get the fd from the tls.Conn, I had to use reflection

    func FDFromTLSConn(conn net.Conn) int {
   	tls := reflect.TypeOf(conn) == reflect.TypeOf(&tls.Conn{})
   	tcpConn := reflect.Indirect(reflect.ValueOf(conn)).FieldByName("conn")
   	if tls {
    		tcpConn = reflect.Indirect(tcpConn.Elem())
   	}
   	fdVal := tcpConn.FieldByName("fd")
   	pfdVal := reflect.Indirect(fdVal).FieldByName("pfd")
   
   	return int(pfdVal.FieldByName("Sysfd").Int())
   }

you also have to set the fd to non-blocking

    func setNonblock(fd int, nonblocking bool) (err error) {
    	return syscall.SetNonblock(fd, nonblocking)
    }

agnivade added a commit to mattermost/mattermost that referenced this issue Mar 18, 2021
A *crypto/tls.Conn does not expose the underlying TCP connection
or even a File method to get the underlying file descriptor
like the way a *net/TCPConn does. Therefore the netpoll code would
fail to get the file descriptor.

Relevant issue here: mailru/easygo#3

It is indeed possible to use reflect black magic to get the unexported
member, but I have found unexpected errors during writing to the websocket
by getting the file descriptor this way. I do not want to spend time investigating
this especially since this is already released.

Once this is out, we can decide on the right way to fix this, most probably
by proposing to expose the File method or some other way.

https://mattermost.atlassian.net/browse/MM-34000

```release-note
Fix an issue where websockets wouldn't work with TLS connections.
In that case, we just fall back to the way it works for Windows machines,
which is to use a separate goroutine for reader connection.
```
@agnivade
Copy link

Even that doesn't work properly. I kept getting errors during writing to the websocket:
write tcp [::1]:8065->[::1]:22191: write: bad file descriptor
write tcp [::1]:8065->[::1]:22391: write: connection reset by peer

This needs to be supported properly from the standard library.

@acgreek
Copy link

acgreek commented Mar 18, 2021

I'm on a mac and that worked for me, so it could be the difference between kqueue and epoll. I replaced this code with opening a tcp connection and then promoting it to a tls.Conn

 ln, err := net.Listen("tcp", *addr)
 if err != nil {
     panic(err)
 }
 defer ln.Close()
 tcpConn, err := ln.(*net.TCPListener).AcceptTCP()
 if err != nil {
     panic(err)
 }
 tlsConn := tls.Server(tcpConn, tlsConfig)
 if err := tlsConn.Handshake(); err != nil {
     panic(err)
 }
 desc := netpoll.Must(netpoll.HandleRead(tcpConn))

agnivade added a commit to mattermost/mattermost that referenced this issue Mar 18, 2021
* MM-34000: Use non-epoll mode for TLS connections

A *crypto/tls.Conn does not expose the underlying TCP connection
or even a File method to get the underlying file descriptor
like the way a *net/TCPConn does. Therefore the netpoll code would
fail to get the file descriptor.

Relevant issue here: mailru/easygo#3

It is indeed possible to use reflect black magic to get the unexported
member, but I have found unexpected errors during writing to the websocket
by getting the file descriptor this way. I do not want to spend time investigating
this especially since this is already released.

Once this is out, we can decide on the right way to fix this, most probably
by proposing to expose the File method or some other way.

https://mattermost.atlassian.net/browse/MM-34000

```release-note
Fix an issue where websockets wouldn't work with TLS connections.
In that case, we just fall back to the way it works for Windows machines,
which is to use a separate goroutine for reader connection.
```

* Ignore logging errors on non-epoll

On non-epoll systems, we needed to return an error
to break from the loop. But in that case, there is no
need to log the error
agnivade added a commit to mattermost/mattermost that referenced this issue Mar 18, 2021
* MM-34000: Use non-epoll mode for TLS connections

A *crypto/tls.Conn does not expose the underlying TCP connection
or even a File method to get the underlying file descriptor
like the way a *net/TCPConn does. Therefore the netpoll code would
fail to get the file descriptor.

Relevant issue here: mailru/easygo#3

It is indeed possible to use reflect black magic to get the unexported
member, but I have found unexpected errors during writing to the websocket
by getting the file descriptor this way. I do not want to spend time investigating
this especially since this is already released.

Once this is out, we can decide on the right way to fix this, most probably
by proposing to expose the File method or some other way.

https://mattermost.atlassian.net/browse/MM-34000

```release-note
Fix an issue where websockets wouldn't work with TLS connections.
In that case, we just fall back to the way it works for Windows machines,
which is to use a separate goroutine for reader connection.
```

* Ignore logging errors on non-epoll

On non-epoll systems, we needed to return an error
to break from the loop. But in that case, there is no
need to log the error
mattermost-build pushed a commit to mattermost-build/mattermost that referenced this issue Mar 18, 2021
* MM-34000: Use non-epoll mode for TLS connections

A *crypto/tls.Conn does not expose the underlying TCP connection
or even a File method to get the underlying file descriptor
like the way a *net/TCPConn does. Therefore the netpoll code would
fail to get the file descriptor.

Relevant issue here: mailru/easygo#3

It is indeed possible to use reflect black magic to get the unexported
member, but I have found unexpected errors during writing to the websocket
by getting the file descriptor this way. I do not want to spend time investigating
this especially since this is already released.

Once this is out, we can decide on the right way to fix this, most probably
by proposing to expose the File method or some other way.

https://mattermost.atlassian.net/browse/MM-34000

```release-note
Fix an issue where websockets wouldn't work with TLS connections.
In that case, we just fall back to the way it works for Windows machines,
which is to use a separate goroutine for reader connection.
```

* Ignore logging errors on non-epoll

On non-epoll systems, we needed to return an error
to break from the loop. But in that case, there is no
need to log the error

(cherry picked from commit 2743089)
agnivade added a commit to mattermost/mattermost that referenced this issue Mar 18, 2021
* MM-33233: Fix double close of webconn pump (#17026)

Automatic Merge

* MM-34000: Use non-epoll mode for TLS connections (#17172)

* MM-34000: Use non-epoll mode for TLS connections

A *crypto/tls.Conn does not expose the underlying TCP connection
or even a File method to get the underlying file descriptor
like the way a *net/TCPConn does. Therefore the netpoll code would
fail to get the file descriptor.

Relevant issue here: mailru/easygo#3

It is indeed possible to use reflect black magic to get the unexported
member, but I have found unexpected errors during writing to the websocket
by getting the file descriptor this way. I do not want to spend time investigating
this especially since this is already released.

Once this is out, we can decide on the right way to fix this, most probably
by proposing to expose the File method or some other way.

https://mattermost.atlassian.net/browse/MM-34000

```release-note
Fix an issue where websockets wouldn't work with TLS connections.
In that case, we just fall back to the way it works for Windows machines,
which is to use a separate goroutine for reader connection.
```

* Ignore logging errors on non-epoll

On non-epoll systems, we needed to return an error
to break from the loop. But in that case, there is no
need to log the error
jwilander added a commit to mattermost/mattermost that referenced this issue Mar 30, 2021
* Translations update from Weblate (#17133)

* Translated using Weblate (Swedish)

Currently translated at 98.9% (2162 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/sv/

Translated using Weblate (Swedish)

Currently translated at 100.0% (2165 of 2165 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/sv/

* Translated using Weblate (Bulgarian)

Currently translated at 100.0% (2165 of 2165 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/bg/

* Update translation files

Updated by "Cleanup translation files" hook in Weblate.

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/

Update translation files

Updated by "Cleanup translation files" hook in Weblate.

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/

* Translated using Weblate (Dutch)

Currently translated at 100.0% (2186 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/nl/

Translated using Weblate (Dutch)

Currently translated at 100.0% (2167 of 2167 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/nl/

Translated using Weblate (Dutch)

Currently translated at 100.0% (2167 of 2167 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/nl/

* Translated using Weblate (Romanian)

Currently translated at 98.5% (2154 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/ro/

* Translated using Weblate (Korean)

Currently translated at 83.8% (1832 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/ko/

* Translated using Weblate (Chinese (Simplified))

Currently translated at 97.7% (2137 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/zh_Hans/

* Translated using Weblate (Dutch)

Currently translated at 100.0% (2186 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/nl/

* Translated using Weblate (Turkish)

Currently translated at 100.0% (2186 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/tr/

* Translated using Weblate (Bulgarian)

Currently translated at 100.0% (2186 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/bg/

Co-authored-by: MArtin Johnson <martinjohnson@bahnhof.se>
Co-authored-by: Nikolai Zahariev <nikolaiz@yahoo.com>
Co-authored-by: Tom De Moor <tom@controlaltdieliet.be>
Co-authored-by: Viorel-Cosmin Miron <cosmin@uhlhost.net>
Co-authored-by: Ji-Hyeon Gim <potatogim@potatogim.net>
Co-authored-by: aeomin <lin@aeomin.net>
Co-authored-by: Kaya Zeren <kayazeren@gmail.com>

* Extracting mail service into shared libs (#17030)

* Extracting mail service into shared libs

* Fixing golangci-lint

* MM-31054 demote user (#16990)

* changing SchemeAdmin to false on demotion

* Added test

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* MM-33789: Revert fallback to master for GetAllProfilesInChannel (#17119)

* MM-33789: Revert fallback to master for GetAllProfilesInChannel

This fixes a regression introduced in #16911.

It was causing problems with too many invalidations and overloading the writer instance for big installations.
Reverting this does not affect correctness at all because it was done out of abundance of caution and the
idea at that point was it was to be done for all caches.

https://mattermost.atlassian.net/browse/MM-33789

```release-note
NONE
```

* fix gofmt issues

* api4/user: add promte/demote uset to local router (#17036)

* update translation string id to be more descriptive (#17002)

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* MM-33836: Detect and upgrade incorrect HTTP version for websocket handshakes (#17142)

Our proxy configuration was historically incorrect, due to which
a lot of customers have that in their setups. As a result, strictly
following the websocket RFC results in a breaking change.

For now, we transparently upgrade the version header to 1.1, if we detect 1.0.
If a client was sending 1.0, it wouldn't have worked anyways because persistent
connections were introduced from 1.1 onwards.

https://mattermost.atlassian.net/browse/MM-33836

```release-note
WebSocket handshakes done with HTTP version lower than 1.1 will result in a warning,
and the server will transparently upgrade the version to 1.1 to comply with the
websocket RFC.

This is done to work around incorrect nginx (and other proxy) configs that do not set
the proxy_http_version directive to 1.1.

This facility will be removed in a future Mattermost version and it is strongly recommended
to fix the proxy configuration to correctly use the websocket protocol.
```

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* unparam lint (#16927)

* fix "always receives ..." lint err

* add unparam lint check

* fix failed test

* rm details param

* ignore unparam lint

* magic string replaced with model.NewRandomString

* rm unused enableComplianceFeatures param

* generate random message inside createPost

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* Bump Incident Collaboration plugin to v1.6.0 (#17143)

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* MM-31396: Fix a possible deadlock with SidebarCategories (#17109)

This is a deadlock due to reversed locking order of the SidebarChannels
and SidebarCategories table.

I could not find the exact culprit query from the deadlock output
because it only shows the last query a transaction is running. And from looking
at the code, the only query that runs "UPDATE SidebarCategories SET DisplayName = ?, Sorting = ? WHERE Id = ?"
is UpdateSidebarCategories. But for the deadlock to happen, it has to lock
SidebarChannels _first_, and then _then_ lock SidebarCategories.

Looking a bit more throughly, I found that DeleteSidebarCategory does indeed
lock the tables in an inverse way and if DeleteSidebarCategory runs concurrently with
UpdateSidebarCategories, they will deadlock.

Here's how it will happen.

```
tx1
DELETE FROM SidebarChannels WHERE CategoryId = 'xx';

tx2
UPDATE SidebarCategories SET DisplayName='dn' WHERE Id='xx';

tx2
DELETE FROM SidebarChannels WHERE (ChannelId IN ('yy') AND CategoryId = 'xx');

tx1
DELETE FROM SidebarCategories WHERE Id = 'xx';
```

And then we see:
ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction

To fix this, we simply reorder the Delete query to lock the SidebarCategories first,
and then SidebarChannels.

In fact, any transaction updating/deleting rows from those two tables should always operate on that order
if possible.

https://mattermost.atlassian.net/browse/MM-31396

```release-note
Fixed a database deadlock that can happen if a sidebar category is updated and deleted at the same time.
```

* Moving markdown to corelibs (#16916)

* Moving markdown to corelibs

* Fix golangci-lint problem

* Renaming corelibs to pkg

* Fixing golangci-lint

* Renaming from pkg to shared

* Fixing gofmt

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* Improve file search results with postgres (#17112)

* Improve file search results with postgres

* Adding channel id to search results

* Fixing file info saves

* [MM-33498] - Correct Admin Console Reference (#17076)

* [MM-33498] - Correct Admin Console Reference

* Fix translations

* Fix translations

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* [MM-33603] Fix nil dereference panic in (*App).CreatePost() (#17124)

* Fix possible nil dereference

* fixed nil dereference for unfollowed thread

Co-authored-by: Eli Yukelzon <reflog@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* Mm 29605 read permission s3 bucket (#16977)

Automatic Merge

* Making the posts tests a bit more robust (#17151)

* Address outstanding merging issues with cloud branch (#17160)

* Disable unparam (#17167)

Automatic Merge

* MM-34000: Use non-epoll mode for TLS connections (#17172)

* MM-34000: Use non-epoll mode for TLS connections

A *crypto/tls.Conn does not expose the underlying TCP connection
or even a File method to get the underlying file descriptor
like the way a *net/TCPConn does. Therefore the netpoll code would
fail to get the file descriptor.

Relevant issue here: mailru/easygo#3

It is indeed possible to use reflect black magic to get the unexported
member, but I have found unexpected errors during writing to the websocket
by getting the file descriptor this way. I do not want to spend time investigating
this especially since this is already released.

Once this is out, we can decide on the right way to fix this, most probably
by proposing to expose the File method or some other way.

https://mattermost.atlassian.net/browse/MM-34000

```release-note
Fix an issue where websockets wouldn't work with TLS connections.
In that case, we just fall back to the way it works for Windows machines,
which is to use a separate goroutine for reader connection.
```

* Ignore logging errors on non-epoll

On non-epoll systems, we needed to return an error
to break from the loop. But in that case, there is no
need to log the error

* MM-34080: Removing sqlite entirely (#17188)

* MM-34080: Removing sqlite entirely

The initial commit missed removing this blank import.
So the library still remained in our vendor directory.
Removing it for good now.

Bye bye sqlite.

https://mattermost.atlassian.net/browse/MM-34080

* fix go.mod

* store/post: fix searching for phrases in postgres (#17042)

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* MM-34124: Enable race tests on master branch (#17192)

Automatic Merge

* Bump version (#17195)

Co-authored-by: Mmbot <mmbot@mattermost>

* Makefile: Refactor test-server-race (#17196)

test-server-race wasn't using the same set of steps
that the test-server step did. Therefore one test was failing.

Refactored it such that scripts/test.sh can be used to run
normal and race tests as well

```release-note
NONE
```

* MM-28090 User settings api when ldap sync (#16822)

Automatic Merge

* MM-29607 allow user to deactivate MFA even when disabled (#16994)

Automatic Merge

* Fix initialism errors (PR-3)  (#17062)

* Fix initialism errors

* Revert some changes and regenerate file

* Update client4.go

* Update group_test.go

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* [MM-34129] - Cloud 'Congratulations!' email format broken (#17198)

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* Revert "Fix initialism errors (PR-3)  (#17062)" (#17202)

This reverts commit ea61458. This was causing panic in the plugins because the client and the plugin API changed with this PR

* MM-33544 is_following prop in getPosts API methods (#17093)

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* Translations update from Weblate (#17191)

* Translated using Weblate (Dutch)

Currently translated at 100.0% (2186 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/nl/

Translated using Weblate (Dutch)

Currently translated at 100.0% (2186 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/nl/

Translated using Weblate (Dutch)

Currently translated at 100.0% (2186 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/nl/

* Translated using Weblate (Turkish)

Currently translated at 100.0% (2186 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/tr/

* Translated using Weblate (Japanese)

Currently translated at 98.5% (2155 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/ja/

* Translated using Weblate (French)

Currently translated at 83.8% (1833 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/fr/

* Translated using Weblate (French)

Currently translated at 83.9% (1836 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/fr/

* Translated using Weblate (Spanish)

Currently translated at 93.5% (2044 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/es/

* Translated using Weblate (Russian)

Currently translated at 95.1% (2080 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/ru/

* Translated using Weblate (Swedish)

Currently translated at 100.0% (2186 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/sv/

* Translated using Weblate (French)

Currently translated at 84.0% (1837 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/fr/

* Translated using Weblate (Russian)

Currently translated at 95.1% (2080 of 2186 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/ru/

Co-authored-by: Tom De Moor <tom@controlaltdieliet.be>
Co-authored-by: Kaya Zeren <kayazeren@gmail.com>
Co-authored-by: kaakaa <stooner.hoe@gmail.com>
Co-authored-by: Pierre JENICOT <pierre.jenicot@ac-lille.fr>
Co-authored-by: Cyril LD <c.laguilhon.debat@petitclou.de>
Co-authored-by: jesus.espino <jesus@mattermost.com>
Co-authored-by: Edward Smirnov <ed@microolap.com>
Co-authored-by: MArtin Johnson <martinjohnson@bahnhof.se>
Co-authored-by: Elisabeth Kulzer <elisabeth.kulzer@mattermost.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* Split CWS url in two: public and API (#17100)

Signed-off-by: Mario de Frutos <mario@defrutos.org>

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* Stop trying to build mattermost-redux in CI (#17206)

* [MM-33794] Improve password generation during bulk import (#17147)

Automatic Merge

* revertws (#17216)

* Revert "MM-34000: Use non-epoll mode for TLS connections (#17172)"

This reverts commit 2743089.

* Revert "MM-33233: Fix double close of webconn pump (#17026)"

This reverts commit 0f98620.

* Revert "MM-33836: Detect and upgrade incorrect HTTP version for websocket handshakes (#17142)"

This reverts commit 4c5ea07.

* revert i18n

* Revert "MM-21012: Revamp websocket implementation (#16620)"

This reverts commit a246104.

* fix go.mod

* Trigger CI

* [MM-34166] Fix Flaky Test TestGenerateSupportPacket (#17207)

* fix flaky tests

* Revert "fix flaky tests"

This reverts commit 255ce27.

* Fix flaky test

* Fix equal

* MM-33893: Disable TCP_NO_DELAY for websocket connections  (#17129)

* MM-33893: Disable TCP_NO_DELAY for websocket connections

In very large installations, websocket messages cause too much
traffic congestion by sending too small packets and thereby cause
a drop in throughput.

To counter this, we disable the TCP_NO_DELAY flag for websocket
connections. This has shown to give noticeable improvements in
load tests.

We wrap this in a feature flag for now to let it soak in Community
first.

```release-note
NONE
```

https://mattermost.atlassian.net/browse/MM-33893

* fix gorilla specific conn

* MM-34171: Fix racy test TestSentry (#17215)

* MM-34171: Fix racy test TestSentry

The NewServer call sets the global mlog Info variable.
But before that, if we call UpdateConfig with the functional options,
then the store.Set method will call mlog.Info before
it could be set.

To fix that, we prepare the updated config
and pass that directly to NewServer to avoid
having to call UpdateConfig.

https://mattermost.atlassian.net/browse/MM-34171

```release-note
NONE
```

* disable watcher

* Trying yet again

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* app/slashcommands/command_loadtest: fix decoding error (#17199)

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* MM-33945: Update dependencies (#17201)

* MM-33945: Update dependencies

Ran `make update-dependencies`

https://mattermost.atlassian.net/browse/MM-33945

```release-notes
NONE
```

* fix test

* Update to correct mailing address in outgoing server and portal email (#17223)

* Update cloud_welcome_email.html

* Update en.json

* Prevent User.Timezone field to overflow DB column capacity (#17220)

* MM-34086: Skippping TestStartServerTLSOverwriteCipher (#17225)

https://mattermost.atlassian.net/browse/MM-34086

```release-note
NONE
```

* MM-34271: Skip test TestLoggingAfterInitialized (#17226)

https://mattermost.atlassian.net/browse/MM-34271

```release-note
NONE
```

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* Upgrade db and version (#17230)

* [MM-34164] Fix flaky TestS3FileBackend (#17229)

* Fix flaky test

* Disable dumping data on error

* [MM-33640] storetest/team_store: use NewTestId for fields like names etc. (#17204)

* storetest/team_store: use NewTestId for fields like names etc.

* reflect review comments

* fix tests

* we love good old C

* remove rand

* [MM-32822] Improve JobServer state handling (#16959)

Automatic Merge

* Remove server from WebsocketRouter (#17238)

App contains server.
Server contains WebsocketRouter.

There is no need for WebsocketRouter to contain
server too. As is evident because the code never used it.

```release-note
NONE
```

* pre-package Incident Collaboration v1.7.0 (#17234)

* MM-33402: Invalidated the LRU cache for channel members after moderat… (#17171)

* MM-33402: Invalidated the LRU cache for channel members after moderation changes.

* MM-33402: Handles cache invalidation error.

* Revert "MM-33402: Handles cache invalidation error."

This reverts commit 2d30d24.

* MM-33402: Un-exports method.

* MM-33402: Removes log warning from previous code use.

* MM-33402: Handles possible (store.ChannelStore).GetMembers error.

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* MM-27380 Move collapsed categories to server (#17194)

* Translations update from Weblate (#17298)

* Added translation using Weblate (English (Australia))

* Translated using Weblate (English (Australia))

Currently translated at 100.0% (2189 of 2189 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/en_AU/

* Translated using Weblate (Russian)

Currently translated at 99.8% (2187 of 2190 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/ru/

Translated using Weblate (Russian)

Currently translated at 100.0% (2189 of 2189 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/ru/

Translated using Weblate (Russian)

Currently translated at 98.4% (2156 of 2189 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/ru/

* Translated using Weblate (Dutch)

Currently translated at 100.0% (2190 of 2190 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/nl/

Translated using Weblate (Dutch)

Currently translated at 100.0% (2189 of 2189 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/nl/

* Translated using Weblate (Chinese (Simplified))

Currently translated at 99.3% (2175 of 2189 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/zh_Hans/

* Update translation files

Updated by "Cleanup translation files" hook in Weblate.

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/

* Translated using Weblate (English (Australia))

Currently translated at 100.0% (2190 of 2190 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/en_AU/

Translated using Weblate (English (Australia))

Currently translated at 100.0% (2189 of 2189 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/en_AU/

* Translated using Weblate (Turkish)

Currently translated at 100.0% (2190 of 2190 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/tr/

Translated using Weblate (Turkish)

Currently translated at 100.0% (2189 of 2189 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/tr/

* Translated using Weblate (Korean)

Currently translated at 84.6% (1854 of 2190 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/ko/

* Deleted translation using Weblate (English (Australia))

* Translated using Weblate (Swedish)

Currently translated at 100.0% (2191 of 2191 strings)

Translation: mattermost-languages-shipped/mattermost-server
Translate-URL: https://translate.mattermost.com/projects/mattermost/mattermost-server_master/sv/

Co-authored-by: Matthew Williams <en_AU@controlaltdieliet.be>
Co-authored-by: Edward Smirnov <ed@microolap.com>
Co-authored-by: Tom De Moor <tom@controlaltdieliet.be>
Co-authored-by: aeomin <lin@aeomin.net>
Co-authored-by: Matthew Williams <Matthew.Williams@outlook.com.au>
Co-authored-by: Kaya Zeren <kayazeren@gmail.com>
Co-authored-by: teamzamong <heekang@korea.ac.kr>
Co-authored-by: MArtin Johnson <martinjohnson@bahnhof.se>

* fix race in unit test re logging (#17235)

* Add feature flag for apps (#16851)

* Add feature flag for apps

* Update default to false

* Add plugin version Feature Flag

* Fix typo

* Only force shutdown, and leave the enable status dependant on the user (defaulting to enable)

* Remove unneeded tracking of status

* Handle plugin init on startup for locally installed plugin

* MM-33818: Add replica lag metric (#17148)

* MM-33818: Add replica lag metric

We add two new metrics for monitoring replica lag:
- Monitor absolute lag based on binlog distance/transaction queue length.
- Monitor time taken for the replica to catch up.

To achieve this, we add a config setting to run a user defined SQL query
on the database.

We need to specify a separate datasource field as part of the config because
in some databases, querying the replica lag value requires elevated credentials
which are not needed for usual running of the application, and can even be a security risk.

Arguably, a peculiar part of the design is the requirement of the query output to be in a (node, value)
format. But since from the application, the SQL query is a black box and the user can set any query
they want, we cannot, in any way templatize this.

And as an extra note, freno also does it in a similar way.

The last bit is because we need to have a separate datasources, now we consume one extra connection
rather than sharing it with the pool. This is an unfortunate result of the design, and while one extra
connection doesn't make much of a difference in a single-tenant scenario. It does make so, in a multi-tenant scenario.
But in a multi-tenant scenario, the expectation would already be to use a connection pool. So this
is not a big concern.

https://mattermost.atlassian.net/browse/MM-33818

```release-note
Two new gauge metrics were added:
mattermost_db_replica_lag_abs and mattermost_db_replica_lag_time, both
containing a label of "node", signifying which db host is the metric from.

These metrics signify the replica lag in absolute terms and in the time dimension
capturing the whole picture of replica lag.

To use these metrics, a separate config section ReplicaLagSettings was added
under SqlSettings. This is an array of maps which contain three keys: DataSource,
QueryAbsoluteLag, and QueryTimeLag. Each map entry is for a single replica instance.

DataSource contains the DB credentials to connect to the replica instance.

QueryAbsoluteLag is a plain SQL query that must return a single row of which the first column
must be the node value of the Prometheus metric, and the second column must be the value of the lag.

QueryTimeLag is the same as above, but used to measure the time lag.

As an example, for AWS Aurora instances, the QueryAbsoluteLag can be:

select server_id, highest_lsn_rcvd-durable_lsn as bindiff from aurora_global_db_instance_status() where server_id=<>

and QueryTimeLag can be:

select server_id, visibility_lag_in_msec from aurora_global_db_instance_status() where server_id=<>

For MySQL Group Replication, the absolute lag can be measured from the number of pending transactions
in the applier queue:

select member_id, count_transaction_remote_in_applier_queue FROM performance_schema.replication_group_member_stats where member_id=<>

Overall, what query to choose is left to the administrator, and depending on the database and need, an appropriate
query can be chosen.
```

* Trigger CI

* Fix tests

* address review comments

* Remove t.Parallel

It was spawning too many connections,
and overloading the docker container.

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>

* Update PULL_REQUEST_TEMPLATE.md (#17208)

Automatic Merge

* MM-32882: Add an unread badge/marker to the Main Menu icon and the ‘Plugin Marketplace’ Menu Item (#16992)

* Fix lint issue

* Fix typo

Co-authored-by: Weblate (bot) <hosted@weblate.org>
Co-authored-by: MArtin Johnson <martinjohnson@bahnhof.se>
Co-authored-by: Nikolai Zahariev <nikolaiz@yahoo.com>
Co-authored-by: Tom De Moor <tom@controlaltdieliet.be>
Co-authored-by: Viorel-Cosmin Miron <cosmin@uhlhost.net>
Co-authored-by: Ji-Hyeon Gim <potatogim@potatogim.net>
Co-authored-by: aeomin <lin@aeomin.net>
Co-authored-by: Kaya Zeren <kayazeren@gmail.com>
Co-authored-by: Jesús Espino <jespinog@gmail.com>
Co-authored-by: Anurag Shivarathri <anurag6713@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Agniva De Sarker <agnivade@yahoo.co.in>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Mahmudul Haque <mahmudulhaque@protonmail.com>
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
Co-authored-by: Allan Guwatudde <guwats10@gmail.com>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Co-authored-by: Eli Yukelzon <reflog@gmail.com>
Co-authored-by: Max Erenberg <max.erenberg@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Mmbot <mmbot@mattermost>
Co-authored-by: Haardik Dharma <dharmahaardik08@gmail.com>
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
Co-authored-by: kaakaa <stooner.hoe@gmail.com>
Co-authored-by: Pierre JENICOT <pierre.jenicot@ac-lille.fr>
Co-authored-by: Cyril LD <c.laguilhon.debat@petitclou.de>
Co-authored-by: jesus.espino <jesus@mattermost.com>
Co-authored-by: Edward Smirnov <ed@microolap.com>
Co-authored-by: Elisabeth Kulzer <elisabeth.kulzer@mattermost.com>
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
Co-authored-by: Hossein <hahmadia@users.noreply.github.com>
Co-authored-by: Jason Blais <13119842+jasonblais@users.noreply.github.com>
Co-authored-by: Elisabeth Kulzer <elikul@elikul.de>
Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com>
Co-authored-by: Martin Kraft <martinkraft@gmail.com>
Co-authored-by: Matthew Williams <en_AU@controlaltdieliet.be>
Co-authored-by: Matthew Williams <Matthew.Williams@outlook.com.au>
Co-authored-by: teamzamong <heekang@korea.ac.kr>
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
Co-authored-by: Daniel Espino García <larkox@gmail.com>
Co-authored-by: Amy Blais <amy_blais@hotmail.com>
Co-authored-by: catalintomai <56169943+catalintomai@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants