Can't add attribute

[Diego91RA]

Hello. When I try to add new attribute to the archiving list using Configurator app or by command in "Test Device", I see errors in log files.

Thu Jul 23 14:28:50 2020: Event tango://mpd-tof-sc-2.he.jinr.ru:10000/mpd/tof/hvsys_202_scalar/umes_14.idl5_archive not found in event callback map !!!
Thu Jul 23 14:29:02 2020: Event tango://mpd-tof-sc-2.he.jinr.ru:10000/mpd/tof/hvsys_202_scalar/umes_14.idl5_attr_conf not found in event callback map !!!
Tango exception
Severity = ERROR
Error reason = Consistency Error
Desc : Attempt to update the ttl for an attribute that does not exist. Attribute: tango://mpd-tof-sc-2.he.jinr.ru:10000/mpd/tof/hvsys_202_scalar/umes_14
Origin : HdbppTxUpdateTtl.hpp:store:100

Tango exception
Severity = ERROR
Error reason = Consistency Error
Desc : This attribute [tango://mpd-tof-sc-2.he.jinr.ru:10000/mpd/tof/hvsys_202_scalar/umes_14] does not exist in the database. Unable to work with this attribute until it is added.
Origin : DbConnection.cpp:fetchLastHistoryEvent:469

Tango exception
Severity = ERROR
Error reason = Consistency Error
Desc : This attribute [tango://mpd-tof-sc-2.he.jinr.ru:10000/mpd/tof/hvsys_202_scalar/umes_14] does not exist in the database. Unable to work with this attribute until it is added.
Origin : DbConnection.tpp:storeDataEvent:120

Tango exception
Severity = ERROR
Error reason = Consistency Error
Desc : This attribute [tango://mpd-tof-sc-2.he.jinr.ru:10000/mpd/tof/hvsys_202_scalar/umes_14] does not exist in the database. Unable to work with this attribute until it is added.
Origin : DbConnection.cpp:storeParameterEvent:301

Previously configured attributes, but deleted from ES, are added without errors again.

LibConfiguration property:

connect_string=user=hdb_event_sub password=hdbpp host=localhost port=5432 dbname=hdb
logging_level=error
log_file=false
log_console=false
libname=libhdb++timescale.so

PostgreSQL 11, TimescaleDB 1.6.0

Also I've recently updated hdbpp-cm, because there were problems with transactions from CM DS (picture from Zabbix). Can be connected?

[Diego91RA]

The same situation with clean install.
Debian 9, PSQL 12, TimescaleDB 1.7.2, Tango 9.3.4-rc6, all hdbpp repos are cloned from github.

@bourtemb @chedburgh any ideas?

[Diego91RA]

I've inserted configuration from console and it works.

[gscalam]

Hi, reason is that configuration in the DB has been moved from hdbpp-cm to hdbpp-es, but modification is still in progress and not yet merged in master branch, you should try integrated-build branch. @dlacoste-esrf should be working on this.

[Diego91RA]

Now I have error compiling integrated-build, HdbClient.h not found.

What version of libhdbpp should I use with TimescaleDB? https://github.com/tango-controls-hdbpp/libhdbpp-timescale/tree/master/thirdparty/libhdbpp or https://github.com/tango-controls-hdbpp/libhdbpp?

[dlacoste-esrf]

Hi,
you should use https://github.com/tango-controls-hdbpp/libhdbpp

there is as well a custom cmake flag FETCH_LIBHDBPP, so that libhdbpp is downloaded and built with the event subscriber.

Sorry for the sad state of the repositories right now, I'll try to fix it up soon.

[Diego91RA]

Hello.
I've compiled the DS with cmake flag. When I try to run it (with -v5 option) I have such output.

1597132187 [140323005146944] DEBUG sys/tools/es_test HdbEventSubscriber::init_device() create device sys/tools/es_test
1597132187 [140323005146944] DEBUG sys/tools/es_test subscribeRetryPeriod = 60
1597132187 [140323005146944] DEBUG sys/tools/es_test subscribeChangeAsFallback = 0
1597132187 [140323005146944] INFO sys/tools/es_test HdbEventSubscriber id=0
1597132187 [140323005146944] DEBUG sys/tools/es_test CONFIGURING CONTEXTS: adding ALWAYS <-> 0
1597132187 [140323005146944] DEBUG sys/tools/es_test CONFIGURING CONTEXTS: adding RUN <-> 1
1597132187 [140323005146944] DEBUG sys/tools/es_test CONFIGURING CONTEXTS: adding SHUTDOWN <-> 2
1597132187 [140323005146944] DEBUG sys/tools/es_test CONFIGURING CONTEXTS: adding SERVICE <-> 3
1597132187 [140323005146944] DEBUG dserver/hdb++es-srv/test Database::Database(): TANGO host mpd-ecal-sc-1.he.jinr.ru port 10000
1597132187 [140323005146944] DEBUG dserver/hdb++es-srv/test Entering DeviceImpl destructor for device sys/tools/es_test
1597132187 [140323005146944] DEBUG dserver/hdb++es-srv/test Leaving DeviceImpl destructor for device sys/tools/es_test
1597132187 [140323005146944] DEBUG dserver/hdb++es-srv/test Entering DeviceClass destructor for class HdbEventSubscriber
1597132187 [140323005146944] DEBUG dserver/hdb++es-srv/test Leaving DeviceClass destructor for class HdbEventSubscriber
1597132187 [140323005146944] DEBUG dserver/hdb++es-srv/test Entering DeviceImpl destructor for device dserver/hdb++es-srv/test
1597132187 [140323005146944] DEBUG dserver/hdb++es-srv/test Leaving DeviceImpl destructor for device dserver/hdb++es-srv/test
1597132187 [140323005146944] DEBUG dserver/hdb++es-srv/test Entering DeviceClass destructor for class DServer
1597132187 [140323005146944] DEBUG dserver/hdb++es-srv/test Leaving DeviceClass destructor for class DServer
Tango exception
Severity = ERROR
Error reason = API_MemoryAllocation
Desc : Can't allocate memory in server while building command(s) or device(s) for class number 1
Origin : DServer::init_device

Received a CORBA_Exception
Exiting

[gscalam]

I have the same issue, it looks like it is crashing in HdbppTimescaleDbUtils::extractConfig

[bourtemb]

It looks like Tango received a std::bad_alloc exception during the device server initialization.
https://github.com/tango-controls/cppTango/blob/8fd3b4d85ad43bfac8f0e50bfdee22a98ac0c574/cppapi/server/dserver.cpp#L385

[chedburgh]

Is no one following the combined guide on hdbpp-timescale-project?

https://github.com/tango-controls-hdbpp/hdbpp-timescale-project/blob/master/doc/build.md

Since some of the branches have been merged, you will need to update what the integrated project builds, but it should checkout and build everything for you. You no longer need to build the individual components.

[gscalam]

It looks like Tango received a std::bad_alloc exception during the device server initialization.
https://github.com/tango-controls/cppTango/blob/8fd3b4d85ad43bfac8f0e50bfdee22a98ac0c574/cppapi/server/dserver.cpp#L385

As I already said, this seems a bug in libhdbpp-timescale, since bypassing HdbppTimescaleDbUtils::extractConfig make it work.

[chedburgh]

Maybe this is version mismatch? Since some components have there updated branch merged. Some have not been tagged correctly either. Can I suggest trying:

hdbpp_cm - master
hdbpp_es - intergated branch
libhdbpp - master (v2.0)

If the issue is in HdbppTimescaleDbUtils::extractConfig(), just put some cout statements in and see what configuration parameter it is crashing on. Is there a config parameter without an equals sign?

There is information here on the config parameters: https://github.com/tango-controls-hdbpp/libhdbpp-timescale/blob/master/doc/configuration.md

[gscalam]

Maybe this is version mismatch? Since some components have there updated branch merged. Some have not been tagged correctly either. Can I suggest trying:

You are right! It happens only when using libhdbpp-timescale not project-build2 with libhdbpp master (which is actually v2). So HdbppTimescaleDbFactory::create_db is called with the wrong parameters ending with a vector configuration not initialized so making explode extractConfig

[gscalam]

The right combination for v.2 at the moment is
hdbpp_cm - master
hdbpp_es - integrated-build
libhdbpp - master (v2.0)
libhdbpp-timescale - project-build2

[chedburgh]

Excellent. Suggested TODO:

• I have opened tickets to tag some repos correctly, lets fix the tags ASAP.
• hdbpp-es needs the integrated build branch reviewing and merging ASAP, then tagging correctly.
• hdbpp-timescale-project needs its build updated to embed the know working tags.

Its vital the repositories are tagged correctly when changes occur, otherwise we can hit some madness.

[gscalam]

I am sorry, but I was (again) wrong: with the above combination I have

HdbClient: Error loading symbol getDBFactory from library: libhdb++timescale.so

I am still missing something.

[chedburgh]

DId you use libhdbpp-timescale branch project-build2? If so, try master.

[gscalam]

No master have the v1 signature of HdbppTimescaleDbFactory::create_db, probably batch-insert is the right one, testing now.

[chedburgh]

Its been a while since I was on this, but I think batch-insert just branches from project-build2. I may need to look into this tonight. Something seems to have been mixed up somewhere.

[gscalam]

Same error with batch-insert.

[chedburgh]

There seems to be something fishy here.

I made libhdbpp-timescale no longer dependent on libhdbpp. It includes the headers it needs itself. Yet I see reference to it in the cmake file. Maybe there is a missing commit from me on the project-build2 branch?

project-build2 is correct. Investigating for a few mins.

[chedburgh]

Right, libhdbpp-timescale is implementing getDBFactory, but not exporting it.

Perhaps we need to add

extern "C"
{
    typedef hdbpp::DBFactory *getDBFactory_t();
    hdbpp::DBFactory *getDBFactory();
}

To the end of AbstractDb.hpp? This needs to be outside the namespace. It can live in one of the other public headers also.

[gscalam]

I don't know.
With the same combination but using libhdbpp-mysql - libhdbpp2 branch it is working, but it looks quite similar

[chedburgh]

I have compiled and run a test application via the hdbpp-timescale-project without the error. I will push the changes to that project I made.

[chedburgh]

I also pushed the fix to libhdbpp-timescale (that I posted above). Its on the project-build2 branch. Using the hdbpp-timescale-project this should all just work. I don't have time to test individual components.

I'll setup a project board to get some of these tickets/PR's merged in an orderly fashion, if that helps.

Individually, as mentioned by @gscalamera I think the following will work:

hdbpp_cm - master
hdbpp_es - integrated-build
libhdbpp - master (v2.0)
libhdbpp-timescale - project-build2

[Diego91RA]

I've tried this set and ES still has memory allocation error.

hdbpp_cm - master
hdbpp_es - integrated-build
libhdbpp - master (v2.0)
libhdbpp-timescale - project-build2

hdbpp-timescale-project cmake has multiple errors, here are logs:
CMakeOutput.log
CMakeError.log

[dlacoste-esrf]

We have to fix the repos, but I am not sure how long it will take to make them clean again…

If you want a quick fix, you have to rebuild libhdbpp-timescale, but on the CMakeLists.txt in src there is a flag BYPASS_LIBHDBPP, it should be removed, as it is intended to be true for the new versions.
This way you will compile HdbClient.cpp on libhdbpp-timescale.
Then on this project, change the CMakeList.txt to link directly on the previous library, so in your CMakelists.txt, line 77 replace libhdb++.so.2 with libhdb++timescale.so.2. This way it should be ok.

Sorry it's a bit of a mess right now, we will try to fix it as fast as possible

[chedburgh]

What are you compiling on? I see pthread missing in the errors!

CMakeFiles/cmTC_ce417.dir/src.c.o: In function `main':
src.c:(.text+0x2f): undefined reference to `pthread_create'
src.c:(.text+0x3b): undefined reference to `pthread_detach'
src.c:(.text+0x47): undefined reference to `pthread_cancel'
src.c:(.text+0x58): undefined reference to `pthread_join'
src.c:(.text+0x6c): undefined reference to `pthread_atfork'

[chedburgh]

I updated the hdbpp-timescale-project to reflect the merged and tagged dependencies. Everything is now set for v2.0.0 apart from libhdbpp-timescale.

This works and builds for me on Ubuntu 20. Afraid I don't have time to go deeper into the other errors.

[Diego91RA]

@dlacoste-esrf It's ok, I'm just trying to test and help fixing this, take your time.

[Diego91RA]

What are you compiling on? I see pthread missing in the errors!

It's Debian 9 with manually updated CMake 3.18 to match requirements.

Freshly installed Debian 10 with CMake 3.13 has similar problems with pthread.
CMakeError.log

Also there is an error detecting PostgreSQL:

CMake Error at /usr/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find PostgreSQL (missing: PostgreSQL_TYPE_INCLUDE_DIR) (found
version "12.4 (Debian 12.4-1.pgdg100+1)")
Call Stack (most recent call first):
/usr/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-3.13/Modules/FindPostgreSQL.cmake:175 (find_package_handle_standard_args)
external/libhdbpp-timescale/thirdparty/libpqxx/src/CMakeLists.txt:15 (find_package)

[gscalam]

Regarding the PostgreSQL error you can try passing -DPostgreSQL_TYPE_INCLUDE_DIR=/usr/include/postgresql to cmake

[Diego91RA]

If you want a quick fix, you have to rebuild libhdbpp-timescale, but on the CMakeLists.txt in src there is a flag BYPASS_LIBHDBPP, it should be removed, as it is intended to be true for the new versions.
This way you will compile HdbClient.cpp on libhdbpp-timescale.
Then on this project, change the CMakeList.txt to link directly on the previous library, so in your CMakelists.txt, line 77 replace libhdb++.so.2 with libhdb++timescale.so.2. This way it should be ok.

Thanks, it works. I've successfully subscribed to an attr and can see data via Viewer. I hope that soon we will be able to install the HDB without any edits. I'll fork repos as a temporary solution.

Regarding the PostgreSQL error you can try passing -DPostgreSQL_TYPE_INCLUDE_DIR=/usr/include/postgresql to cmake

It solved problem with include dir, but PThread is still missing.