Skip to content

Commit

Permalink
Add first/last timestamps to expire tables
Browse files Browse the repository at this point in the history
This adds two timestamp (with timezone) columns to all expire tables
called "first" and "last". When an entry is added, both are set to the
current (transaction) timestamp. When an entry already exists a new
insert will result in the "last" timestamp being updated.

Having these two timestamps allows various expire/updating strategies,
for instance:
* update oldest entry
* update entry that didn't change for the longest time
* update older entries but only if there are no recent changes, which
  indicates that there might be more changes coming

For backwards compatibility the code detects which table format is used
and falls back to the old behaviour if the timestamp columns aren't
there.
  • Loading branch information
joto committed Sep 18, 2023
1 parent ca7a1df commit fb0d5d5
Showing 1 changed file with 25 additions and 10 deletions.
35 changes: 25 additions & 10 deletions src/expire-output.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -59,10 +59,22 @@ expire_output_t::output_tiles_to_table(quadkey_list_t const &tiles_at_maxzoom,

pg_conn_t connection{conninfo};

connection.exec("PREPARE insert_tiles(int4, int4, int4) AS"
" INSERT INTO {} (zoom, x, y) VALUES ($1, $2, $3)"
" ON CONFLICT DO NOTHING",
qn);
auto const result = connection.exec("SELECT * FROM {} LIMIT 1", qn);

if (result.num_fields() == 3) {
// old format with fields: zoom, x, y
connection.exec("PREPARE insert_tiles(int4, int4, int4) AS"
" INSERT INTO {} (zoom, x, y) VALUES ($1, $2, $3)"
" ON CONFLICT DO NOTHING",
qn);
} else {
// new format with fields: zoom, x, y, first, last
connection.exec("PREPARE insert_tiles(int4, int4, int4) AS"
" INSERT INTO {} (zoom, x, y) VALUES ($1, $2, $3)"
" ON CONFLICT (zoom, x, y)"
" DO UPDATE SET last = CURRENT_TIMESTAMP(0)",
qn);
}

auto const count = for_each_tile(
tiles_at_maxzoom, m_minzoom, m_maxzoom, [&](tile_t const &tile) {
Expand All @@ -76,10 +88,13 @@ expire_output_t::output_tiles_to_table(quadkey_list_t const &tiles_at_maxzoom,
void expire_output_t::create_output_table(pg_conn_t const &connection) const
{
auto const qn = qualified_name(m_schema, m_table);
connection.exec("CREATE TABLE IF NOT EXISTS {} ("
" zoom int4 NOT NULL,"
" x int4 NOT NULL,"
" y int4 NOT NULL,"
" PRIMARY KEY (zoom, x, y))",
qn);
connection.exec(
"CREATE TABLE IF NOT EXISTS {} ("
" zoom int4 NOT NULL,"
" x int4 NOT NULL,"
" y int4 NOT NULL,"
" first timestamp with time zone DEFAULT CURRENT_TIMESTAMP(0),"
" last timestamp with time zone DEFAULT CURRENT_TIMESTAMP(0),"
" PRIMARY KEY (zoom, x, y))",
qn);
}

0 comments on commit fb0d5d5

Please sign in to comment.