Gorse v0.3.2

Feature

• Support modify configuration via environment variables (#359).
  There are 8 environment variables available:

Environment Variable	Configuration
GORSE_CACHE_STORE	cache_store	database for caching.
GORSE_DATA_STORE	data_store	database for persist data.
GORSE_MASTER_PORT	port	master port
GORSE_MASTER_HOST	host	master host
GORSE_MASTER_HTTP_PORT	http_port	HTTP API port
GORSE_MASTER_HTTP_HOST	http_host	HTTP API host
GORSE_MASTER_JOBS	n_jobs	number of working jobs
GORSE_SERVER_API_KEY	api_key	secret key for RESTful APIs

• (Experimental) Support IVF-based neighborhood searching (#363).
• (Experimental) Support HNSW based recommended items searching (#368).

Fix

• Split large Redis writes to batches (#352).
• Allow empty JSON string in relational databases (#355).
• Fix offset overflow in recommend API (#365).
• Optimize dashboard overview page layout (#369).

Upgrade Guide

• Configuration: IVF-based neighborhood searching and HNSW based recommended items searching are experimental features.
  • Use enable_xxx_index to enable experimental features.
  • xxx_index_recall is the expected recall of approximate searching.
  • xxx_index_fit_epoch is the number of epochs to adapt indices.

These approximate vector searching indices trade tolerable accuracy (recall) with larger throughput. The index builder tries to reach xxx_index_recall in xxx_index_fit_epoch. The building process will stop when xxx_index_recall or xxx_index_fit_epoch reached. Index-based searching costs more memory than brute force searching and its building process costs additional time, but they are negligible compared to the benefits.

# Enable approximate item neighbor searching using vector index.
enable_item_neighbor_index = false

# Minimal recall for approximate item neighbor searching.
item_neighbor_index_recall = 0.8

# Maximal number of fit epochs for approximate item neighbor searching vector index.
item_neighbor_index_fit_epoch = 3

# Enable approximate user neighbor searching using vector index.
enable_user_neighbor_index = false

# Minimal recall for approximate user neighbor searching.
user_neighbor_index_recall = 0.8

# Maximal number of fit epochs for approximate user neighbor searching vector index.
user_neighbor_index_fit_epoch = 3

# Enable approximate collaborative filtering recommend using vector index.
enable_collaborative_index = false

# Minimal recall for approximate collaborative filtering recommend.
collaborative_index_recall = 0.9

# Maximal number of fit epochs for approximate collaborative filtering recommend vector index.
collaborative_index_fit_epoch = 3

• Redis: Remove incompatible stale cache.

redis-cli KEYS "item_neighbors*" | xargs redis-cli DEL
redis-cli KEYS "user_neighbors*" | xargs redis-cli DEL

gorse v0.3.1

Feature

• Update latest items in real-time on the server node (#342).
• Update categorized popular items in real-time on the server node (#343).

Performance

• Optimize copier by skipping equal maps and reusing slice space (#337).
• Add data skipping indices for ClickHouse (#349).
• Limit the number feedback used in fallback item-based similarity recommendation (#350).

Fix

• Fix concurrent map read in configuration (#338 authorized by @winwill2012, #347).
• Fix worker failure when there is no feedback (#340).
• Fix inefficient implementation of hidden items (#341).

Upgrade Guide

• Configuration: A new option has been added to limit the number feedback used in fallback item-based similarity recommendation. Larger value means more accurate recommendation but spend more time.

# The number of feedback used in fallback item-based similar recommendation. The default values is 10.
num_feedback_fallback_item_based = 20

• ClickHouse: Add indices manually if using ClickHouse as data storage.

ALTER TABLE feedback ADD INDEX user_index user_id TYPE bloom_filter(0.01) GRANULARITY 1;
ALTER TABLE feedback ADD INDEX item_index item_id TYPE bloom_filter(0.01) GRANULARITY 1;

• Redis: Remove incompatible stale cache.

redis-cli KEYS "popular_items*" | xargs redis-cli DEL
redis-cli KEYS "latest_items*" | xargs redis-cli DEL

gorse v0.3.0

Features

• Support hidden items. Hidden items are used in training but not in recommendation (#178).
• Support item categories. Recommendations are generated for each category (#307).

Read 2.2 Item Management in the docment for detailed usage.

Performance

• Optimize deep copy by resuing memory (#315).

Upgrade Guide

Since new attributes IsHidden and Categories are added in v0.3.0. Using the follwing SQL to update the schema if using MySQL.

ALTER TABLE items ADD is_hidden BOOL NOT NULL DEFAULT FALSE;
ALTER TABLE items ADD categories json NOT NULL;

gorse v0.2.8

Feature

• Support modify users and items by PATCH method (8a6de0e).
• Support enable or disable click-through prediction model in configuration (2e4260c).
• Support explore latest items and popular items in personalized recommendation (2e4260c).

Fix

• Fix incompatibility of MariaDB (#294).
• Fix "Invalid Date" in dashboard (8c33b7d).
• Fix auto_insert_item = false not working (#301).
• Skip empty feedback list when inserting feedback (#300).

Upgrade Guide

• In configuration file：
  • enable_click_through_prediction enable/disable click-through prediction.
  • explore_recommend defines the proportion of latest/popular items in offline recommendation.
• In RESTful API, PATCH methods are implemented for users and items.
• If there are feedback come from nonextsted items casued by #299, remember to delete them in database. Eg., delete them in MySQL:

delete from feedback where item_id not in (select item_id from items)

gorse v0.2.7

Features

• Support offset and limit for recommendation result fetching (#287).
• Support insert feedback with future timestamps (#283).
• Support write-back-delay for recommendation result fetching (#283).
• Support prometheus metrics for master, worker, server and database (#289).
• Support write logs to files (#289).
• Support multiple read feedback types (#288).
• Support custom location for local cache (#292).

Performance

• Replace encoding/gob with custom encoder/decoder (#284).
• Replace message based model distribution with stream based protocol (#284).
• Add index for item_id in feedback table (#286).

Fix

• Fix unclosed result sets in database interfaces (#275).

Upgrade Guide

• In configuration file, read_feedback_type has been replaced by read_feedback_types, which is an array of strings.
• In recommendation API:
  • write-back has been replaced by write-back-type.
  • Use write-back-delay to set delayed timestamps of write back feedbacks.
  • Use offset to load paged recommendation results.
• In command line:
  • Use --log-path to specify the path of logging file.
  • Use --cache-path to specify the path of local cache file.

gorse v0.2.6

Performance

• Cache items in workers during offline recommendation (8bbe7a6).
• Skip cold items and users during fitting CCD recommender (dc3c66f).
• Speed up neighbor searching by bitset and timestamps (7698ce7, f6d78bc, 06f604d).
• Optimize data loading by channel (5236e8c).
• Optimize memory usage (2f34bcd, eaecd64).

Fix

• Start HTTP server before loading datasets (#256).
• Send large model over gRPC (#246).
• Handle UTF-8 characters when reading lines from a csv file (#251, authorized by @amaaazing).
• Replace unsupported new SQL syntax with compatible SQL syntax (#249 authorized by @hetao29).
• Disable authorization for dashboard (a863e90).
• Fix responses in swagger apidocs (#262, authorized by @ccfish86).

gorse v0.2.5

Features

• Support user import and export.
• Save statistics to data store instead of cache store.
• Use IDF-based similarity for users and items.
• Remove confusing click feedback type (#191).
• Support user-based recommendation (#242).
• Support accuracy/precision/recall/AUC metrics for CTR prediction (#204).
• Support multi-source recommendation.

Fix

• Handle multiple occurrence users/items when loading datasets from ClickHouse (5bf340a).

Upgrade Guide

In configuration file:

• click_feedback_types has been removed.
• neighbor_type has been replaced by item_neighbor_type and user_neighbor_type.
• enable_latest_recommend, enable_popular_recommend, enable_user_based_recommend, enable_item_based_recommend and enable_collaborative_recommend are added.
• fallback_recommend becomes a slice of strings.

gorse v0.2.4

Features

• Support PostgresSQL as data storage (#201).
• Support ClickHouse as data storage (#184).
• Support user and password in Redis DSN (#216, authorized by @lbw114007).
• Implement task monitor to track the progress of all tasks.
• Implement new task scheduler.
• Implement label based similarity (#131).

Fix

• Not distribute invalid model (#214).
• Division by zero (#221, authorized by @lbw114007).

Upgrade Guide

• In the configuration file, search_jobs and fit_jobs are replaced by n_jobs, neighbor_type is added to set the type of neighbors.

gorse v0.2.3

Features

• Support linux arm64 (#208).

Fix

• Remove historical items when injecting latest items during offline recommendation (0a9dfa6).
• Fix wrong corresponding target values of click/read feedback (13bdf1e).

gorse v0.2.2

Features

• Rank items from collaborative filternig by click-through-rate prediction to improve recommendation accuracy (#17).
• Address cold-start problem by inserting latest items to the result from collaborative filternig (#151).
• Disable click-through-rate prediction when labels are unavaliable.
• Enable RESTful server logging.

Fix

• Error 1069: Too many keys specified; max 64 keys allowed (#176).
• Panic on unknown users and items (#185).
• lastNumUsers, lastNumItems, lastNumFeedback always zero (#197).
• Documents has been updated (#180).