mysql - How often should I upload documents to CloudSearch (Solr)? -

-

here use case:

i use mysql primary data store , cloudsearch searching. database contains tables: threads, comments, upvotes, users.

i created expression sort search results based on "trending" using upvotes , created_at date (hacker news hot algorithm). expression called "trend", , used in cloudsearch query this: /search?q=superman&sort=trend+desc

(upotes-1)/pow(floor((_time-created_at)/3600000)+2, 1.8)

right now, when user upvotes thread or comment, stored in mysql database. question how should keep upvotes in sync cloudsearch?

the 2 options see:

  1. immediately insert (replace) upvote in mysql, update score on cloudsearch. involves sending single document upload on every upvote, ensures real-time accuracy.
  2. immediately insert (replace) upvote in mysql, keep upvote in cache somewhere (redis?). once every hour, upload upvotes cloudsearch.

what best way handle situation?

it depends on lot of things

  1. your solr setup, how many servers, how memory, cpu, storage, how many documents, index size per shard/server etc.

  2. how many "estimated" upvotes expecting? if take option 1, easier decide if can how estimate number.

    since using solrcloud, has nrt feature makes sure documents available searches. again depends on current document corpus, , how many updates per second or minute expecting.

if know number of upvotes (updates solr) , if have sufficiently servers, go option 1, since reduce overhead of manitaining database, , logic updating upvotes every hour solr.

you can setup couple of test servers, , stress testing find out exact number of updates under solr's performance reduce.

i know not give exact yes or no, said, depends on particular use case.


Comments

Post a Comment

Popular posts from this blog

php - Zend Framework / Skeleton-Application / Composer install issue -

-
i trying create zend framework application using skeleton in netbeans. when run composer install, following error message: "c:\php\php.exe" "c:\composer\composer.phar" "--ansi" "--no-interaction" "update" "--dev" using deprecated option "dev". dev packages installed default now. loading composer repositories package information updating dependencies (including require-dev) requirements not resolved installable set of packages. problem 1 - package requires php >=5.5 php version (5.4.42) not satisfy requirement. problem 2 - installation request zendframework/zendframework 2.5.1 -> satisfiable zendframework/zendframework[2.5.1]. - zendframework/zendframework 2.5.1 requires php >=5.5 -> php version (5.4.42) not satisfy requirement. done. apparently, zend framework only needs php 5.3+ contradicts above. need use php 5.4. my composer.json following: { "name": "
Read more

c# - Better 64-bit byte array hash -

-
i need hash algorithm produces 64-bit hash code ( long ) fewer collisions string.gethashcode() , fast (no expensive calls cryptographic functions). here's implementation of fnv still shows 3% of collisions after testing 2 million random strings. need number way lower. void main() { const string chars = "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz!#@$%^&*()_+}{\":?><,./;'[]0123456789\\"; const int n = 2000000; var random = new random(); var hashes = new hashset<long>(); int collisions = 0; for(int = 0; < n; i++) { var len = random.next(chars.length); var str = new char[len]; (int j = 0; j < len; j++) { str[j] = chars[random.next(chars.length)]; } var s = new string(str); if(!hashes.add(get64bithash( s ))) collisions++; } console.writeline("collision percentage after " + n + " random strings: " + ((doubl
Read more

python - PyCharm Type error Message -

-
i'm newbie trying out ml/dbn in python (3.4) under pycharm (comm.ed 4.5). following definition def make_thetas(xmin,xmax,n): xs = np.linespace(xmin,xmax,n) widths = (xs[1:] - xs[:-1])/2.0 thetas = xs[:-1] + widths return thetas throws error "class 'tuple' not define ' sub ', '-' operator cannot used on instance" on - operator on third line (widths = ....) ideas on how code running under pycharm - works alright in interactive python window. thx. to all, after g'g lot have found workaround seems work within pycharm. workaround because (even python newbie) expected python not need explicit typecasting, not when using datatypes imported lib such numpy. def make_thetas(xmin,xmax,n): xs = np.array(np.linspace(xmin,xmax,n)) widths = (xs[1:] - xs[:-1])/2.0 thetas = xs[:-1]+ widths return thetas using docstrings type-hinting such below did not work def make_thetas(xmin,xmax,n): "&q
Read more