Commit Graph

131 Commits

Author SHA1 Message Date
Vishnu Ks 8a1794caa3 message: Store the value of first_visible_message_id in Realm table.
This eliminates a bunch of potentially buggy caching code, with no
material negative side effects.
2018-12-12 15:11:17 -08:00
Tim Abbott e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Tim Abbott 209dd5db67 actions: Add a function for changing realm subdomains.
This is initial work, which will help us establish habits of using a
well-tested approach for renaming a Zulip organization (since as part
of https://github.com/zulip/zulip-mobile/issues/3142, we'll likely
need to make this function do more).
2018-11-15 14:39:14 -08:00
Pragati Agrawal d5df0377cc settings_users: Support guest user in admin-user-table.
This supports guest user in the user-info-form-modal as well as in the
role section of the admin-user-table.

With some fixes by Tim Abbott and Shubham Dhama.
2018-10-29 12:33:35 -07:00
Steve Howell 76deb30312 preview: Hash cache keys for preview urls.
We don't want really long urls to lead to truncated
keys, or we could theoretically have two different
urls get mixed up previews.

Also, this suppresses warnings about exceeding the
250 char limit.

Finally, this gives the key a proper prefix.
2018-10-14 09:28:57 -07:00
Yago González f6219745de users: Get all API keys via wrapper method.
Now reading API keys from a user is done with the get_api_key wrapper
method, rather than directly fetching it from the user object.

Also, every place where an action should be done for each API key is now
using get_all_api_keys. This method returns for the moment a single-item
list, containing the specified user's API key.

This commit is the first step towards allowing users have multiple API
keys.
2018-08-08 16:35:17 -07:00
Tim Abbott 6f7e12ea19 docs: Add subsystem documentation for caching. 2018-07-31 17:00:45 -07:00
Anders Kaseorg 195cc78470 zerver/lib/cache.py: Avoid shelling out for mkdir.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-19 10:43:37 -07:00
Shubham Dhama 01555e8772 streams: Handle guest user ids for stream settings changes' events. 2018-06-04 11:35:37 -07:00
Vishnu Ks 372e9740ac events: Add date_joined to user_dict. 2018-05-17 07:49:35 -07:00
Steve Howell 4332fd64f7 Add submessages to message payloads. 2018-05-16 15:13:33 -07:00
Aditya Bansal a68376e2ba zerver/lib: Change use of typing.Text to str. 2018-05-12 15:22:39 -07:00
Tim Abbott 707af5ab56 cache: Remove a now-unnecessary TODO.
We solved the problem the TODO raised by using a different type
annotation syntax, and I'm not sure whether that refactor would
actually improve the code.
2018-03-16 11:32:14 -07:00
neiljp (Neil Pilgrim) 966ca7015f mypy: Finalize migration of cache.py to python3 function annotation.
- Use forward declarations of some types from models.py to avoid cycles.
- Remove cache.py from linter rule exclude list to ensure it stays that way.
2018-03-16 11:29:12 -07:00
neiljp (Neil Pilgrim) 005cb6bd03 mypy: Improve [get_]cache_with_key typing & use py3 annotation. 2018-03-16 11:29:12 -07:00
Tim Abbott 3d8e45f1cb cache: Avoid caching /help/ documentation page content.
This should make it a lot less annoying to edit these pages locally,
without regressing the test performance which motivated the cache.
2018-03-05 09:26:58 -08:00
Umair Khan 0eca2e102d cache: Add ignore_unhashable_lru_cache function.
This is a wrapper over lru_cache function. It adds following features on
top of lru_cache:

    * It will not cache result of functions with unhashable arguments.
    * It will clear cache whenever zerver.lib.cache.KEY_PREFIX changes.
2018-02-09 18:14:08 -08:00
rht 9161f8c39b zerver/lib: Remove u prefix from strings. 2018-02-05 12:12:58 -08:00
Vishnu Ks 036dc53d20 messages: Rename last_visible_message_id to first_visible_message_id. 2018-01-22 19:53:44 -08:00
Vishnu Ks e6d3f8895f messages: Create function to calculate first visible message id. 2018-01-04 08:52:40 -05:00
rht ee546a33a3 zerver/lib: Use python 3 syntax for typing.
Edited by tabbott to improve various line-wrapping decisions.
2017-11-28 17:15:14 -08:00
rht 229a8b38c0 zerver/lib: Use Python 3 syntax for typing for several files.
This adds a number of annotations that had been missed in previous
passes.
2017-11-28 17:02:24 -08:00
Tim Abbott 646ba5b9e5 bulk_get_users: Fix issues with users in multiple realms.
The previous implementation had a subtle caching bug: because it was
sharing its cache with the `get_user_profile_by_email` cache, if a
user happened to have an email in that cache, we'd return it, even
though that user didn't match `base_query`.

This causes `get_cross_realm_users` to no longer have a problematic
caching bug.
2017-11-27 14:34:45 -08:00
rht fef7d6ba09 zerver/lib: Remove u prefix from strings.
License: Apache-2.0
Signed-off-by: rht <rhtbot@protonmail.com>
2017-11-03 15:34:37 -07:00
neiljp (Neil Pilgrim) 304e411944 mypy: Remove unused FuncT TypeVar in cache.py. 2017-10-31 00:02:17 -07:00
Tim Abbott f2e3e779eb mypy: Properly annotate generic_bulk_cached_fetch.
Along with fixing some minor bugs, this requires extracting out the
default functions so that we can do type: ignores on them properly.

While we're at it, we switch to the Python 3 syntax.
2017-10-28 10:07:15 -07:00
Tim Abbott 73c27e1277 cache: Fix type aliasing of cached_objects.
Previously, it was converted from a CompressedItemT to an ItemT
without changing the variable name.
2017-10-28 10:01:44 -07:00
Tim Abbott 94c1da7025 cache: Move generic_bulk_cached_fetch typevars up a bit. 2017-10-28 10:00:43 -07:00
neiljp (Neil Pilgrim) c063ba72a2 mypy: Improve typing of cache_with_key and cache decorators.
Fixes #1348.
2017-10-28 08:57:49 -07:00
Steve Howell df93a99b50 Cache only one row per message.
Before this change, we populated two cache entries for each
message that we sent.  The entries were largely redundant,
with the only difference being whether we sent the content
as raw markdown or as the rendered HTML.

This commit makes it so we only have one cache entry per
message, and it includes both content and rendered_content.

One legacy source on confusion here is that `content`
changes meaning when you're on the front end.  Here is the
situation going forward:

    database:
        content = raw
        rendered_contented = rendered

    cache entry:
        content = raw
        rendered_contented = rendered

    payload for the frontend:
        content = raw (for apply_markdown=False)
        content = rendered (for apply_markdown=True)
2017-10-26 16:35:28 -07:00
Steve Howell 14d2d4e506 Fix bug in flush_user_profile().
Every time we updated a UserProfile object, we were calling
delete_display_recipient_cache(), which churns the cache and
does an extra database hop to find subscriptions.  This was
due to saying `updated_fields` instead of `update_fields`.

This made us prone to cache churn for fields like UserProfile.pointer
that are fairly volatile.

Now we use the helper function changed().  To prevent the
opposite problem, we use all the fields that could invalidate
the cache.
2017-10-25 11:30:56 -07:00
Steve Howell c8875693c8 Extract changed() helper in flush_user_profile().
The verbose style of `changed` is partly to appease mypy.
2017-10-25 11:29:09 -07:00
Steve Howell b94c062368 Make the realm user cache include non-active users.
This is a prepatory commit that adds non-active users to
the realm user cache.  It mostly involves name changes and
removing an `is_active` filter from the relevant DB query.

The only consumer of this cache is `get_raw_user_data`, which
now filters on `is_active` in a dictionary comprehension (but
this will get moved around a bit in a subsequent commit).
2017-10-25 11:18:30 -07:00
rht 035ed93111 zerver/lib: remove `import six`. 2017-09-27 19:10:28 -07:00
rht 2e12fe5e2e zerver/lib: Remove print_function. 2017-09-27 18:05:45 -07:00
rht f43e54d352 zerver/lib: Remove absolute_import. 2017-09-27 10:00:39 -07:00
Juliana Bacelar 928dd06cc8 linter: Add lint rule banning 'import os.path' 2017-09-22 10:32:21 -07:00
Steve Howell 8ad7133351 Cache active_user_ids() more directly.
We now have a dedicated cache for active_user_ids() that only
stores a list of user_ids.

Before this commit, active_user_ids() used a cache of UserProfile
dictionaries, so it incurred unnecessary deserialization costs for
all the user fields that it sliced away in a list comprehension.

Because the cache is skinnier here, we also need to invalidate it
less frequently.  Basically, all we care about is new users, realm
deactivations, and user deactivations.

It's hard to measure how much this will improve performance, because
the speedup for any operation here is pretty minor, but we use this
function a lot, so hopefully it will make the overall system more
healthy.
2017-09-20 10:31:33 -07:00
Steve Howell 26735eeeac Only require realm_id for get_active_user_dicts_in_realm().
This is a preparatory commit that will eventually allow us
to avoid fetching realm info that we don't need, in other
parts of the codebase.
2017-09-20 10:31:33 -07:00
Steve Howell 0966bf1a48 Simplify get_stream_cache_key().
Before this commit, we could pass in either a Realm object
or a realm_id to get_stream_cache_key().  Now we consistently
pass it a realm_id.
2017-09-20 10:31:33 -07:00
Tim Abbott b8e7369dee mypy: Remove type: ignores not needed in Python 3. 2017-08-25 11:04:20 -07:00
Tim Abbott eeabed9119 models: Add new get_user_profile_by_api_key helper.
This results in a slight performance increase.
2017-08-24 23:17:08 -07:00
Abhijeet Kaur af7e08acb0 bots: Add UI to view bot types of existing bots in "Your bots".
Tweaked by tabbott for more standard internationalization.
2017-06-15 10:08:31 -07:00
Eklavya Sharma 690b6025fb mypy: Fix return type of a function. 2017-05-24 18:43:51 -07:00
Konstantin Gukov dd76222a3f Fetch system bots using new get_system_bot function.
This eliminate a bunch of uninteresting calls to
get_user_profile_by_email.
2017-05-23 10:30:40 -07:00
Vishnu Ks bdf7c6c02f models: Add get_user function.
This is intended to replace get_user_profile_by_email.
2017-05-22 11:26:44 -07:00
Tim Abbott 0990246289 avatar: Fix memcached query loop fetching bots.
Similar to the related issue with users, the new avatar storage had
accidentally added database queries in a loop to this code path.
2017-05-09 22:33:27 -07:00
Tim Abbott 8e47dc73bd avatar: Fix loop doing database queries in register.
Due to the refactoring of the avatar URL codepath that added realm IDs
to the URLs, we ended up calling `get_user_profile_by_email` inside
`get_avatar_url`, which in turns was called in a loop over all users
in a realm.

Needless to say, this resulted in a significant performance problem.

We fix this issue by passing in the data needed to compute the avatar
URL, rather than looking it up by email address.
2017-05-09 22:33:27 -07:00
Aditya Bansal 821be4519c pep8: Add compliance with rule E261 to cache.py. 2017-05-07 23:21:50 -07:00
Umair Khan 4d543217ba cache: Take hash of KEY_PREFIX to limit key size.
Key size of Memcached records should be less than 256.
2017-05-05 18:23:40 +05:00