Commit Graph

191 Commits

Author SHA1 Message Date
rht 8990b1046d zerver: Remove inheritance from object. 2017-11-06 08:53:48 -08:00
neiljp (Neil Pilgrim) 186181c2a0 mypy: Specify type parameters in convert_term in narrow_parameter. 2017-11-04 19:47:45 -07:00
Greg Price 68b0a419ec decorator: Cut a bunch of dead imports of two view decorators.
Saw these when grepping for these two decorators; they're actually
more numerous than the surviving use sites are.  Cut out the noise.
2017-11-04 19:27:00 -07:00
Steve Howell 9689168bf6 Simplify highlight_string logic.
This change avoids some complicated byte arithmetic by
extracting some local variables and being more explicit
about using bytes vs. using strings.
2017-10-31 12:45:58 -07:00
Sampriti Panda ee61e1be2c search: Fix misaligned highlights due to unicode characters.
tsearch_extras returns search offsets in bytes but our highlight
function treated them as character offsets.  Added a check to subtract
extra bytes if the tsearch search backend is being used.

Fixes #4084.
Fixes #7021.
2017-10-31 09:06:46 -07:00
Steve Howell 8b012c6210 Extract get_personal_recipient(). 2017-10-28 17:57:39 -07:00
Steve Howell a28841e8aa Extract get_stream_recipient().
Do you call get_recipient(Recipient.STREAM, stream_id) or
get_recipient(stream_id, Recipient.STREAM)?  I could never
remember, and it was not very type safe, since both parameters
are integers.
2017-10-28 17:57:39 -07:00
Tim Abbott 5b5dffebe7 get_messages_backend: Tweak how search fields are added.
This is a no-op, but makes mypy happier, since previously it wanted to
know the type of the {} we were passing to `.update()`.
2017-10-28 09:58:51 -07:00
Steve Howell 655f37a34b Rename subject_name in send_message_backend(). 2017-10-27 10:48:11 -07:00
Steve Howell df93a99b50 Cache only one row per message.
Before this change, we populated two cache entries for each
message that we sent.  The entries were largely redundant,
with the only difference being whether we sent the content
as raw markdown or as the rendered HTML.

This commit makes it so we only have one cache entry per
message, and it includes both content and rendered_content.

One legacy source on confusion here is that `content`
changes meaning when you're on the front end.  Here is the
situation going forward:

    database:
        content = raw
        rendered_contented = rendered

    cache entry:
        content = raw
        rendered_contented = rendered

    payload for the frontend:
        content = raw (for apply_markdown=False)
        content = rendered (for apply_markdown=True)
2017-10-26 16:35:28 -07:00
derAnfaenger 1792dcbd09 tests: Call real consume method of queue processors.
This switches to more real tests for a first batch of
queue_json_publish() calls that don't cause trouble when
used with call_consume_tests=True.
2017-10-26 14:58:03 -07:00
Steve Howell a1d887c950 Use check_bool validator in get_messages_backend().
This is the more modern way in our codebase to validate
bool fields.
2017-10-20 15:51:57 -07:00
Steve Howell 0e106a2488 Add client_gratavar support to GET /messages.
Clients fetching messages can now specify that they are able
to compute their avatar, and if they set client_gratavar to
True in the request (w/our normal encoding scheme), then the
backend will not compute it, and the payload will be smaller.

The fix starts with get_messages_backend.  The flag gets
passed down through these functions:

    * MessageDict.post_process_dicts.
    * MessageDict.set_sender_avatar.

We also fix up the callers for post_process_dicts to explicitly
pass in the client_gravatar path, but for now they all just hard
code the value to False.
2017-10-20 15:49:21 -07:00
Steve Howell d909355dc2 refactor: Move methods from models.py -> lib/messages.py.
Message.get_raw_db_rows is moved to MessageDict, since its
implementation details are highly coupled to other methods
in MessageDict.

And then sew_messages_and_reactions comes along for the
ride.

We eventually want to move Reaction.get_raw_db_rows to there
as well.
2017-10-16 23:37:10 -07:00
Tim Abbott 339e206c90 highlight_html_differences: Improve logging output.
Now at least it will give the message ID, and thus be possible to
debug.
2017-10-11 23:38:29 -07:00
Steve Howell 3910448b1d Extract MessageDict.post_process_dicts().
Introduce MessageDict.post_process_dicts() will allow us
the ability to do the following:

    * use less memory in the cache for repeated data
    * prevent cache invalidation
    * format data according to different client needs

The first use of this function is pretty inconsequential, but
it sets us up for more consequential changes.

In this commit we defer the MessageDict.hydrate_recipient_info
step until after we pull data out of the cache.  This impacts
cache size as follows:

    * streams - negligibly bigger
    * PMs/huddles - slimmer due to not needing to repeat
                    sender data like email/full_name

Again, the main point of this change is to start setting up
the infrastructure to do post-processing.
2017-10-11 11:37:16 -07:00
Steve Howell aae0b2a826 Notify offline users about edited stream messages.
We now do push notifications and missed message emails
for offline users who are subscribed to the stream for
a message that has been edited, but we short circuit
the offline-notification logic for any user who presumably
would have already received a notification on the original
message.

This effectively boils down to sending notifications to newly
mentioned users.  The motivating use case here is that you
forget to mention somebody in a message, and then you edit
the message to mention the person.  If they are offline, they
will now get pushed notifications and missed message emails,
with some minor caveats.

We try to mostly use the same techniques here as the
send-message code path, and we share common code with the
send-message path once we get to the Tornado layer and call
maybe_enqueue_notifications.

The major places where we differ are in a function called
maybe_enqueue_notifications_for_message_update, and the top
of that function short circuits a bunch of cases where we
can mostly assume that the original message had an offline
notification.

We can expect a couple changes in the future:

    * Requirements may change here, and it might make sense
      to send offline notifications on the update side even
      in circumstances where the original message had a
      notification.

    * We may track more notifications in a DB model, which
      may simplify our short-circuit logic.

In the view/action layer, we already had two separate codepaths
for send-message and update-message, but this mostly echoes
what the send-message path does in terms of collecting data
about recipients.
2017-10-03 15:57:06 -07:00
rht 106375bd68 zerver/views: remove `import six`. 2017-09-27 17:03:03 -07:00
Steve Howell 646abb57b7 refactor: Extract get_user_info_for_message_updates.
We'll want to expand this to get users that were mentioned in
the prior message, but this commit is just a refactoring.
2017-09-27 16:01:50 -07:00
rht 15ca13c8de zerver/views: Remove absolute_import. 2017-09-27 10:00:39 -07:00
Steve Howell f13cb94d84 Have exclude_topic_mutes() accept a stream id.
We want to convert stream names to stream ids as close
to the "edges" of our system as possible, so we let our
caller do the work of finding the stream id for a stream
narrow.
2017-09-20 10:31:33 -07:00
Steve Howell ba397b5109 Use user_ids, not full objects, in render path.
There is no reason for either render_incoming_message() or
render_markdown() to require full UserProfile objects just to
triage alert words.

By only asking for user_ids, we save extra queries in two
callpaths and we make it easier to start using user_ids in
do_send_messages().
2017-09-12 04:22:55 -07:00
Steve Howell 8c4a5a9f7a Extract exclude_topic_mutes.
This is mostly a pure code move, but I cleaned up the code
slightly to use early-return.
2017-08-29 16:53:38 -04:00
Steve Howell 87c4961597 Add zerver/lib/topic_mutes.py
This is mostly pure code extraction.

It also removes some dead code in update_muted_topic, where
were updating muted_topics spuriously before calling
do_update_muted_topic.
2017-08-24 14:20:35 -07:00
Steve Howell 2b4faaa847 Support non-search queries in /json/messages/matches_narrow.
For filters like has:link, where the web app doesn't necessarily
want to guess whether incoming messages meet the criteria of the
filter, the server is asked to query rows that match the query.

Usually these queries are search queries, which have fields for
content_matches and subject_matches.  Our logic was handling those
correctly.

Non-search queries were throwing an exception related to tuple
unpacking.  Now we recognize when those fields are absent and
do the proper thing.

There are probably situations where the web app should stop hitting
this endpoint and just use its own filters.  We are making the most
defensive fix first.

Fixes #6118
2017-08-23 01:07:57 -07:00
Steve Howell 47bcedbc1b Fix server searches for is:mentioned and is:alerted.
Before this change, server searches for both
`is:mentioned` and `is:alerted` would return all messages
where the user is specifically mentioned (but not
at-all mentions).

Now we follow the JS semantics:

    is:mentioned -- all mentions, including wildcards
    is:alerted  -- has an alert word

Here is one relevant JS snippet:

        } else if (operand === 'mentioned') {
            return message.mentioned;
        } else if (operand === 'alerted') {
            return message.alerted;

And here you see that `mentioned` is OR'ed over both mention flags:

    message.mentioned = convert_flag('mentioned') || convert_flag('wildcard_mentioned');

The `alerted` flag on the JS side is a simple mapping:

    message.alerted = convert_flag('has_alert_word');

Fixes #5020
2017-08-16 11:29:12 -07:00
Tim Abbott ffc1ceeaa4 lint: Ban use of get_stream in the rest of messages.py.
This will help avoid potential future security bugs.
2017-08-15 10:40:02 -07:00
Tim Abbott 842bf77efb unread: Convert mark_topic_as_read to use stream IDs.
This is safer in its handling of potentially renamed streams.
2017-08-15 10:40:02 -07:00
Tim Abbott f3a41ac796 unread: Convert mark_stream_as_read to use stream IDs.
The logic is simpler and more robust.
2017-08-15 10:40:02 -07:00
Steve Howell 89f9017686 Create new endpoints for marking streams/topics as read.
The new endpoints are:
    /json/mark_stream_as_read: takes stream name
    /json/mark_topic_as_read: takes stream name, topic name

The /json/flags endpoint no longer allows streams or topics
to be passed in as parameters.
2017-08-15 10:17:29 -07:00
Steve Howell 60cc8fd58a Extract do_mark_stream_messages_as_read.
This function optimizes marking streams and topics as read,
by using UserMessage.where_unread(), which uses a partial
index on the "read" flag.

This also simplifies the code path for ordinary message
flag updates.

In order to keep 100% line coverage, I simplified the
logging in update_message_flags, so now all requests
will show the "actually" format.

This is an interim step toward creating dedicated endpoints
for marking streams/topics as reads, so we do error checking
with asserts for flag/operation, so we don't introduce a
temporary translation string.
2017-08-15 10:09:10 -07:00
Steve Howell ffe4f09c88 Extract update_stream_topic_message_flags.
This is mostly a pure code extraction, except that we now
disregard the `messages` option for stream/topic updates,
since the web app always passes in an empty list (and this
commit is really just an incremental step toward creating
new endpoints.)
2017-08-15 10:09:10 -07:00
Tim Abbott a6a3915dd7 messages: Move is_public_stream out of views code.
The main purpose of this change is to minimize access to get_stream in
views code.
2017-08-15 10:06:26 -07:00
Steve Howell 63f7b9a579 Remove "all" option for flag-updating endpoint.
The "all" option for 'message/flags' was dangerous, as it could
apply to any of our flags.  The only flag it made sense for, the
"read" flag, now has a dedicated endpoint.
2017-08-04 14:10:46 -07:00
Steve Howell 541156792e Add /mark_all_as_read endpoint.
This change simplifies how we mark all messages as read.  It also
speeds up the backend by taking advantage of our partial index
for unread messages.  We also use a new statsd indicator.
2017-08-04 14:10:46 -07:00
neiljp (Neil Pilgrim) b19e62b7ce mypy: Initialize search_term to {} rather than None & avoid Optional. 2017-08-03 11:09:50 -07:00
Tim Abbott a9fa1a5527 api: Migrate /json/messages_in_narrow off legacy API.
This completes the major endpoint migrations to eliminate legacy API
endpoints from Zulip.

There's a few other things that will happen naturally, so I believe
this fixes #611.
2017-07-31 13:08:06 -07:00
Greg Price 9faa44af60 JsonableError: Optionally carry error codes and structured data.
This provides the main infrastructure for fixing #5598.  From here,
it's a matter of on the one hand upgrading exception handlers -- the
many except-blocks in the codebase that look for JsonableError -- to
look beyond the string `msg` and pass on the machine-readable full
error information to their various downstream recipients, and on the
other hand adjusting places where we raise errors to take advantage
of this mechanism to give the errors structured details.

In an ideal future, I think all exception handlers that look (or
should look) for a JsonableError would use its contents in structured
form, never mentioning `msg`; but the majority of error sites might
continue to just instantiate JsonableError with a string message.  The
latter is the simplest thing to do, and probably most error types will
never have code looking for them specifically.

Because the new API refactors the `to_json_error_msg` method which was
designed for subclasses to override, update the 4 subclasses that did
so to take full advantage of the new API instead.
2017-07-24 16:41:22 -07:00
Greg Price 6dfb46dc08 JsonableError: Rename `status_code` and rely more on its default.
With #5598 there will soon be an application-level error code
optionally associated with a `JsonableError`, so rename this
field to make clear that it specifically refers to an
HTTP status code.

Also take this opportunity to eliminate most of the places
that refer to it, which only do so to repeat the default value.
2017-07-24 16:41:22 -07:00
Vishnu Ks 0d6e92d946 messages: Use get_user_including_cross_realm in create_mirrored_message_users. 2017-07-18 17:14:05 -07:00
Vishnu Ks 9ebf5090d2 messages: Use get_user_including_cross_realm in by_group_pm_with function. 2017-07-18 17:14:05 -07:00
Vishnu Ks fc8b6f69bd messages: Use get_user_including_cross_realm in by_pm_with function. 2017-07-18 17:14:05 -07:00
Vishnu Ks 7b78224c42 messages: Use get_user_including_cross_realm in by_sender function. 2017-07-18 17:14:05 -07:00
Vishnu Ks d99032e896 views: Add user_realm attribute to class NarrowBuilder. 2017-07-18 17:14:05 -07:00
Vishnu Ks 479945a1d5 views: Remove unused imports from messages.py. 2017-07-18 17:14:05 -07:00
Durga Akhil Mundroy 146dfa6f0b org-permissions: Add allow_edit_history organiztion setting.
This new setting controls whether or not users are allowed to see the
edit history in a Zulip organization.  It controls access through 2
key mechanisms:

* For long-ago edited messages, get_messages removes the edit history
  content from messages it sends to clients.

* For newly edited messages, clients are responsible for checking the
  setting and not saving the edit history data.  Since the webapp was
  the only client displaying it before this change, this just required
  some changes in message_events.js.

Significantly modified by tabbott to fix some logic bugs and add a
test.
2017-07-16 10:10:06 -07:00
Steve Howell 475eb21a5e Revert commits related to client_message_id.
I pushed a bunch of commits that attempted to introduce
the concept of `client_message_id` into our server, as
part of cleaning up our codepaths related to messages you
sent (both for the locally echoed case and for the host
case).

When we deployed this, we had some strange failures involving
double-echoed messages and issues advancing the pointer that appeared
related to #5779.  We didn't get to the bottom of exactly why the PR
caused havoc, but I decided there was a cleaner approach, anyway.
2017-07-14 12:13:35 -07:00
Steve Howell 8fbb55df85 Introduce client_message_id on the server.
We are deprecating local_id/local_message_id on the Python server.
Instead of the server knowing about the client's implementation of
local id, with the message id = 9999.01 scheme, we just send the
server an opaque id to send back to us.

This commit changes the name from local_id -> client_message_id,
but it doesn't change the actual values passed yet.

The goal for client_key in future commits will be to:
    * Have it for all messages, not just locally rendered messages
    * Not have it overlap with server-side message ids.

The history behind local_id having numbers like 9999.01 is that
they are actually interim message ids and the numerical value is
used for rendering the message list when we do client-side rendering.
2017-07-13 23:42:27 -04:00
Greg Price 7b526b3946 narrow: Add comments explaining how we limit to legitimately-visible results.
Two wrinkles here:

* It's actually a little subtle why `ok_to_include_history` is
  correct; in particular, it's not true that a term `stream:foo` will
  always limit the query to the stream `foo`.  For this, add an
  explanatory comment backed up by an assert.

* The TODO comment in `messages_in_narrow_backend` about assuming this
  is a search, I'm pretty sure doesn't matter; it seems to only be
  saying that we return the set of fields we would for a search.
  They're harmless to send, and in any case it doesn't appear to be
  true anymore that the client only calls this for a search: the
  `can_apply_locally` function also causes narrows with `has:` to go
  to the server.  So just take that comment out.
2017-06-30 11:47:40 -04:00
Greg Price 7785a1046b narrow: Describe the key security invariant of NarrowBuilder.
(After writing the term "invariant" a few times in these comments and
now this commit message, my inner mathematician reminds me that this
property is properly termed a "monovariant" -- something does change,
but it changes only in one direction.  Pretty sure saying "invariant"
communicates better here, though.)
2017-06-30 11:47:40 -04:00