Commit Graph

19962 Commits

Author SHA1 Message Date
Pweaver (Paul Weaver) 48815204e4 Changes node_modules to default to the same args for generate_sha1sum.
Since we can use both perfer_offline=True and False in a since build
prefer_offline shouldn't be used as a cache key or it will confuse the
cleanup script. Since yarn install (if successful) should be idempotent.
This will probably be ok.
2017-08-05 12:29:16 -07:00
Pweaver (Paul Weaver) 1afaa67c7c deps: Change npm to yarn for reliablity, security, and speed. 2017-08-05 12:29:06 -07:00
Pweaver (Paul Weaver) f444c68b62 Indent install-node code block in preperation for yarn migration. 2017-08-05 12:18:47 -07:00
neiljp (Neil Pilgrim) 3ca34bebdc mypy: Use Callable as parameter for simulated_queue_client.
Previously the type was annotated as 'type'.
2017-08-05 12:08:59 -07:00
neiljp (Neil Pilgrim) 6717f361ec mypy: Amend tornado_redirected_to_list calls to use Mapping. 2017-08-05 12:08:59 -07:00
neiljp (Neil Pilgrim) 353f6fdb88 mypy: Hoist RuleList type alias out of function. 2017-08-05 12:08:59 -07:00
Tim Abbott 27b79e7189 populate_db: Fix duplicate test data when run multithreaded.
The previous implementation resulted in each of the 10 threads using
the same random seed and thus same messages.

Fixes #6028.
2017-08-05 11:22:11 -07:00
Rishi Gupta 4bf8ac2498 generate_test_data: Replace source with non-Gutenberg text.
It's hard to find literature with the community tone we're going for, that
is consistent with the Zulip code of conduct, etc.

This commit removes the special tooling for Gutenberg plays, and changes the
text to be some mixture of scigen, Communications From Elsewhere,
chat.zulip.org, and various books from the public domain.
2017-08-05 11:04:35 -07:00
Rishi Gupta 4350b52740 generate_test_data: Fix remove_line_breaks algorithm. 2017-08-05 11:04:35 -07:00
Tim Abbott dd49bec93c Fix changing email addresses back after email change.
We apparently were not correctly clearing the user_profile's email
address from caches when changing email addresses, which meant that
trying to look up the old email in the user_profile caches would still
work.

Fixes #6035.
2017-08-05 10:49:44 -07:00
Rishi Gupta 251bd3e577 registration.py: Remove first_in_realm from accounts_register.
The parent commit implies first_in_realm == realm_creation.
2017-08-05 10:17:14 -07:00
Rishi Gupta ddbf8b9c3c management: Remove create_realm.py.
This no longer does the correct thing (in terms of onboarding emails,
default streams, etc), and is tempting for new server admins to use.

Once we remove it we'll also have the invariant that we can't have a realm
without a user, which will simplify accounts_register a bit.
2017-08-05 10:17:14 -07:00
Vishnu Ks e7f5da83ac management: Create list realms command.
Fixes #5917.
2017-08-04 14:25:03 -07:00
Steve Howell 63f7b9a579 Remove "all" option for flag-updating endpoint.
The "all" option for 'message/flags' was dangerous, as it could
apply to any of our flags.  The only flag it made sense for, the
"read" flag, now has a dedicated endpoint.
2017-08-04 14:10:46 -07:00
Steve Howell bc6543a236 Use do_mark_all_as_read() in bankrupt_users command. 2017-08-04 14:10:46 -07:00
Steve Howell 541156792e Add /mark_all_as_read endpoint.
This change simplifies how we mark all messages as read.  It also
speeds up the backend by taking advantage of our partial index
for unread messages.  We also use a new statsd indicator.
2017-08-04 14:10:46 -07:00
Steve Howell eafd5a731a Remove dead code: messages.flags.send_force_collapse. 2017-08-04 13:31:26 -07:00
Steve Howell cc94d6179c Remove dead code: messages.flags.send_force_expand. 2017-08-04 13:31:26 -07:00
Steve Howell 437cabe34c Just use message.unread when rendering.
By the time we render messages, we will have set message.unread,
so we don't need to calculate it from flags.

We add a line to the local-echo path to make this explicit
in that code.
2017-08-04 13:31:26 -07:00
Steve Howell f81f9a26e8 Call message.set_message_booleans() in update_messages().
When we learn about updated message, a bunch of flag/boolean
fields concern us:

    starred
    mentioned
    alerted
    is_me_message

We now set booleans consistently with how we set new incoming
messages.
2017-08-04 13:31:26 -07:00
Steve Howell 187ec3a922 Set message.unread in set_message_booleans().
The sooner we make message.unread consistent with message.flags,
the better.
2017-08-04 13:31:26 -07:00
Steve Howell abe7f4cc40 Extract message_store.set_message_booleans().
We'll want to reuse this for message updates, and we'll eventually
want to bypass this for local echo.
2017-08-04 13:31:26 -07:00
Steve Howell ff54d52589 Extract message.set_read_flag().
This code adds 'read' to message.flags and sets message.unread
to false.

It's not clear that the boolean message.unread is used in any
meaningful way, but we set it to false to avoid confusion.  The
bankruptcy code was not doing this before.

Another quirk that existed before was that you could get two
'read' flags in a message when you declared bankruptcy.  It's
also plausible that this could happen if you marked a message
as read via two different ways.  It probably did not cause
user-facing bugs, but it would be confusing for troubleshooting.

Fixes #5032.
2017-08-04 13:31:26 -07:00
Steve Howell d5681556d5 Call unread.process_loaded_messages earlier when inserting.
We want to maintain the invariant that unread.js always knows
about unread messages as soon as they are loaded.
2017-08-04 13:31:26 -07:00
Steve Howell a07bd70449 Inline message_util.do_unread_count_updates(). 2017-08-04 13:31:26 -07:00
Steve Howell 7cae3fa916 Use bucketers in unread_topic_counter. 2017-08-04 13:31:26 -07:00
Steve Howell 66c2ca9a2a unread: Add make_bucketer().
This class encapsulates reverse lookups to make deletions
O(1).  It is also a bit smarter about not creating
throwaway objects, by avoiding setdefault.
2017-08-04 13:31:26 -07:00
Steve Howell 42f3b3ec00 unread: Add make_id_set().
This is a better abstraction for a set of ids, plus all the
dictionaries are kind of confusing.
2017-08-04 13:31:26 -07:00
Steve Howell 241f095213 Extract unread_ops.process_read_messages_event().
The new method borrows some code from the event loop
and unread_ops.mark_messages_as_read, and it is now
flexible about message_ids being marked as unread
even when there is no corresponding message in the
message store.  For that scenario we still want to
update our data structures, which wasn't happening
before this change.  (Generally, this was a non-issue
up until now, but it will become a bigger issue when
we start loading unread message ids from the server.)
2017-08-04 13:31:26 -07:00
Steve Howell 00a3f6b5f2 Extract process_newly_read_message(). 2017-08-04 13:31:26 -07:00
Steve Howell 8125667f90 Add unread.id_flagged_as_unread().
This function allows us to see whether unread.js thinks a message
id is unread (as opposed to looking at the message itself).  This
method is useful when we get notifications from the server that a
message has been read.  In the future, we may not actually have
a local copy of an unread message, but we'll still know that it is
unread based on page_params.  We'll want to update the data in that
case.

Going forward, we'll want to deprecate message.flags for most use
cases and just use the unread.js data structures to track unread
messages.
2017-08-04 13:31:26 -07:00
Steve Howell c5e4ce5278 Optimize show_unreads --fix approach.
This now breaks the process of cleaning up unread counts for
non-active streams into a three step process.

This allows us to use our unread message flags index, at least
in testing on dev.  Here is the relevant excerpt from explain
analyze:

    Bitmap Index Scan on zerver_usermessage_unread_message_id
2017-08-04 11:04:43 -07:00
Pweaver (Paul Weaver) 21cad9cc24 webpack: Convert webpack config to typescript.
This helps catch bugs in the configuration.
2017-08-04 11:01:37 -07:00
Pweaver (Paul Weaver) d0b8e3f0c8 webpack: Merge webpack configuration into one file.
This fixes the confusing duplication between our various webpack config files.
2017-08-04 11:01:37 -07:00
Pweaver (Paul Weaver) 42a2e5dfa3 webpack: Turn webpack.config.js into a function.
This is a prerequisite for refactoring this.
2017-08-04 11:01:37 -07:00
Greg Price cc30226af0 deliver_email: When disabled, sleep forever rather than exit.
This makes supervisor see the service as cheerfully running
and let it alone, rather than constantly retry starting it.
Because the crash/restart loop means repeatedly spending a
couple of seconds loading Django and the app, separated by
brief periods while supervisor notices the crash and acts
on it, it was actually consuming about 30-50% CPU on the
zulipchat.com staging server.
2017-08-03 16:28:19 -07:00
Steve Howell e994ff36b5 Add option to fix unread messages for inactive streams. 2017-08-03 13:17:04 -07:00
Steve Howell dad1c9326e Add show_unreads management command.
This should help us troubleshoot anomalies with old unread
messages.
2017-08-03 13:17:04 -07:00
Greg Price 2cc0689bcf api migration: Update the prod "Secure, maintain, and upgrade" doc. 2017-08-03 12:23:29 -07:00
Greg Price 8f73129d6a api migration: Update docs/directory-structure. 2017-08-03 12:23:29 -07:00
Steve Howell 5a10137ad9 Extract message_list.first_unread_message_id().
This de-duplicates code that solved the same problems using
two different phrasings.
2017-08-03 11:32:54 -07:00
Steve Howell c32b9a5d19 Remove dead code related to unread_in_current_view. 2017-08-03 11:32:54 -07:00
Steve Howell 325d7f0f57 unread: Simplify code to mark messages as read.
We now call the function mark_as_read(), and it only requires
passing in a message_id.
2017-08-03 11:32:54 -07:00
Steve Howell eac155d3b8 unread: Use reverse_lookup in unread_topic_counter. 2017-08-03 11:32:54 -07:00
Steve Howell 1927a6ca45 Simplify unread_topic_count.update().
The prior implementation was needlessly complex.  Both del() and
add() are cheap and idempotent.

With this change we no longer bother to delete a topic from a
dictionary when its last message is mark as read, since it doesn't
really help performance.  We add a line to the tests to maintain
100% line coverage.
2017-08-03 11:32:54 -07:00
Steve Howell fad4ed4e78 unread: Use reverse_lookup in unread_pm_counter.
This allows us to use simply a message_id to mark a PM as read.
2017-08-03 11:32:54 -07:00
neiljp (Neil Pilgrim) b19e62b7ce mypy: Initialize search_term to {} rather than None & avoid Optional. 2017-08-03 11:09:50 -07:00
neiljp (Neil Pilgrim) f96f71e439 mypy: assert Popen.stdout based on API, in tools/lint. 2017-08-03 11:09:50 -07:00
neiljp (Neil Pilgrim) e772e89fe2 mypy: Add None return path for RateLimitMiddleware.process_exception(). 2017-08-03 11:03:14 -07:00
neiljp (Neil Pilgrim) 0e68c44ca4 mypy: Allow get_display_recipient_by_id type_id to be Optional[int].
zerver/message.py used it in this way previously when the type was not
a stream, so the type has been set to match usage and implementation.
Also added docstring to clarify this for the specific function.
2017-08-03 11:03:14 -07:00