Commit Graph

28724 Commits

Author SHA1 Message Date
Steve Howell 272b954790 hipchat import: Add option to mask content.
Masking content can be useful for testing
out conversions where you're dealing
with data from customers and want to avoid
inadvertently reading their content (while
still having semi-realistic messages).
2018-10-25 08:31:01 -05:00
Steve Howell 6e8ae2e3fd hipchat import: Support private stream subscribers.
We now create private stream subscriptions that are
based off of `members` and `owner` from room data
in `rooms.json`.
2018-10-25 08:31:01 -05:00
Steve Howell 25f532ca2f refactor: Break up build_subscriptions.
Having two smaller functions should make it
easier to customize the behavior for each specific
use case.  The only reason they were ever coupled
was to keep ids in sequence, but the recent NEXT_ID
changes make that a non-issue now.
2018-10-25 08:31:01 -05:00
Steve Howell 2ed9fbd25b conversions: Use NEXT_ID for recipient and subscription ids.
The NEXT_ID scheme seems pretty robust, so I'm fixing a
few easy places.
2018-10-25 08:31:01 -05:00
Steve Howell 50f76e58ce conversions: Make NEXT_ID a true singleton.
We now instantiate NEXT_ID in sequencer.py, which avoids
having multiple modules make multiple copies of a sequencer
and possibly causing id collisions.
2018-10-25 08:31:01 -05:00
neiljp (Neil Pilgrim) a2767df51d mypy: Enforce strict-optional checking of decorator.py. 2018-10-25 07:24:47 -05:00
neiljp (Neil Pilgrim) 06c2748a98 mypy: Remove various mypy.ini exclusions due to REQ/None.
Current mypy supports returning None without error.
See python/mypy#4214.
2018-10-25 07:24:47 -05:00
Vishnu Ks 0cf4a75fb5 coverage: Enforce full test coverage for corporate/views.py. 2018-10-24 19:39:23 -07:00
Vishnu Ks 052640dfd5 billing: Add test for error during source replacement. 2018-10-24 19:39:23 -07:00
Vishnu Ks 20de90066b billing: Add test for exception during initial upgrade. 2018-10-24 18:54:53 -07:00
Vishnu Ks a7f4ea4b2c billing: Add test for billing disabled settings. 2018-10-24 18:54:18 -07:00
Tim Abbott 462b9c80c0 bots: Add basic documentation for duplicate bot names feature. 2018-10-24 17:01:34 -07:00
Steve Howell 551fc7f165 bots: Prevent bots from having duplicate full names.
Bots are not allowed to use the same name as
other users in the realm (either bot or human).

This is kind of a big commit, but I wanted to
combine the post/patch (aka add/edit) checks
into one commit, since it's a change in policy
that affects both codepaths.

A lot of the noise is in tests.  We had good
coverage on the previous code, including some places
like event testing where we were expediently
not bothering to use different names for
different bots in some longer tests.  And then
of course I test some new scenarios that are relevant
with the new policy.

There are two new functions:

    check_bot_name_available:
        very simple Django query

    check_change_bot_full_name:
        this diverges from the 3-line
        check_change_full_name, where the latter
        is still used for the "humans" use case

And then we just call those in appropriate places.

Note that there is still a loophole here
where you can get two bots with the same
name if you reactivate a bot named Fred
that was inactive when the second bot named
Fred was created.  Also, we don't attempt
to fix historical data.  So this commit
shouldn't be considered any kind of lockdown,
it's just meant to help people from
inadvertently creating two bots of the same
name where they don't intend to.  For more
context, we are continuing to allow two
human users in the same realm to have the
same full name, and our code should generally
be tolerant of that possibility.  (A good
example is our new mention syntax, which disambiguates
same-named people using ids.)

It's also worth noting that our web app client
doesn't try to scrub full_name from its payload in
situations where the user has actually only modified other
fields in the "Edit bot" UI.  Starting here
we just handle this on the server, since it's
easy to fix there, and even if we fixed it in the web
app, there's no guarantee that other clients won't be
just as brute force.  It wasn't exactly broken before,
but we'd needlessly write rows to audit tables.

Fixes #10509
2018-10-24 16:59:57 -07:00
Steve Howell 3a267b5cc6 Have ESC zoom out topics if you're zoomed in.
This is the natural behavior that most users will
probably expect.  If you need to go to All Messages when
topics are zoomed in, you can just hit ESC twice.
2018-10-24 16:54:35 -07:00
Steve Howell 06b1aece31 Simplify narrow/search interactions.
Before this change, if you hit ESC, then hotkey
code would call search.clear_search, which would
call narrow.deactivate(), which would then use
`$('#search_query')` to clear a value, but then
let search.clear_search blur the input and
disable the exit button.  It was all confusing.

Things are a bit more organized now.

Now the code works like this:

    hotkey.process_escape_key
        Just call narrow.deactivate.

    $('#search_exit').on('click', ...):
        Just call narrow.deactivate.

    narrow.deactivate:
        Just call search.clear_search_form

    search.clear_search_form:
        Just do simple jquery stuff.  Don't
        change the entire user's narrow, not
        even indirectly!

There's still a two-way interaction between
the narrow.js module and the search.js module,
but in each direction it's a one-liner.

The guiding principle here is that we only
want one top-level API, which is narrow.deactivate,
and that does the whole "kitchen sink" of
clearing searches, closing popovers, switching
in views, etc.  And then all the functions it
calls out to tend to have much smaller jobs to
do.

This commit can mostly be considered a refactoring, but the
order of operations changes slightly.  Basically, as
soon as you hit ESC or click on the search "X", we
clear the search widget.  Most users won't notice
any difference, because we don't have to hit the
server to populate the home view.  And it's arguably
an improvement to give more immediate feedback.
2018-10-24 16:54:35 -07:00
Steve Howell 08f0690333 left sidebar: Fix scrolling bugs with zoomed topics.
If you zoom into "more topics" for a stream that has
a LOT of topics, and then scroll down to the bottom,
and then zoom out by selecting "All messages" or
similar upper-left-sidebar options, we now try to scroll
the more recently active stream back into place after we scroll
out.

Before this change, it was possible for your lower left
sidebar to appear empty, as it would keep the
scroll offset from "more topics".
2018-10-24 16:54:35 -07:00
Steve Howell 4bfb49c3a4 Streamline closing topic lists.
If our topic list isn't zoomed in, avoid calling
stream_list.zoom_out_topics().

This commit also introduces `zoomed_in` to track
our topic zooming state.
2018-10-24 16:54:35 -07:00
Steve Howell 047a81dd2c refactor: Move clear_topics into topic_zoom.js.
This sets up some subsequent changes.
2018-10-24 16:54:35 -07:00
Steve Howell acb7149386 Extract topic_zoom.js.
This small modules nicely breaks down the
responsibilities of topic_list and stream_list
when it comes to zooming in and out of topics
(also known as hitting "more topics" or "All
Streams).

Before this, neither module was clearly in
charge, and there were kind of complicated
callback mechanisms.  The stream_list code
was asking topic_list to create click handlers
that called back into stream_list.

Now we just topic_zoom set up its own click
handlers and delegate out to the other two
modules.
2018-10-24 16:54:35 -07:00
Steve Howell fe6df1c222 hipchat import: Fix bug w/rogue UserMessage records.
This bug was introduced very recently and is an
aliasing bug.  It caused extra UserMessage rows to
be created as we inadvertently updated the underlying
subscriber_map sets for multiple messages.

This probably mostly affected PMs.

It's doubtful the bug ever got out into the field.
2018-10-24 18:44:18 -05:00
Tim Abbott 9b0265452f prod_settings_template: Document email mirror needs MX records.
The docs are still pretty much written for expert sysadmins, but this
should help a lot in avoiding confusing about what step is missing.
2018-10-24 15:02:08 -07:00
Tim Abbott 5cec566cb9 queue_processors: Rewrite MissedMessageWorker to always wait.
Previously, MissedMessageWorker used a batching strategy of just
grabbing all the events from the last 2 minutes, and then sending them
off as emails.  This suffered from the problem that you had a random
time, between 0s and 120s, to edit your message before it would be
sent out via an email.

Additionally, this made the queue had to monitor, because it was
expected to pile up large numbers of events, even if everything was
fine.

We fix this by batching together the events using a timer; the queue
processor itself just tracks the items, and then a timer-handler
process takes care of ensuring that the emails get sent at least 120s
(and at most 130s) after the first triggering message was sent in Zulip.

This introduces a new unpleasant bug, namely that when we restart a
Zulip server, we can now lose some missed_message email events;
further work is required on this point.

Fixes #6839.
2018-10-24 14:43:36 -07:00
Tim Abbott 9ed3fe3596 events: Improve logging for batched missed-message email handler. 2018-10-24 11:21:51 -07:00
Aaditya Arora f4d4c55d92 dependencies: Upgrade mypy to version 0.641.
This requires upgrading a few other packages that mypy depends on.
2018-10-24 10:48:05 -07:00
Rishi Gupta 458169928c billing: Rename Zulip Premium to Zulip Standard. 2018-10-24 10:42:16 -07:00
Vishnu Ks 53f1714613 stripe: Fix the button glitch in /upgrade page. 2018-10-24 10:12:39 -07:00
Rishi Gupta 9a3e277cc6 portico: Add HipChat import to /features. 2018-10-23 17:55:13 -07:00
Rishi Gupta a4d3606bad billing: Assume we are charging right away when adding new users.
The actual implementation of the change will be a cron job that runs once a
day and generates invoices for anyone with an account_balance > 0.

There are currently no tests for that part of the flow, so no tests had to
change.
2018-10-23 16:19:47 -07:00
Steve Howell 409e2b4134 hipchat import: Support sender_id == 0 use case. 2018-10-23 17:27:37 -05:00
Steve Howell 48b5b2f5d5 hipchat import: Start adding tests.
These test are for the handling of HipChat
sender info.  The data formats are somewhat
inconsistent and sometimes require us to
generate "mirror" users, so this is potentially
fragile code if we don't cover it well.
2018-10-23 17:27:37 -05:00
Steve Howell 876a72c467 hipchat import: Extract get_hipchat_sender_id(). 2018-10-23 17:27:37 -05:00
Steve Howell 481488a35e Extract make_subscriber_map().
We extract this function and put it in the shared
library `import_util.py`.

Also, we make it one time higher up in the call
stack, rather than re-building it for every batch
of messages.  I doubt this was super expensive, but
there's no reason to repeatedly execute this.
2018-10-23 17:27:37 -05:00
Steve Howell 737e02a2e6 hipchat import: Fix PM messages.
Before this fix, we were creating two copies of every
PM Message in zerver_message with only corresponding
UserMessage row.

Now we only create one PM Message per message, which
we accomplish by making sure we only use imported
messages from the sender's history.json file.  And
then we write UserMessage rows for both participants
by making sure to include sender_id in the set of
user_ids that feeds into making UserMessage.  For
the case where you PM yourself, there's just one
UserMessage row.

It does not appear that we need to support huddles
yet.
2018-10-23 17:27:37 -05:00
Steve Howell 53436b4b41 import: Rename id_maps -> ID_MAP. 2018-10-23 17:27:37 -05:00
Steve Howell bd9e4ef0c8 import: Use pub_date to sort message ids.
When we create new ids for message rows, we
now sort the new ids by their corresponding
pub_date values in the rows.

This takes a sizable chunk of memory.

This feature only gets turned on if you
set sort_by_date to True in realm.json.
2018-10-23 17:27:37 -05:00
Steve Howell d1ff903534 refactor: Rename build_user -> build_user_profile.
This makes greps less confusing.
2018-10-23 17:27:37 -05:00
Rishi Gupta b26e8eca62 portico: Redirect /plans to zulipchat.com/plans for self-hosted realms. 2018-10-23 14:47:00 -07:00
Rishi Gupta fbd884f934 billing: Rename MAX_INVITES_PREMIUM.
Largely to have parallel structure with INVITES_DEFAULT_REALM_DAILY_MAX.
2018-10-23 14:47:00 -07:00
Rishi Gupta a7fe6b1c34 billing: Fix max_invites for PREMIUM_FREE.
We could migrate all the current PREMIUM_FREE organizations to have more
invites, but this setting mainly affects orgs right as they are starting, so
it's probably fine.
2018-10-23 14:47:00 -07:00
Eeshan Garg 4c0890e8b0 webhooks/github: Handle empty 'requested_reviewers' key.
We recently received a bug report that implied that for certain
payloads, the `requested_reviewers` key was empty whereas a
singular `requested_reviewer` key containing one reviewer's
information was present in its stead. Naturally, this raised
some not so pretty IndexError exceptions.

After some investigation and generating a few similar payloads,
I discovered that in every case both the `requested_reviewers`
and the `requested_reviewer` keys were correctly populated, so I
had to manually edit the payload to reproduce the error on my end.

My guess is that this anomaly goes back to when GitHub's reviewer
request feature was new and didn't support requesting multiple
reviewers, and that the singular `requested_reviewer` key could
possibly just be there for backwards compatibility or might just
be mere oversight. Either way, the solution here is to look for the
plural `requested_reviewers` key, and if that is empty, fall back
to the singular `requested_reviewer` key.
2018-10-23 14:41:24 -07:00
Shubham Padia 5ea7feee7e upgrade-zulip-from-git: Support specifying tag or commit ID for refname.
Fixes #10706.
Issue: Before this commit, the `refname` positional argument to
`upgrade-zulip-from-git` script would run successfully for a branch
name on the given remote, but the script would fail if it was
provided with a tag or commit ID.
Solution: 'git clone -q -b refname LOCAL_GIT_CACHE_DIR deploy_path`
would be split into two commands:
1.) `git clone -q LOCAL_GIT_CACHE_DIR deploy_path`
2.) `git checkout -b deploy_timestamp refname` which makes a new
branch with the same name as the timestamp used in make_deploy_path.
2018-10-23 11:11:08 -07:00
neiljp (Neil Pilgrim) 9f8d778d20 integrations: Update Matrix and IRC docs.
[with edits by Rishi Gupta]
2018-10-22 18:04:22 -07:00
Rishi Gupta 1c264dedc5 api docs: Change Shakespeare quote.
The previous quote doesn't come off well if you don't know the reference
(which the majority of our users will not).
2018-10-22 15:44:36 -07:00
Scott Lee b4f95aa124 doc: Expand documentation for SSH port forwarding.
This adds additional links and examples to make it much more clear how
to do secure SSH port-forwarding with a remote development environment.
2018-10-22 14:53:52 -07:00
Tim Abbott d69d71c59f i18n: Sync translation data from transifex. 2018-10-22 14:50:46 -07:00
Tim Abbott adf918cf5a upgrade-zulip-from-git: Fix linter error from too-long line. 2018-10-22 14:11:40 -07:00
Eeshan Garg 3a47a1a627 webhooks/greenhouse: Ignore ping events.
"ping" type events are usually just Greenhouse's way of indicating
that a webhook has been set up.
2018-10-22 12:27:48 -07:00
Steve Howell c58a99b156 PM sidebar: Expand PM sidebar for huddles.
This fixes a regression from here:

    88b4a9f2d7

The fix didn't account for how huddles are
represented as comma-delimited strings.

We also simplify the logic by extracting a
function and doing early-exit for simple
cases.
2018-10-22 12:22:26 -07:00
Steve Howell 328e2ff316 Fix "Copy link to conversation" links.
This cleans up the code for stream links
and creates nicer, more correct links for
PMs.

Fixes #10605
2018-10-22 12:22:26 -07:00
Steve Howell adf616d3f1 Add pm_perma_link helper. 2018-10-22 12:22:26 -07:00