Commit Graph

5521 Commits

Author SHA1 Message Date
Tim Abbott 06f8fd529d realm: Add clear error message for too-long realm name.
This fixes an exception we saw in production, as well as fixing the
form to not offer ultra-long names that we don't allow.
2017-08-22 22:04:52 -07:00
Tim Abbott 669b1b8adb mypy: Fix an annotation. 2017-08-22 17:39:24 -07:00
Steve Howell 110cfcb640 Support --all options in fix_unreads.
(We also changed the email metavar to accept multiple emails.)
2017-08-22 20:06:30 -04:00
Steve Howell 752bf14443 fix_unreads: Improve cursor handling.
Before adding the context manager, we were leaking cursors, which
was a non-issue for updating individual users, but it's easy to
fix.
2017-08-22 20:06:26 -04:00
Steve Howell c5e083857b fix_unreads: Mark pre-pointer messages as unread (unless muted). 2017-08-22 20:03:18 -04:00
Steve Howell 8b16c6060c Split out fix_unreads management command.
Most of the code in show_unreads is for diagnosising unread
counts issues, and we may not use that often.

We're creating a dedicated fix_unreads management command with
less clutter.
2017-08-22 19:38:44 -04:00
Tim Abbott 7c37fc8909 auth: Improve robustness of not sending duplicate login emails.
Previously, we had a very not-robust check on the URL, which might
have caused cases like Google auth registration to not do the right
thing.
2017-08-22 16:26:30 -07:00
Preston Hansen c31ef34491 enqueue_digest_emails: Only send digest emails weekly, on Tuesday.
Fixes: #6114.
2017-08-22 15:07:53 -07:00
Rishi Gupta 4e7d87b676 onboarding.py: Add link to getting-your-organization-started-with-zulip.
Also has a few minor updates to the rest of the text.
2017-08-22 14:26:17 -07:00
Rishi Gupta d296f29fa7 emails: Link to organization getting started guide in followup_day1. 2017-08-22 14:26:17 -07:00
Rishi Gupta 7c3f20d2ba registration: Set is_realm_admin on user creation.
This makes it easier for later parts of the user creation/onboarding process
to condition on whether the user is a realm admin.

No change in behavior.
2017-08-22 14:26:17 -07:00
Umair Khan a5c05f9812 push_notification: Retry event in case of error.
Fixes #5301
2017-08-22 11:16:48 -07:00
Umair Khan 62cae23601 queue: Extract event retry into retry_event().
This commit takes the code from zerver.lib.outgoing_webhook.request_retry
and creates a new generic function called retry_event.
2017-08-22 11:16:48 -07:00
Steve Howell 81e3f489f2 Use sender realm in user_profiles_from_unvalidated_emails.
This change is mostly based on a similar commit from hackerkid
in a feature branch.  It borrows both code and ideas.  Some of
it's my own stuff, as I was working on a newer branch.

We now call get_user_including_cross_realm_email() inside of
user_profiles_from_unvalidated_emails(), instead of using
get_user_profile_by_email.

This requires a few of our callers to pass down sender into us.

One consequence of this change is that we change the symptoms
for trying to send to emails outside of your realm.  In some
cases, we simply raise an error that an email is invalid to us
instead of getting into the deeper validate_recipient_user_profiles
check.
2017-08-22 10:42:15 -07:00
Steve Howell 54edecd510 Replace adddressee.for_email() with for_user_profile().
This requires us to change not just the immediate caller, but
also some of their callers, to pass user_profile objects around
instead of emails.
2017-08-22 10:42:15 -07:00
Steve Howell c61c0f3edc Use user_profile, not email, in check_message().
We are trying to convert emails to user_profiles earlier in
the codepath.  This may cause subtle changes in which errors
appear, but it's probably generally good to report on bad
addressees sooner than later.
2017-08-22 10:42:15 -07:00
Steve Howell c56a631b65 Move user_profiles_from_unvalidated_emails -> addressee.py.
This will avoid circular dependencies.
2017-08-22 10:42:15 -07:00
Steve Howell 32f18f77d4 Extract two methods from recipient_for_emails.
We now have these methods:
    user_profiles_from_unvalidated_emails
    recipient_for_user_profiles
2017-08-22 10:42:15 -07:00
Steve Howell 30d37d1270 Add Addressee class.
This class simplifies the calling sequence to methods like
check_message and _internal_prep_message, and it's also more
type safe.

Checking for message types is encapsulated with calls to is_stream()
and is_private().  There are also shortcut constructors when you
know that the type of the address (stream vs. private), which is often.
2017-08-22 10:42:15 -07:00
Tim Abbott 58edf75962 api: Allow incoming webhook bots to use the send_message API. 2017-08-22 10:06:34 -07:00
Steve Howell e12da79732 create_realm: Redirect to a nice error page for SMTP errors.
Fixes #3850
2017-08-21 15:39:26 -07:00
Steve Howell 99e08fe49a accounts_home: Redirect to nice error page for SMTP errors. 2017-08-21 15:39:26 -07:00
Steve Howell ac80d1df0a Add /config-error/smtp endpoint for SMTP errors. 2017-08-21 15:39:26 -07:00
Aditya Bansal 7a2667ce53 soft_deactivation: Fix flaky tests by seeding message for users.
In this we basically seed a single message for the user which will
be soft deactivated by sending a stream message / group PM to
ensure that is has at least one UserMessage row, since in real
world every human user will always have at least one User Message
row.
2017-08-18 07:02:11 -04:00
Tim Abbott ef722c5688 tests: Remove unused ujson imports. 2017-08-17 09:05:52 -07:00
Umair Khan 6e87108d46 result.json: Upgrade test_users. 2017-08-17 09:03:35 -07:00
Umair Khan 212571ab86 result.json: Upgrade test_subs. 2017-08-17 09:03:35 -07:00
Umair Khan eb04fc006e result.json: Upgrade test_settings. 2017-08-17 09:03:35 -07:00
Umair Khan d44c1e7070 result.json: Upgrade test_realm_emoji. 2017-08-17 09:03:35 -07:00
Umair Khan 7b85e40ef5 result.json: Upgrade test_realm_domains. 2017-08-17 09:03:35 -07:00
Umair Khan 90099ffeb2 result.json: Upgrade test_reactions. 2017-08-17 09:03:35 -07:00
Umair Khan 351426df8b result.json: Upgrade test_presence. 2017-08-17 09:03:35 -07:00
Umair Khan 4c8933b844 result.json: Upgrade test_narrow. 2017-08-17 09:03:35 -07:00
Umair Khan 7bb9ac0014 result.json: Upgrade test_messages. 2017-08-17 09:03:35 -07:00
Umair Khan 025f7247cf result.json: Upgrade test_external. 2017-08-17 09:03:35 -07:00
Umair Khan 2b7bb6e965 result.json: Upgrade test_events. 2017-08-17 09:03:35 -07:00
Umair Khan de5f43ea47 result.json: Upgrade test_bots. 2017-08-17 09:03:35 -07:00
Umair Khan 1c09f95e0e result.json: Upgrade test_auth_backends. 2017-08-17 09:03:35 -07:00
Umair Khan abed403a81 result.json: Upgrade test_classes. 2017-08-17 09:03:35 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Greg Price aa0ecd79d6 zephyr: Fix bogus uses of `force_bytes` that break on Python 3.
An expression like `force_bytes(chr(...))`, on Python 3 where the
`force_bytes` finds itself with something to do because `chr` returns
a text string, gives the UTF-8 encoding of the given value as a
Unicode codepoint.

Here, we don't want that -- rather we want the given value as a
single byte.  We can do that with `struct.pack`.

This fixes an issue where the "Link with Webathena" flow was producing
invalid credential caches when run on Python 3, breaking the Zephyr
mirror for any user who went through it anew.
2017-08-16 17:23:57 -07:00
Tim Abbott 7cfb4e195f do_deactivate_user: Add acting user to RealmAuditLog. 2017-08-16 16:23:41 -07:00
Tim Abbott 98aae162dc do_change_full_name: Include original name in RealmAuditLog. 2017-08-16 16:23:41 -07:00
Tim Abbott d43f5ceeec database: Add database index for wildcard mentions. 2017-08-16 13:28:04 -07:00
Tim Abbott 8bb812c8a9 database: Add database index for alert words. 2017-08-16 12:39:01 -07:00
Steve Howell 3a825ef22c Add create_large_indexes management command.
This management command creates the same indexes as migrations
82, 83, and 95, which are all indexes on the huge UserMessage
table. (*)

This command quickly no-ops with clear messaging when the
indexes already exist, so it's idempotent in that regard. (If
somebody somehow creates an index by the same name incorrectly,
they can always drop it in dbshell and re-run this command.)

If any of the migrations have not been run, which we detect simply
by the existence of the indexes, then we create them using a
`CREATE INDEX CONCURRENTLY` command.  This functionality in
postgres allows you to create indexes against large tables
without disrupting queries against those tables.  The tradeoff
here is that creating indexes concurrently takes significantly
longer than doing them non-concurrently.

Since most tables are small, we typically just use regular
Django migrations and run them during a brief interval while
the app is down.

For indexes on big tables, we will want to run this command
as part of the upgrade process, and we will want to run
it while the app is still up, otherwise it's pointless.

All the code in create_indexes() is literally copy/pasted
from the relevant migrations, and that scheme should work
going forward.  (It uses a different implementation of
create_index_if_not_exist than the migrations use, but the
code is identical lexically in the function.)

If we ever do major restructuring of our large tables, such
as UserMessage, and we end up droppping some of these indexes,
then we will need to make this command migrations-aware.  For
now it's safe to assume that indexes are generally additive in
nature, and the sooner we create them during the upgrade process,
the better.

(*) UserMessage is huge for large installations, of course.
2017-08-16 12:38:51 -07:00
Tim Abbott 5061223dd3 streams: Send stream creation events to new zephyr mirror subscribers.
This fixes a frequent JS exception that we would get when subscribing
to a new stream in the Zephyr realms.
2017-08-16 12:09:15 -07:00
Tim Abbott fa4968da92 actions: Extract send_stream_creation_event. 2017-08-16 12:03:44 -07:00
Steve Howell 47bcedbc1b Fix server searches for is:mentioned and is:alerted.
Before this change, server searches for both
`is:mentioned` and `is:alerted` would return all messages
where the user is specifically mentioned (but not
at-all mentions).

Now we follow the JS semantics:

    is:mentioned -- all mentions, including wildcards
    is:alerted  -- has an alert word

Here is one relevant JS snippet:

        } else if (operand === 'mentioned') {
            return message.mentioned;
        } else if (operand === 'alerted') {
            return message.alerted;

And here you see that `mentioned` is OR'ed over both mention flags:

    message.mentioned = convert_flag('mentioned') || convert_flag('wildcard_mentioned');

The `alerted` flag on the JS side is a simple mapping:

    message.alerted = convert_flag('has_alert_word');

Fixes #5020
2017-08-16 11:29:12 -07:00
Tim Abbott 7d49ce13b1 bugdown: Fix mypy error with empty return. 2017-08-16 11:29:12 -07:00