Commit Graph

703 Commits

Author SHA1 Message Date
Vishnu Ks bec875a9af import realm: Use processes for resizing avatar images.
This should significantly improve the data import performance when
importing large open source realms from Slack.

Fixes #11009.
2019-01-25 12:37:12 -08:00
Anders Kaseorg c0792808e2 deliver_email, deliver_scheduled_messages: Remove lock file.
Tim believes it’s useless.

https://github.com/zulip/zulip/pull/11285#discussion_r248110926

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:56:49 -08:00
Anders Kaseorg 25d6107a4f docs: Avoid hardcoded /tmp paths in miscellaneous documentation.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:16:12 -08:00
Anders Kaseorg b37ed12df2 export: Let mkdtemp pick its favorite directory.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-01-15 16:08:48 -08:00
Aditya Bansal aae90d5d6b deliver_scheduled_messages: Make notification bot deliver reminders.
We do this since we are yet to figure out how the entire realm
internal bots scenerio should work and therefore for the timming
we will use notification bot to deliver the reminders.
2019-01-14 12:11:27 -08:00
Harshit Bansal 00ca98348a ldap: Shorten the name of logger of `sync_ldap_user_data` command. 2019-01-14 09:50:30 -08:00
Harshit Bansal 5037b2c5c4 management: Fix a bug in parser of `sync_ldap_user_data` command.
After recent changes for extending the `sync_ldap_user_data` to
update a single user, it was not running without specifying any
arguments.
2019-01-14 09:50:30 -08:00
Harshit Bansal 71761bc2da ldap: Add a setting to automatically deactivate non_matching users.
Fixes: #11151.
2019-01-13 19:04:09 -08:00
Harshit Bansal 348f370b79 management: Extract `sync_user_from_ldap()`. 2019-01-13 18:51:50 -08:00
Harshit Bansal e09ef69a8d management: Extend `sync_ldap_user_data` to allow update of a single user. 2019-01-13 18:51:50 -08:00
Tim Abbott 53436766c1 hipchat: Improve import of public room subscribers.
Now, if you pass an api_key, we'll initialize the public room
subscribers to be whatever they were at the time the import happened.

Also, document the situation on the caveats section.
2019-01-09 16:50:00 -08:00
Tim Abbott a3ac94fa26 management: Support multi-line usage in all management commands. 2019-01-09 10:39:29 -08:00
Mateusz Mandera 3e5f89f2fe email_mirror: Add send_to_email_mirror management command.
Closes #11195. We add a management command to allow us to send emails
to the email mirror directly. The command doesn't require any
configuring of email sending or receiving for the email mirror,
it passes the emails directly using the process_message function.
2019-01-09 10:36:16 -08:00
sahil839 4dcccf32f8 zerver: Add feature for notification at rename of a stream.
Feature of sending notification to the stream using notification bot
is added. user_profile is also passed to do_rename_stream for using
the name of user who renamed the stream in notification.
Notification is sent to the stream using
internal_send_stream_message in do_rename_stream.

Fixes #11034.
2019-01-07 19:13:01 -08:00
Tim Abbott 848b2f687c export: Add support for public-streams-only exports.
Previously, this wasn't an explicit feature of the export tool.

Note that the current version still includes metadata on private
streams and private message recipients, just not their messages.
2019-01-07 16:52:02 -08:00
Tim Abbott c6371bb3ef export: Add option to upload exports to S3.
This should make it more convenient to operationalize providing
exports from Zulip Cloud.

Fixes #11178.
2019-01-07 15:28:17 -08:00
Tim Abbott ff9714eb32 register_server: Prompt for ToS inside the command.
This way, we don't need to tell new sysadmins how to rerun the script.
2019-01-02 11:19:36 -08:00
Tim Abbott 22f39ff8e7 list_realms: Display '' as string_id for root domain realm. 2018-12-30 10:36:18 -08:00
Tim Abbott e48c9245ff list_realms: Fix the --all option.
I don't think anyone's used this in years, but it could be handy.
2018-12-30 10:36:18 -08:00
Hudda 50e14dbe97 list_realms: Print the full domains of the organizations.
This adds a column to display the full domains of realms.  Tweaked by
tabbott to use fewer columns.

Fixes part of #11015.
2018-12-30 10:36:00 -08:00
Rishi Gupta 0844c9a6ec emails: Update text for password_reset.
Note that a pretty common use case for this is a realm admin sending this to
everyone after an import from HipChat or Slack. So this adds the realm_name
to the title (so that there is something they might recognize) and kept the
wording generic enough to accommodate the user not having clicked anything
to get this email.

Also strengthens the tests a bit to better test the complicated template
logic.
2018-12-20 16:26:19 -08:00
Tim Abbott d005f4edd3 management: Fix incorrect create_user docsting on ToS. 2018-12-16 22:23:35 -08:00
Tim Abbott d9912cb23c management: Fix buggy create_user management command.
The create_user management command incorrectly was double-calling
notify_new_user.
2018-12-16 22:22:52 -08:00
Tim Abbott 67981725ec send_email: Fix migration code path for ScheduledEmail.
The previous migration code path was broken in two ways:

* ScheduledEmail objects generally contain a `None` value for
  whichever of `to_user_id` and `to_email` isn't in use; this could
  result in us sending a [None] to send_email(), which doesn't make
  sense.

* We were calling handle_send_email_format_changes in the wrong order
  with respect to the JSON loading process.

Thanks to Tom Daff for the report!
2018-12-14 12:46:31 -08:00
Tim Abbott 81271b0d20 sync_ldap_user_data: Process deactivated users as well.
Technically, we will only need to process deactivated users for the
purpose of reactivating them (and can ignore, e.g., name changes).
But it's simplest to just process them unconditionally.
2018-12-13 16:24:15 -08:00
Tim Abbott 5dd646f33f ldap: Add support for syncing avatar images from LDAP.
This should make life a lot more convenient for organizations that use
the LDAP integration and have their avatars in LDAP already.

This hasn't been end-to-end tested against LDAP yet, so there may be
some minor revisions, but fundamentally, it works, has automated
tests, and should be easy to maintain.

Fixes #286.
2018-12-13 13:39:22 -08:00
Tim Abbott ee901ac8b1 django: Remove the Django contrib Sites app to fix 2FA QR codes.
Apparently, Django's get_current_site function (used, e.g., in
django-two-factor to look up the domain to use in QR codes) first
tries to use the Sites framework, and if unavailable, does the right
thing (namely, using request.get_host()).

We don't use the Sites framework for anything in Zulip, so the correct
fix is to just remove it.

Fixes #11014.
2018-12-12 15:01:55 -08:00
Joshua Pan 441dca0eaf email: Support delivery email in email change system. 2018-12-06 17:43:26 -08:00
Tim Abbott e603237010 email: Convert accounts code to use delivery_email.
A key part of this is the new helper, get_user_by_delivery_email.  Its
verbose name is important for clarity; it should help avoid blind
copy-pasting of get_user (which we'll also want to rename).
Unfortunately, it requires detailed understanding of the context to
figure out which one to use; each is used in about half of call sites.

Another important note is that this PR doesn't migrate get_user calls
in the tests except where not doing so would cause the tests to fail.
This probably deserves a follow-up refactor to avoid bugs here.
2018-12-06 16:21:38 -08:00
Tim Abbott 02a79b677b send_email: Extract handle_email_format_changes and use.
Apparently, we have a second code path where we might try to call
send_email library functions on old data, namely in the
queue_processors codebase.  So we apply the same migration logic here.
2018-12-04 16:08:18 -08:00
Raymond Akornor 92dc3637df send_email: Add support for multiple recipients.
This adds a function that sends provided email to all administrators
of a realm, but in a single email. As a result, send_email now takes
arguments to_user_ids and to_emails instead of to_user_id and
to_email.

We adjust other APIs to match, but note that send_future_email does
not yet support the multiple recipients model for good reasons.

Tweaked by tabbott to modify `manage.py deliver_email` to handle
backwards-compatibily for any ScheduledEmail objects already in the
database.

Fixes #10896.
2018-12-03 15:12:11 -08:00
Tim Abbott 7a2d93ed6e show_admins: Add assertion to make mypy happy. 2018-11-29 11:51:39 -08:00
Tim Abbott 8a60ab7314 show_admins: Rewrite to use management library.
This makes this command more standardized, and helps avoid future bugs
like the one fixed in the last commit.
2018-11-29 11:40:17 -08:00
Tim Abbott d828eba2f6 show_admins: Fix buggy realm parsing. 2018-11-29 11:40:17 -08:00
Tim Abbott adf27aae4c python: Remove now-unnecessary str_utils library.
This library was absolutely essential as part of our Python 2->3
migration process, but all of its calls should be either no-ops or
encode/decode operations.

Note also that the library has been wrong since the incorrect
refactoring in 1f9244e060.

Fixes #10807.
2018-11-27 11:57:54 -08:00
Raymond Akornor d00b889402 auth: Add an organization reactivation flow with admin confirmation.
This adds a web flow and management command for reactivating a Zulip
organization, with confirmation from one of the organization
administrators.

Further work is needed to make the emails nicer (ideally, we'd send
one email with all the admins on the `To` line, but the `send_email`
library doesn't support that).

Fixes #10783.

With significant tweaks to the email text by tabbott.
2018-11-27 10:58:34 -08:00
Tim Abbott 46acb608b1 tornado: Include port number in logging statements. 2018-11-20 18:45:22 -08:00
Tim Abbott 0cac7e1cd3 tornado: Extract functions for Tornado queue names.
This moves all control for what queue to use for which realm in our
Tornado system to just the sharding.py file; no actual sharding is
done yet.
2018-11-02 17:00:10 -07:00
Tim Abbott ec065e92ee tornado: Store port on SockJS connection object.
This will make it available for use inside our websockets code.
2018-11-02 16:55:33 -07:00
Tim Abbott 75e48459b5 tornado: Support using a port-aware file for dumping event queues.
This should make it possible for there to safely be multiple Tornado
processes running on different ports on the same system.

It may also fix a rare race bug in development, where previously, it
was possible for the Tornados processes for Casper and the main
development server to interfere; I haven't investigated whether this
was a real bug or not, but now those two services will use independent
Tornado files.

We still need to add something to direct traffic between the different
Tornado processes.
2018-11-02 16:47:39 -07:00
Tim Abbott 0976e3bd39 purge_queue: Use more robust Tornado patterns.
We're about to make the list of possible Tornado queue names broader.
2018-11-02 16:06:46 -07:00
Tim Abbott 4c55de4b58 hipchat import: Don't require the output directory to already exist.
This fixes a common exception one might see when trying to use the
HipChat importer for the first time.
2018-11-01 10:09:19 -07:00
Steve Howell 272b954790 hipchat import: Add option to mask content.
Masking content can be useful for testing
out conversions where you're dealing
with data from customers and want to avoid
inadvertently reading their content (while
still having semi-realistic messages).
2018-10-25 08:31:01 -05:00
Steve Howell 85f1910f93 minor: Add link to hipchat spec to code. 2018-10-13 16:43:28 -07:00
Vishnu Ks 6972de21be management: Add command to scrub a realm of personal data. 2018-10-11 15:30:26 -07:00
Steve Howell faea26783b Create convert_hipchat_data.
This is a very early version of a tool to convert Hipchat
tar files into data files that can be used by the Zulip
import process.

We include the most fundamental entities--users and
streams.  Customers who don't care about past messages
or customizations could start an instance off of this
and start communicating.

Of course, there are a lot of things missing in the
initial version:

    * messages!
    * file assets -- avatars, emojis, attachments
    * probably lots of other minor things

We currently ignore any incoming dates from Hipchat data
and just use the current time.  This is consistent with
other imports.

We also don't have any docs yet, although the process
will be extremely similar to the "Slack" process:

    https://zulipchat.com/help/import-from-slack

Also, there's a comment at the top of convert_hipchat_data.py
that describes how to test this in dev mode.

I tested this by following the steps in the comment above.
The users just "show up" in /devlogin, so that's nice, and
you can send messages to other users.  To verify the stream
data you have to go into the gear menu and click on "All
Streams", then you can subscribe and send a message.

Production users will need to get new passwords and
re-subscribe to streams.  We will probably auto-subscribe
all users to public streams.
2018-10-02 16:55:16 -07:00
Tim Abbott 69b2315a6e password reset: Deduplicate code for reset URL generation. 2018-08-23 12:06:02 -07:00
Vishnu Ks 0f628dc4aa management: Fix send_password_reset_email command.
This had been broken in a recent refactoring of the password-reset
code path.

Fixes #10379.
2018-08-23 12:05:49 -07:00
Tim Abbott 972e1d2d8b ldap: Improve exception handling in user data sync tool.
At some point, the most likely exception here stopped being
IntegrityError.

Fixes #9218.
2018-08-20 10:40:53 -07:00
Rishi Gupta c3a912f8b6 management commands: Add sleep_forever to lib/management. 2018-08-17 09:20:51 -07:00