Commit Graph

220 Commits

Author SHA1 Message Date
Umair Khan 19e2551e82 mypy: Change type to Dict for SignupWorker. 2017-10-05 23:14:19 -07:00
Umair Khan d95d34a66a Retry email failures in missed-message emails queue.
Fixes #6518.
2017-10-03 10:35:07 -07:00
Umair Khan 7d6ddaad26 mypy: Use Dict instead of Mapping in queues. 2017-10-03 10:35:07 -07:00
Greg Price 2a832386dc queue: Move more received-event logs to DEBUG level.
These can spew a lot of noise when starting the dev server, and aren't
in a super helpful form for normal server administration anyway.
2017-09-28 18:32:42 -07:00
Tim Abbott 630ecf9669 queue: Move user presence log events to DEBUG log level.
These are pretty noisy, and generally of little interest, since
presence requests come with a server log entry anyway.
2017-09-28 14:35:55 -07:00
rht 2ae47525d3 zerver/worker: Remove `import six`. 2017-09-27 17:06:16 -07:00
rht 41d8db373a zerver/worker: Remove absolute_import. 2017-09-27 10:00:39 -07:00
Tim Abbott 84926ff321 slow_queries: Log slow queries even if not reported via Zulip.
This should help make these logs more useful for debugging on an
arbitary Zulip system.
2017-09-22 14:11:07 -07:00
Tim Abbott ec61a070b4 signups: Add basic logging for new signup processing. 2017-09-22 14:11:07 -07:00
Tim Abbott f78dd33037 invites: Add logging for newly sent invitation emails. 2017-09-22 14:11:07 -07:00
Rishi Gupta a7c8770f97 emails: Move enqueue_welcome_emails outside of signups queue.
The only thing this queue should do is sign you up for the newsletter, since
it is only populated if newsletter_data is not None.
2017-09-22 06:20:33 -07:00
Tim Abbott f706f657c0 signup: Fix invitation emails not being cleared properly.
Previously, invitation reminder emails were only being cleared after a
successful signup if newsletter_data was available, since that was the
circumstance in which we were calling the relevant queue processor
code.  Now, we (1) clear them when a human user finishes signing up
and (2) correctly clear them using the 'address' field of
ScheduleEmail, not user_id.
2017-09-21 06:15:11 -07:00
Steve Howell ba397b5109 Use user_ids, not full objects, in render path.
There is no reason for either render_incoming_message() or
render_markdown() to require full UserProfile objects just to
triage alert words.

By only asking for user_ids, we save extra queries in two
callpaths and we make it easier to start using user_ids in
do_send_messages().
2017-09-12 04:22:55 -07:00
neiljp (Neil Pilgrim) 116b8dd9c9 mypy: Set assign_queue() parameter queue_type to not be Optional. 2017-08-07 21:27:50 -07:00
Abhijeet Kaur 1deb58b178 bots: Add complete test-coverage for bot_lib.py file.
Also, add error handling for get_bot_handler instead of
throwing an assertion error.
2017-07-27 15:50:29 -07:00
Abhijeet Kaur 6f60c65a65 embedded bots: Add tests for verification of embedded bot services.
Add test to check if the embedded bot service being used is in the
registry or not.
Add test to check if the bot being added to the registry has a valid
bot corresponding to it.
Move 'get_bot_handler' to 'zerver/lib/bot_lib.py' as it is an independent
function, not related to the 'EmbeddedBotWorker' class that it was
previously a part of.
2017-07-24 17:14:14 -07:00
Elliott Jin 6a61a8a431 outgoing webhooks: Consolidate interfaces into lib/outgoing_webhook.py 2017-07-24 14:10:14 -07:00
vaibhav 87dcd5442a outgoing webhook system: Minor fixes.
This fixes some error message strings and skips converting request_data
into json. From now, conversion would be the responsibility of interface.
Also, base_url is now not passed into event structure.
2017-07-24 14:10:14 -07:00
Abhijeet Kaur 5980d420a8 Embedded bots: Fix minor errors to make embedded bots/service run.
Splitting bot_lib.py file into 2 files led to unnecessary
redirection of the code workflow. For an embedded bot/service to
send a reply, it was being redirected 3 times.

First, the code flow comes to "EmbeddedBotHandler" class to send
reply, then it goes to the common function in "zulip_bots/lib.py",
then it would come back to "EmbeddedBotHandler". Later on, if we
create an abstract class, from where the bot work flow would
directly hit and then from there it is classified into
EmbeddedBotHandler or ExternalBotHandler and accordingly it would
get redirected.

Now, first the bot flow goes to it's handler class External or
Embedded (where we pass that this is External or Embedded bot as
parameter) and then goes to a common point and then comes back to
the same class.
2017-07-20 10:22:52 -07:00
Eeshan Garg 148bb4db09 requirements: Update requirements/ to install bots/API packages.
This is required, since we just reorganized the python-zulip-api
repository into 3 packages.

A nice side effect is that we get to eliminate some now-unnecessary
code for editing sys.path.
2017-07-18 00:10:30 -07:00
Rishi Gupta 0f4b71b766 confirmation: Liberate get_link_for_object from ConfirmationManager. 2017-07-17 23:18:47 -07:00
Rishi Gupta 36dbb76516 emails: Rename clear_followup_emails_queue. 2017-07-17 16:05:38 -07:00
Rishi Gupta 5b3e6af2e5 emails: Remove only emails of the correct type when clearing queue. 2017-07-17 16:05:38 -07:00
Rishi Gupta f51bd898dc notifications: Change clear_followup_emails_queue to take a user_id. 2017-07-17 16:05:38 -07:00
Rishi Gupta 898269bbac email: Change send_email to raise exception on failure.
More in line with how we do error handling in the rest of Zulip.
2017-07-16 16:56:39 -07:00
Rishi Gupta eacdb0b302 emails: Change welcome emails to use to_user_id. 2017-07-16 16:56:39 -07:00
Rishi Gupta b0d325b8c5 emails: Change send_future_email to accept a to_user_id.
Also changes digest emails to use a to_user_id instead of a to_email.
2017-07-16 16:56:39 -07:00
Vishnu Ks ba4ea7dd8a notifications.py: Replace get_user_profile_by_email.
Replace with get_user.
2017-07-13 00:45:24 +05:30
Aditya Bansal d46cf59b0d pep8: Add compliance with rule E261 to worker/queue_processors.py. 2017-07-11 11:55:02 -07:00
Rishi Gupta 8fed9eeb75 confirmation: Make host a required argument in get_link_for_object.
Removes some lines of test from test_email_change.py. The relevant code path
was never utilized by the code itself, just by the tests.
2017-07-07 18:53:00 -07:00
Umair Khan ee6e07d13f message_sender: Use response_data variable.
response_data already contains the result of ujson.loads.
2017-07-07 16:33:36 -07:00
Umair Khan 0e8231d0f1 process_queue: Recover gracefully after PostgreSQL restart.
- For threaded workers:
Django's autoreloader catches SIGQUIT(3) to reload the program. If
a process being watched by autoreloader exits with status code 3,
reloader will restart the process. To reload, we send SIGUSR1(10)
signal from consumers to a handler in process_queue which then
exits with status code 3.

- For single worker per process:
Catch the SIGUSR1 and quit; supervisorctl will restart the worker
automatically.

Fixes #5512
2017-07-07 16:33:15 -07:00
James Rowan 0951666cbb emails: Confirmation emails should come from the NOREPLY address.
This prevents users from accidentally sending a confirmation link
specific to their account to their Zulip administrator if they reply
to the invitation, invitation reminder, account confirmation, or new
email confirmation emails.
2017-07-05 15:18:33 -07:00
James Rowan d88e7308bf emails: Add a FromAddress class to control access to certain settings emails.
No change in behavior.

Also makes the first step towards converting all uses of
settings.ZULIP_ADMINISTRATOR and settings.NOREPLY_EMAIL_ADDRESS to
FromAddress.*.

Once everything is converted, it will be easier to ensure that future
development doesn't break backwards compatibility with the old style of
settings emails.
2017-07-04 14:25:01 -07:00
James Rowan 368bd66d8b emails: Refactor send_email functions to take both a sender name and address.
This will allow for customized senders for emails, e.g. 'Zulip Digest' for
digest emails and 'Zulip Missed Messages' for missed message emails.

Also:
* Converts the sender name to always be "Zulip", if the from_email used to
  be settings.NOREPLY_EMAIL_ADDRESS or settings.ZULIP_ADMINISTRATOR.

* Changes the default value of settings.NOREPLY_EMAIL_ADDRESS in the
  prod_setting_template to no longer have a display name. The only use of
  that display name was in the email pathway.
2017-07-04 14:25:01 -07:00
vaibhav 7f9b4d3a94 Outgoing Webhook System: Add usage of Interfaces in DoRestCall. 2017-06-28 11:11:21 -04:00
Abhijeet Kaur cf0040c402 bots: De-duplicate removing at-mention pattern for calling bots.
Similar code appeared at two places (the code is meant to remove the
leading @-mention before passing the remainder of the message to the
bot handler)—both 'zerver/worker/queue_processors.py' and
'api/bots_api/bot_lib.py'.

Remove function from the queue_processors.py file and let the entire message
be processed (includes removing of @-mention) at bot_lib.py.
2017-06-26 08:33:25 -04:00
Abhijeet Kaur c42d935be8 bots: Move "EmbeddedBotHandler" class to "zerver/lib/bot_lib.py" file.
This would keep embedded classes for zulip at one place, that is, in
"zerver" directory. This also fixes break in PyPI package for bindings.
2017-06-21 16:01:16 -04:00
Abhijeet Kaur a45c224fda bots: Update EmbeddedBotWorker class in queue_processors.py.
Update EmbeddedBotWorker.get_bot_api_client in
'zerver/workers/queue_processors.py' to return an EmbeddedBotHandler
object instead of ExternalBotHandler object.

This change is as a followup for splitting the BotHandler class into
two separate classes for external and embedded bots.
2017-06-21 10:22:53 -04:00
Abhijeet Kaur 68777e93a0 bots: Rename BotHandlerApi class to ExternalBotHandler.
This change is done for better understanding of the class functionality
from its class name. Now there will be 3 different classes for bots,
namely 'ExternalBotHandler', 'FlaskBotHandler' and  'EmbeddedBotHandler'.
2017-06-21 10:22:53 -04:00
Rishi Gupta 15b967fc3e emails: Move support_email into a common context. 2017-06-10 01:25:44 -07:00
Rishi Gupta 056489c247 emails: Move verbose_support_offers into a common context.
Server settings should just be added to the context in build_email, so that
the individual email pathways (and later, the email testing framework)
doesn't have to worry about it.
2017-06-10 01:16:03 -07:00
Rohitt Vashishtha db9918f3d6 bots: Move contrib_bots to api/bots*.
This will make it convenient to include these bots in Zulip API
releases on pypi.

Fix #5009.
2017-06-01 12:31:54 -07:00
Elliott Jin 0ec9e54954 bots: Add queue and QueueProcessingWorker for embedded bots. 2017-05-25 15:00:51 -07:00
Konstantin Gukov dd76222a3f Fetch system bots using new get_system_bot function.
This eliminate a bunch of uninteresting calls to
get_user_profile_by_email.
2017-05-23 10:30:40 -07:00
vaibhav 287aaa21b6 Outgoing Webhook System: Add outgoing webhook queue worker. 2017-05-23 08:21:08 -07:00
Tim Abbott e55b4b0d43 invites: Fix buggy access to non-existant user_profile.name. 2017-05-10 14:31:24 -07:00
Tim Abbott a85c066722 confirmation: Fix race with invite and email change. 2017-05-10 14:31:24 -07:00
Rishi Gupta 8321bd3f92 notifications: Refactor missed message queue to use send_email.
This commit also changes the sender from "Zulip <NOREPLY_EMAIL_ADDRESS>" to
NOREPLY_EMAIL_ADDRESS when it is not set explicitly.
2017-05-05 14:38:25 -07:00
Rishi Gupta 6a8ed81439 send_email: Remove tags argument from send_future_email.
The tags argument was only being used by the Mandrill pathway, which is no
longer around.
2017-05-05 14:38:25 -07:00
Rishi Gupta 72ee5e5159 email: Make context for invitation_reminder template jsonable.
We are about to change send_future_email in a way that will require all
callers to use jsonable context.
2017-05-05 14:38:25 -07:00
Rishi Gupta 68c172192b email.py: Change recipients argument of send_future_email to to_email. 2017-05-05 14:38:25 -07:00
Rishi Gupta 0d4c1b0467 notifications: Move send_future_email to zerver/lib/send_email.py. 2017-05-05 14:38:25 -07:00
Rishi Gupta d70e09b41d notifications: Change sender arg of send_future_email to from_email.
This commit is a step towards the goal of replacing most of the
send_future_email pathway with a call to send_email.

Note that this commit changes the default value of sender from "Zulip
<NOREPLY_EMAIL_ADDRESS>" to "NOREPLY_EMAIL_ADDRESS". NOREPLY_EMAIL_ADDRESS
will soon be changed to have the Zulip in front.
2017-05-05 14:38:25 -07:00
Rishi Gupta 7741e099fc notifications.py: Merge send_local_email_template_with_delay into callers.
Note that the correctness of this commit relies on the fact that
send_future_email also sets the sender to settings.NOREPLY_EMAIL_ADDRESS by
default (in the body of the function).
2017-05-05 14:20:32 -07:00
Rishi Gupta 26ac344b19 email: Move and rename invitation_reminder_email templates. 2017-05-03 23:26:14 -07:00
vaibhav 8881b5eb9f Outgoing Webhook System: Check for @-mentioned outgoing webhook bots.
Also puts them into a processing queue, though the queue processor
does nothing.

Rewritten by tabbott to avoid unnecessary database queries in
do_send_messages.
2017-05-02 09:22:04 -07:00
Umair Khan be65125d3d Handle unicode characters in email mirror.
Fixes #2328.
2017-04-05 12:18:41 -07:00
K.Kanakhin 6a801db1c2 missed-emails-sending: Move email sending to separate queue worker.
- Add new 'missedmessage_email_senders' queue for sending missed messages emails.
- Add the new worker to process 'missedmessage_email_senders' queue.
- Split aggregation missed messages and sending missed messages email
  to separate queue workers.
- Adapt tests for sending missed emails to the new logic.

Fixes #2607
2017-03-07 20:08:40 -08:00
Tim Abbott 6cbd0fcafa feedback: Move feedback logic out of queue_processors.py.
This will make it easier to unit test this logic in the future.
2017-03-06 00:01:57 -08:00
Tim Abbott e1c1f96f9e zilencer: Eliminate submit_feedback indirection. 2017-03-06 00:01:57 -08:00
Tim Abbott 9a08ef5545 zilencer: Remove FEEDBACK_TARGET forwarding logic.
This feature hardcoded zulip.com, and never really made much sense
("feedback" should generally go to the local server administrator, not
to the Zulip development community).
2017-03-06 00:01:57 -08:00
Tim Abbott b1b2d8d2c7 push: Move handle_push_notifications to push_notifications.py. 2017-03-05 18:32:09 -08:00
Raghav Jajodia a3a03bd6a5 mypy: Added Dict, List and Set imports.
Fixed mypy errors associated with the upgrade.
2017-03-04 14:33:44 -08:00
Ayush Jain 455c1919fc Add customizable invite-new-user text.
This makes life a lot easier for people inviting users to a new Zulip
organization, since they can give some form of context now.

Modified by tabbott to clean up CSS, backend code flow, and improve
the formatting of the emails.

Fixes: #1409.
2017-02-21 22:35:01 -08:00
Tim Abbott 8da7976058 queues: Add new system for managing rabbitmq per-queue work.
Our lists of rabbitmq queues was likely to end up out of date, since
there was nothing enforcing that the various lists of queues were
correct or the same as each other.
2017-02-19 16:18:37 -08:00
Tim Abbott 6ad883fb61 docs: Update queuing documentation for new templates. 2017-02-19 13:12:34 -08:00
Tim Abbott 9072d3c350 socket: Transmit HTTP_USER_AGENT for websockets sending.
This significantly simplify the logic for our logging process, making
it the case that websockets message sending requests always are logged
as having the exact same client as a normal AJAX request from that
server.
2017-02-17 15:19:21 -08:00
Tim Abbott 5c911c44d6 MessageSenderWorker: Reformat environment setup. 2017-02-17 15:00:57 -08:00
Tim Abbott e69c4458c6 errors: Use hostname to report deployment. 2017-01-28 13:07:09 -08:00
Tim Abbott ae35bde3a6 errors: Enable error reports without zilencer enabled. 2017-01-27 10:27:41 -08:00
Tim Abbott 04ae2977fa errors: Add logging for consuming processed error reports.
This should make it a big easier to debug issues with this system.
2017-01-27 10:27:41 -08:00
Tim Abbott dd9e0b8463 errors: Move do_report_error into zerver/lib/. 2017-01-23 23:18:28 -08:00
Tim Abbott d6e38e2a5c lint: Clean up E123 PEP-8 rule. 2017-01-23 21:34:26 -08:00
Tim Abbott 01e0acb282 render_incoming_message: Require a realm object. 2017-01-21 21:37:57 -08:00
Tim Abbott 99c5563bc6 internal_send_message: Make realm argument mandatory.
A lot of care has been taken to ensure we're using the realm that the
message is being sent into, not the realm of the sender, to correctly
handle the logic for cross-realm bot users such as the notifications
bot.
2017-01-21 21:37:30 -08:00
Tim Abbott 46fc136943 feedback-bot: Add logging for successfully submitted feedback. 2017-01-06 21:42:16 -08:00
Rishi Gupta 13d1a636d5 queue_processors.SignupWorker: Upgrade to MailChimp API v3.0.
Removes the dependence on postmonkey, which is a wrapper around
MailChimp API v1.3. MailChimp recommends using their v3.0 API directly,
rather than through a wrapper library.
2016-12-29 16:16:18 -08:00
Rishi Gupta c61938f719 SignupWorker: Use dictionary indexing instead of get for name and email.
We always pass in EMAIL in data, and NAME in merge_vars.
2016-12-29 16:16:18 -08:00
Steve Howell 3d804de962 Fix FeedbackBot queue processor.
Recent changes to the API removed Client.register(), and
this change restores the correct API call, although the
codepath this affects is probably ready for eventual
deprecation.
2016-12-28 12:12:34 -08:00
Igor Tokarev c93f1d4eda Add oembed/Open Graph/Meta tags data retrieval from inline links.
This change adds support for displaying inline open graph previews for
links posted into Zulip.

It is designed to interact correctly with message editing.

This adds the new settings.INLINE_URL_EMBED_PREVIEW setting to control
whether this feature is enabled.

By default, this setting is currently disabled, so that we can burn it
in for a bit before it impacts users more broadly.

Eventually, we may want to make this manageable via a (set of?)
per-realm settings.  E.g. I can imagine a realm wanting to be able to
enable/disable it for certain URLs.
2016-12-07 17:40:18 -08:00
AZtheAsian 1ba150fa85 pep8: Fix E203 violations 2016-12-01 20:37:57 -08:00
Rafid Aslam 7a2282986a pep8: Fix E225 pep8 violations. 2016-11-28 15:21:15 -08:00
Tim Abbott 282f74609c tornado: Move socket code to zerver/tornado/. 2016-11-26 22:29:27 -08:00
paxapy 6d93b3b60c lint: Fix E703 pep8 violations. 2016-11-09 15:18:35 -08:00
paxapy ff1e97603d context_processors: use a common context for emails.
Fixes #1611.
2016-11-08 11:37:16 -08:00
Tim Abbott be3e9e3a05 Fix regression breaking websocket message sending.
In fe1ba6f3eb, we change our auth
decorators etc. to use request.POST/request.GET rather than the (now
removed request.REQUEST).  This broke sending messages via our
websockets codepath, because we failed to update the artificial
requests we generate to use request._post as well.
2016-11-05 11:33:04 -07:00
Tim Abbott 1a88ca4da1 queue_processors: Fix feedback bot sender.
This fixes an issue where emails sent by feedback-bot had a forged
sender and thus sometimes had delivery issues.
2016-10-28 11:04:08 -07:00
Tim Abbott 65321b77ac socket: Switch localhost to 127.0.0.1.
It does the same thing, but is a bit more robust to weird DNS
configurations.
2016-09-13 10:01:57 -07:00
Rishi Gupta 64179b2fd2 confirmation: Use realm host in activation URLs.
This is preparation for merging the subdomains feature.
2016-08-18 20:12:36 -07:00
Tim Abbott 4fbb8c3eee templates: Add new context variables to all templates.
This adds a few new helpful context variables that we can use to
compute URLs in all of our templates:
* external_uri_scheme: http(s)://
* server_uri: The base URL for the server's canonical name
* realm_uri: The base URL for the user's realm

This is preparatory work for making realm_uri != server_uri when we
add support for subdomains.
2016-08-13 16:27:35 -07:00
Tim Abbott 95aee366c8 Enable local browser error reporting if ZILENCER_ENABLED. 2016-08-11 13:49:18 -07:00
Tim Abbott 88550edd9e Annotate queue_processors.py. 2016-08-04 15:57:03 -07:00
Eklavya Sharma b98b049377 Fix string handling in email_mirror queue_processor. 2016-07-28 14:52:09 -07:00
Tim Abbott 3ea4cbb5c3 settings: Extract settings.VERBOSE_SUPPORT_OFFERS. 2016-07-19 15:28:41 -07:00
Eklavya Sharma 2080ff6c2a Decode response.content everywhere except in tests.
response.content is binary data, but code usually assumes it to
be text.  Fix this by decoding response.content where required.
Don't do this in tests yet.
2016-07-12 14:06:29 -07:00
Eklavya Sharma 1f3ce7cf38 zerver/worker/queue_processors.py: Open file in binary.
Open a file in binary mode instead of text mode.
2016-07-09 17:54:54 -07:00
Varshit 4e1060076d Purge 'from typing import *' from zerver/.
This is a partial implementation of #636.
2016-04-07 14:07:07 -07:00
Tim Abbott ee8be22160 Add PEP-484 type annotations to zerver/worker/. 2016-04-03 15:40:23 -07:00
Tim Abbott 2059f650ab Add PEP-484 type annotations to zerver/lib/. 2016-04-03 15:40:23 -07:00