Commit Graph

63 Commits

Author SHA1 Message Date
Tim Abbott be112d2c9d SlowQueryWorker: Don't reference potentially unset STATSD_PREFIX. 2015-11-16 21:10:06 -08:00
Tim Abbott 8c34c40924 Apply Python 3 futurize transform lib2to3.fixes.fix_except. 2015-11-01 08:08:33 -08:00
Tim Abbott 1156a82297 Make ErrorReporter run successfully on Zulip voyager.
(imported from commit 8ab2fb57fb3e580775cb8959e3b8b5a1277ea9d3)
2015-09-25 04:09:16 -07:00
Tim Abbott ae4cb6df35 Make python-postmonkey not a required package.
This is in some ways a regression, but because we don't have
python-postmonkey packaged right now, this is required to make the
Zulip production installation process work on Trusty.

(imported from commit 539d253eb7fedc20bf02cc1f0674e9345beebf48)
2015-09-25 00:22:14 -07:00
David Roe 3f7cb34b00 enterprise => voyager
(imported from commit 04be792bb480d5e5db1c91d296d1000cf1682571)
2015-08-21 10:33:35 -07:00
David Roe edf7e732a2 ENTERPRISE => VOYAGER.
(imported from commit 4f8080b9f506a87ca40bef32e39de5218cba916a)
2015-08-21 10:33:35 -07:00
David Roe d702ddb3cf DEPLOYED => PRODUCTION
(imported from commit a7d365c07916c5103fc721c712db94f9af977640)
2015-08-21 10:33:35 -07:00
Tim Abbott 3e61714035 Make DEPLOYMENT_ROLE_KEY no longer required.
(imported from commit 046ce610d847cc987d3f6d584ccc0bc84b55df84)
2015-08-21 09:01:58 -07:00
Jason Michalski 2a5ec20295 Add support for one time use email address
The one time use address are a unique token which maps to stored stated
in redis. We store the user_id, recipient_id, and subject. When an email
is received at this address it is sent to the stored recipient by the
stored user. Anyone with this address can send a single message as this
user.

(imported from commit 4219417bdc30c033a6cf7a0c7c0939f7d0308144)
2014-10-06 15:27:54 -07:00
Zev Benjamin 6691789cce Add helper function for returning a Redis client
(imported from commit 47f87d388d24343ac6b631181a55287eb8cd4a6d)
2014-02-10 13:23:28 -05:00
Luke Faraone 179be2051d email-mirror: Encode messages as utf-8 before parsing
(imported from commit 7b896b1e24d382f024ab0a487d448d20448f0cca)
2014-01-31 15:33:15 -05:00
Tim Abbott da90d63046 Split out zerver/lib/notifications.py from actions.py.
(imported from commit 784b82834ee4fcb4431e77f8fb1c526f8eec82ad)
2014-01-24 17:33:56 -05:00
Leo Franchi 56d9446c41 [socket] Use socket req ids that contain queue ids
(imported from commit 781eafe60bf117d79ce3a30deea02ec1e875060a)
2014-01-23 16:28:58 -05:00
Luke Faraone fbb2b6945b Queue worker for email mirror processing.
(imported from commit b60049c84622ea457db38c5a5b185bcd108bc59f)
2014-01-15 16:07:18 -05:00
Zev Benjamin db23674749 Do query time tracking at the psycopg2 level instead of the Django level
This allows us to track the query time of SQLAlchemy and raw queries.

(imported from commit 818a4ee41786ffc57b80d7ed1cfba075f29b6ee5)
2014-01-14 11:47:12 -05:00
Luke Faraone 1b5c1ac021 Update style of client strings.
(imported from commit 1516461cf53b2715de68e01f16bb8a8cc33c48ad)
2013-12-09 11:47:52 -05:00
Tim Abbott f7ac58bab5 Revert "Add user_activity_test worker that does nothing."
Now that we've debugged the memory leak, I don't think we need this
anymore.

This reverts commit 1bdc7ee2f72bdebb1cdc94601247834a434614d6.

Conflicts:

	puppet/zulip/files/cron.d/rabbitmq-numconsumers
	puppet/zulip/files/supervisor/conf.d/zulip.conf

(imported from commit ff87f2aebcbc71013fa7a05aedb24e2dcad82ae6)
2013-12-04 10:27:45 -05:00
acrefoot 8ee7536012 Send push notifications more aggresively
Unbundle the push notifications from the missed message queue processors
and handlers. This makes notifications more immediate, and sets things up
for better badge count handling, and possibly per-stream filtering.

(imported from commit 11840301751b0bbcb3a99848ff9868d9023b665b)
2013-11-27 18:00:31 -05:00
Jessica McKellar 2ba9415178 Factor welcome e-mail enqueuing into its own function.
(imported from commit 85054ee3e567d2ba7a4ed94c52a311d02f5e4d43)
2013-11-27 09:30:00 -05:00
Jessica McKellar 8e72084fcb Add one-click unsubscribe links to day 1 and day 2 emails.
(imported from commit 9396f96365430baf894c708f9eaea642fc648435)
2013-11-27 09:29:59 -05:00
Tim Abbott f59fb53662 enterprise: Add option to forward feedback to an email address.
(imported from commit 0ac9bb0427aeba60a8004953fedfdfdc6bbb587c)
2013-11-25 16:59:36 -05:00
Tim Abbott c0e951f843 Add user_activity_test worker that does nothing.
This should help us debug the source of our memory leak problems.

(imported from commit 1bdc7ee2f72bdebb1cdc94601247834a434614d6)
2013-11-22 11:24:48 -05:00
Tim Abbott 9ba820703a Call reset_queries in our queue workers.
This fixes a small memory leak in our queue workers, where we don't
reset the accumulated times contained in our query logging data.

Longer-term, we may want to make something mergable for mainline where
we only store on the connection object the totals; that would be a
fixed amount of emmory per connection and thus not have this problem.

(imported from commit 914fa13acfb576f73c5f35e0f64c2f4d8a56b111)
2013-11-22 11:24:48 -05:00
acrefoot bafb6f8dc3 fixup day1/day2 email templates
(imported from commit 30846fc298d04ac763025e82a60731d158840462)
2013-11-15 21:51:37 -05:00
acrefoot e77d534e78 remove ALLOW_ARBITRARY_SENDERS_LOCAL_EMAIL
(imported from commit 7a374959516d8505685466293f146d95094a7ea7)
2013-11-15 21:31:37 -05:00
acrefoot daffe049ad Don't use hardcoded noreply@zulip.com, zulip@zulip.com, or https://zulip.com
(imported from commit 1132553b63ae23ebcca746f0f65205b97bfee7dc)
2013-11-15 21:31:37 -05:00
acrefoot f991fbff9e Fix clear_followup_emails_queue on Zulip Enterprise.
(imported from commit bf211b205bc39a08340306d06fbb76479a02ed9e)
2013-11-14 11:32:29 -05:00
acrefoot f5199081d5 Add the deliver_email management command
This command should be run continuously via supervisor. It periodically
checks for new email messages to send, and then sends them. This is for
sending email that you've queued via the Email table, instead of mandrill
(as is the case for our localserver/development deploys).

(imported from commit a2295e97b70a54ba99d145d79333ec76b050b291)
2013-11-13 16:41:36 -05:00
Kevin Mehall f7f2ec0aca [puppet] Report enterprise and prod errors to staging.
Errors are sent to a queue processor that posts them to staging,
just like the feedback bot.

(imported from commit 4a8d099672a1b3e48a8bc94148d8b53db73d2c64)
2013-11-13 16:22:21 -05:00
Steve Howell 0fe751425a Always enable SlowQueryWorker.
(imported from commit 05aa98d60c34e95df0b41fe9d25d909fea547ddc)
2013-11-12 20:19:42 -05:00
Steve Howell e93215693a Always drain the slow_queries queue.
(imported from commit e11c25116529fb402d82d349a67fb1dfc40f510c)
2013-11-12 20:19:42 -05:00
Steve Howell cf79e77347 Extract process_one_batch() in SlowQueryWorker.
By extracting the function, we eliminate a redundant sleep
call.

(imported from commit 24d4485019f96ae10e9a00244643b77ff2a7db57)
2013-11-12 20:10:23 -05:00
Zev Benjamin eec195f421 socket: Use our full logging infrastructure
We also now separate out the times for the socket overhead, the
request service time, and the queuing delays.

(imported from commit e1683f7f28b968b86ebb701b0ac29b00ac6d67c3)
2013-11-12 15:24:30 -05:00
Zev Benjamin 24ee4a5d57 socket: Build a real Request object to send through our full stack, including middleware
One quirk here is that the Request object is built in the
message_sender worker, not Tornado.  This means that the request time
only counts time taken for the actual sending and does not account
for socket overhead.  For this reason, I've left the fake logging in
for now so we can compare the two times.

(imported from commit b0c60a3017527a328cadf11ba68166e59cf23ddf)
2013-11-12 15:24:29 -05:00
Zev Benjamin 48a25211fa socket: Record the request status in redis
(imported from commit 6c3f6dc37d1abdcacf105f865ac24483717438ca)
2013-11-07 11:43:34 -05:00
Zev Benjamin 87fc182645 socket: Move `req_id` parameter to top level of protocol payload
(imported from commit 271d84a0b44a2623bec290f64a83a0423698a1ef)
2013-11-07 11:43:34 -05:00
Tim Abbott 68dcc760c3 Clean up some unused imports.
(imported from commit 0c5d8e2a55ba1b8909ba807fee3afe863dcdc226)
2013-11-04 11:51:17 -05:00
Steve Howell 7a67fb34b0 Rename: queue_followup_emails -> enqueue_followup_emails.
(imported from commit c44e790de5b694f8d98b03608231f7a284856254)
2013-11-04 10:20:59 -05:00
Steve Howell c4141eeeb0 Rename: queue_digest_emails -> enqueue_digest_emails.
(imported from commit a72fdf849814ff974117c55bf1b155c494a1b160)
2013-11-04 10:20:59 -05:00
Tim Abbott 29aef33c53 Remove transaction management hacks in workers.
(imported from commit 33b9349bd139907755b52474df8a0da13b73b80d)
2013-11-04 07:31:39 -05:00
Leo Franchi db6550e99a Make built-in bots come from settings.py, and allow localserver-specific ones
(imported from commit e21933e37487314ac986147562817a19227e8960)
2013-11-01 14:13:05 -04:00
Steve Howell 70c1c31b3a Simplify callback protocol for register_json_consumer().
The register_json_consumer() function now expects its callback
function to accept a single argument, which is the payload, as
none of the callbacks cared about channel, method, and properties.

This change breaks down as follows:

    * A couple test stubs and subclasses were simplified.
    * All the consume() and consume_wrapper() functions in
      queue_processors.py were simplified.
    * Two callbacks via runtornado.py were simplified.  One
      of the callbacks was socket.respond_send_message, which
      had an additional caller, i.e. not register_json_consumer()
      calling back to it, and the caller was simplified not
      to pass None for the three removed arguments.

(imported from commit 792316e20be619458dd5036745233f37e6ffcf43)
2013-10-31 13:18:11 -04:00
Steve Howell d7d1888ce0 Add TestWorker queue processor.
TestWorker is for troubleshooting, and it simply appends lines to a file
in /tmp.

(imported from commit 96b646a193a5474f3222b41725742e359b5301ec)
2013-10-30 16:47:16 -04:00
Steve Howell 884e602185 [puppet] Handle exceptions from queue workers.
Subclasses of QueueProcessingWorker that don't override start() will
have their consume() functions wrapped by consume_wrapper(), which
will catch exceptions and log data from troublesome events to a log
file.

We need to do a puppet apply to create /var/log/zulip/queue_error.

(imported from commit 3bd7751da5fdef449eeec3f7dd29977df11e2b9c)
2013-10-29 17:45:31 -04:00
Steve Howell 0b335a7b73 Simplify DigestWorker.
The DigestWorker isn't do any batching of events, so it can
be implemented with a simple consume() handler.

(imported from commit 0560eaf1a6e510adf97448fbe8933d9903d30016)
2013-10-29 11:39:45 -04:00
Leo Franchi a33640c7dd Sleep digest queue worker for 1min between attempts to drain queue
(imported from commit e272dcd33bc89e6813abb0b4dc2c882a8048245e)
2013-10-28 15:47:04 -04:00
Zev Benjamin f241897c5b Add function to get the set of active queue processors
(imported from commit 1b9e52cfaf00827305a8f790091860436739d11c)
2013-10-28 14:30:53 -04:00
Zev Benjamin 822edaa890 Condition the registration of some queues on being deployed
This prevents the ones with external side-effects (like sending real
email) from being accidentally run in dev instances.

(imported from commit 6d9861d721abb29136bfff974de01a9264051436)
2013-10-28 14:30:53 -04:00
Steve Howell 13768495a6 Avoid long locks in process missedmessage_emails queue.
(imported from commit aed4a89244dce19085ed2ad89a7539ab297d8bc9)
2013-10-25 20:45:19 -04:00
Steve Howell 3cbb9f6b08 Avoid long locks in process digest_emails queue.
(imported from commit 49c7441463c5e63e52e5b099bd854bd4ae769f13)
2013-10-25 20:34:25 -04:00