Commit Graph

76 Commits

Author SHA1 Message Date
Tim Abbott 3ea4cbb5c3 settings: Extract settings.VERBOSE_SUPPORT_OFFERS. 2016-07-19 15:28:41 -07:00
Eklavya Sharma 2080ff6c2a Decode response.content everywhere except in tests.
response.content is binary data, but code usually assumes it to
be text.  Fix this by decoding response.content where required.
Don't do this in tests yet.
2016-07-12 14:06:29 -07:00
Eklavya Sharma 1f3ce7cf38 zerver/worker/queue_processors.py: Open file in binary.
Open a file in binary mode instead of text mode.
2016-07-09 17:54:54 -07:00
Varshit 4e1060076d Purge 'from typing import *' from zerver/.
This is a partial implementation of #636.
2016-04-07 14:07:07 -07:00
Tim Abbott ee8be22160 Add PEP-484 type annotations to zerver/worker/. 2016-04-03 15:40:23 -07:00
Tim Abbott 2059f650ab Add PEP-484 type annotations to zerver/lib/. 2016-04-03 15:40:23 -07:00
Tim Abbott cd2348e9ae Run queue processers multithreaded in development.
This change drops the memory used for Python processes run by Zulip in
development from about 1GB to 300MB on my laptop.

On the front of safety, http://pika.readthedocs.org/en/latest/faq.html
explains "Pika does not have any notion of threading in the code. If
you want to use Pika with threading, make sure you have a Pika
connection per thread, created in that thread. It is not safe to share
one Pika connection across threads.".  Since this code only connects
to rabbitmq inside the individual threads, I believe this should be
safe.

Progress towards #32.
2016-03-20 18:04:24 -07:00
Tim Abbott a46647a87a python3: Mark certain strings as unicode strings.
This is required in Python 3 since these strings are combined with
other unicode strings.
2016-03-08 09:14:11 -08:00
Tim Abbott 1d008576f2 QueueProcessingWorker: Add stub consume function and queue_name.
QueueProcessingWorker will now throw errors if there's a misconfigured
queue processor.
2016-02-03 19:47:14 -08:00
Tim Abbott 1ee0706511 Fix missing UTF-8 encoding. 2016-02-03 19:29:30 -08:00
Tim Abbott 5bacda3662 python3: Fix usage of .keys()/.values() to handle iterators.
This fixes the places where we use the result of .keys(), .items(),
and .values() that wouldn't work with an iterator to wrap them with
list().
2016-01-26 21:11:25 -08:00
Tim Abbott 757e89260e Migrate use of StringIO to Python 2+3 compatible six.moves.cStringIO.
And add a check for additional usage of the original StringIO module.
2016-01-26 21:09:43 -08:00
Allie Jones 1bd1291f3c Replace json/send_message endpoint usage with json/message. 2015-12-12 18:14:08 -08:00
Tim Abbott be112d2c9d SlowQueryWorker: Don't reference potentially unset STATSD_PREFIX. 2015-11-16 21:10:06 -08:00
Tim Abbott 8c34c40924 Apply Python 3 futurize transform lib2to3.fixes.fix_except. 2015-11-01 08:08:33 -08:00
Tim Abbott 1156a82297 Make ErrorReporter run successfully on Zulip voyager.
(imported from commit 8ab2fb57fb3e580775cb8959e3b8b5a1277ea9d3)
2015-09-25 04:09:16 -07:00
Tim Abbott ae4cb6df35 Make python-postmonkey not a required package.
This is in some ways a regression, but because we don't have
python-postmonkey packaged right now, this is required to make the
Zulip production installation process work on Trusty.

(imported from commit 539d253eb7fedc20bf02cc1f0674e9345beebf48)
2015-09-25 00:22:14 -07:00
David Roe 3f7cb34b00 enterprise => voyager
(imported from commit 04be792bb480d5e5db1c91d296d1000cf1682571)
2015-08-21 10:33:35 -07:00
David Roe edf7e732a2 ENTERPRISE => VOYAGER.
(imported from commit 4f8080b9f506a87ca40bef32e39de5218cba916a)
2015-08-21 10:33:35 -07:00
David Roe d702ddb3cf DEPLOYED => PRODUCTION
(imported from commit a7d365c07916c5103fc721c712db94f9af977640)
2015-08-21 10:33:35 -07:00
Tim Abbott 3e61714035 Make DEPLOYMENT_ROLE_KEY no longer required.
(imported from commit 046ce610d847cc987d3f6d584ccc0bc84b55df84)
2015-08-21 09:01:58 -07:00
Jason Michalski 2a5ec20295 Add support for one time use email address
The one time use address are a unique token which maps to stored stated
in redis. We store the user_id, recipient_id, and subject. When an email
is received at this address it is sent to the stored recipient by the
stored user. Anyone with this address can send a single message as this
user.

(imported from commit 4219417bdc30c033a6cf7a0c7c0939f7d0308144)
2014-10-06 15:27:54 -07:00
Zev Benjamin 6691789cce Add helper function for returning a Redis client
(imported from commit 47f87d388d24343ac6b631181a55287eb8cd4a6d)
2014-02-10 13:23:28 -05:00
Luke Faraone 179be2051d email-mirror: Encode messages as utf-8 before parsing
(imported from commit 7b896b1e24d382f024ab0a487d448d20448f0cca)
2014-01-31 15:33:15 -05:00
Tim Abbott da90d63046 Split out zerver/lib/notifications.py from actions.py.
(imported from commit 784b82834ee4fcb4431e77f8fb1c526f8eec82ad)
2014-01-24 17:33:56 -05:00
Leo Franchi 56d9446c41 [socket] Use socket req ids that contain queue ids
(imported from commit 781eafe60bf117d79ce3a30deea02ec1e875060a)
2014-01-23 16:28:58 -05:00
Luke Faraone fbb2b6945b Queue worker for email mirror processing.
(imported from commit b60049c84622ea457db38c5a5b185bcd108bc59f)
2014-01-15 16:07:18 -05:00
Zev Benjamin db23674749 Do query time tracking at the psycopg2 level instead of the Django level
This allows us to track the query time of SQLAlchemy and raw queries.

(imported from commit 818a4ee41786ffc57b80d7ed1cfba075f29b6ee5)
2014-01-14 11:47:12 -05:00
Luke Faraone 1b5c1ac021 Update style of client strings.
(imported from commit 1516461cf53b2715de68e01f16bb8a8cc33c48ad)
2013-12-09 11:47:52 -05:00
Tim Abbott f7ac58bab5 Revert "Add user_activity_test worker that does nothing."
Now that we've debugged the memory leak, I don't think we need this
anymore.

This reverts commit 1bdc7ee2f72bdebb1cdc94601247834a434614d6.

Conflicts:

	puppet/zulip/files/cron.d/rabbitmq-numconsumers
	puppet/zulip/files/supervisor/conf.d/zulip.conf

(imported from commit ff87f2aebcbc71013fa7a05aedb24e2dcad82ae6)
2013-12-04 10:27:45 -05:00
acrefoot 8ee7536012 Send push notifications more aggresively
Unbundle the push notifications from the missed message queue processors
and handlers. This makes notifications more immediate, and sets things up
for better badge count handling, and possibly per-stream filtering.

(imported from commit 11840301751b0bbcb3a99848ff9868d9023b665b)
2013-11-27 18:00:31 -05:00
Jessica McKellar 2ba9415178 Factor welcome e-mail enqueuing into its own function.
(imported from commit 85054ee3e567d2ba7a4ed94c52a311d02f5e4d43)
2013-11-27 09:30:00 -05:00
Jessica McKellar 8e72084fcb Add one-click unsubscribe links to day 1 and day 2 emails.
(imported from commit 9396f96365430baf894c708f9eaea642fc648435)
2013-11-27 09:29:59 -05:00
Tim Abbott f59fb53662 enterprise: Add option to forward feedback to an email address.
(imported from commit 0ac9bb0427aeba60a8004953fedfdfdc6bbb587c)
2013-11-25 16:59:36 -05:00
Tim Abbott c0e951f843 Add user_activity_test worker that does nothing.
This should help us debug the source of our memory leak problems.

(imported from commit 1bdc7ee2f72bdebb1cdc94601247834a434614d6)
2013-11-22 11:24:48 -05:00
Tim Abbott 9ba820703a Call reset_queries in our queue workers.
This fixes a small memory leak in our queue workers, where we don't
reset the accumulated times contained in our query logging data.

Longer-term, we may want to make something mergable for mainline where
we only store on the connection object the totals; that would be a
fixed amount of emmory per connection and thus not have this problem.

(imported from commit 914fa13acfb576f73c5f35e0f64c2f4d8a56b111)
2013-11-22 11:24:48 -05:00
acrefoot bafb6f8dc3 fixup day1/day2 email templates
(imported from commit 30846fc298d04ac763025e82a60731d158840462)
2013-11-15 21:51:37 -05:00
acrefoot e77d534e78 remove ALLOW_ARBITRARY_SENDERS_LOCAL_EMAIL
(imported from commit 7a374959516d8505685466293f146d95094a7ea7)
2013-11-15 21:31:37 -05:00
acrefoot daffe049ad Don't use hardcoded noreply@zulip.com, zulip@zulip.com, or https://zulip.com
(imported from commit 1132553b63ae23ebcca746f0f65205b97bfee7dc)
2013-11-15 21:31:37 -05:00
acrefoot f991fbff9e Fix clear_followup_emails_queue on Zulip Enterprise.
(imported from commit bf211b205bc39a08340306d06fbb76479a02ed9e)
2013-11-14 11:32:29 -05:00
acrefoot f5199081d5 Add the deliver_email management command
This command should be run continuously via supervisor. It periodically
checks for new email messages to send, and then sends them. This is for
sending email that you've queued via the Email table, instead of mandrill
(as is the case for our localserver/development deploys).

(imported from commit a2295e97b70a54ba99d145d79333ec76b050b291)
2013-11-13 16:41:36 -05:00
Kevin Mehall f7f2ec0aca [puppet] Report enterprise and prod errors to staging.
Errors are sent to a queue processor that posts them to staging,
just like the feedback bot.

(imported from commit 4a8d099672a1b3e48a8bc94148d8b53db73d2c64)
2013-11-13 16:22:21 -05:00
Steve Howell 0fe751425a Always enable SlowQueryWorker.
(imported from commit 05aa98d60c34e95df0b41fe9d25d909fea547ddc)
2013-11-12 20:19:42 -05:00
Steve Howell e93215693a Always drain the slow_queries queue.
(imported from commit e11c25116529fb402d82d349a67fb1dfc40f510c)
2013-11-12 20:19:42 -05:00
Steve Howell cf79e77347 Extract process_one_batch() in SlowQueryWorker.
By extracting the function, we eliminate a redundant sleep
call.

(imported from commit 24d4485019f96ae10e9a00244643b77ff2a7db57)
2013-11-12 20:10:23 -05:00
Zev Benjamin eec195f421 socket: Use our full logging infrastructure
We also now separate out the times for the socket overhead, the
request service time, and the queuing delays.

(imported from commit e1683f7f28b968b86ebb701b0ac29b00ac6d67c3)
2013-11-12 15:24:30 -05:00
Zev Benjamin 24ee4a5d57 socket: Build a real Request object to send through our full stack, including middleware
One quirk here is that the Request object is built in the
message_sender worker, not Tornado.  This means that the request time
only counts time taken for the actual sending and does not account
for socket overhead.  For this reason, I've left the fake logging in
for now so we can compare the two times.

(imported from commit b0c60a3017527a328cadf11ba68166e59cf23ddf)
2013-11-12 15:24:29 -05:00
Zev Benjamin 48a25211fa socket: Record the request status in redis
(imported from commit 6c3f6dc37d1abdcacf105f865ac24483717438ca)
2013-11-07 11:43:34 -05:00
Zev Benjamin 87fc182645 socket: Move `req_id` parameter to top level of protocol payload
(imported from commit 271d84a0b44a2623bec290f64a83a0423698a1ef)
2013-11-07 11:43:34 -05:00
Tim Abbott 68dcc760c3 Clean up some unused imports.
(imported from commit 0c5d8e2a55ba1b8909ba807fee3afe863dcdc226)
2013-11-04 11:51:17 -05:00