Commit Graph

135 Commits

Author SHA1 Message Date
Leo Franchi ca8187b8cf Email gateway bot variable name changed to EMAIL_GATEWAY_BOT
(imported from commit 01c430bd9b916c9bb152ee821214495333f1d37e)
2013-11-01 16:46:17 -04:00
Leo Franchi db6550e99a Make built-in bots come from settings.py, and allow localserver-specific ones
(imported from commit e21933e37487314ac986147562817a19227e8960)
2013-11-01 14:13:05 -04:00
Tim Abbott ddbfe31449 Remove obsolete management command for mit.edufullname transition.
(imported from commit aed2a2d7bfedd98e2406bb18bf54f953a5ae7ef3)
2013-11-01 10:30:17 -04:00
Tim Abbott 11b92534e8 Remove obsolete management command for unread counts transition.
(imported from commit 52c6acbf4e8e53f86e880e475396d824c729405f)
2013-11-01 10:30:17 -04:00
Tim Abbott 429c41da79 Rename client-activity to client_activity to match other management commands.
(imported from commit b0b2cd372810923b3c808142f7b7eeef285d132d)
2013-10-31 15:29:50 -04:00
Tim Abbott dbd4c187e3 Rename create-stream to create_stream to match other management commands.
(imported from commit 28daad12943ab5eb245d7737934b08f5b29e8f90)
2013-10-31 15:29:50 -04:00
Tim Abbott 5b1e504e13 Rename rename-stream to rename_stream to match other management commands.
(imported from commit 28522ac669a38e3de3f25b28c82ab0c787aecd0b)
2013-10-31 15:29:50 -04:00
Tim Abbott 013e84ef86 Remove obsolete explain_js_error management command.
(imported from commit 92f23e3bb7e6dbbad5162446b5f19e74728cf608)
2013-10-31 15:29:50 -04:00
Tim Abbott 217478e988 Remove obsolete clear_db management command.
do_destroy_rebuild_database is the "new" way to reset one's dev
database.

(imported from commit 8edb707ecd5e6e24b62eef324915cc3a4a5b8eda)
2013-10-31 15:29:50 -04:00
Tim Abbott 384f64d38d Remove obsolete management commands from various field additions.
(imported from commit b44a2b683768c790839e82e2e9c9a18827670113)
2013-10-31 15:29:50 -04:00
Tim Abbott f0580dd28c Remove obsolete management command for activating MIT users.
(imported from commit a965eea8edfc9d73a00770d85cd517343f607338)
2013-10-31 15:29:50 -04:00
Tim Abbott 06d43a6f97 Remove obsolete management commands from the stream color transition.
(imported from commit d6d508d952714a1865ca4fddcb48d0ccedb6f4dd)
2013-10-31 15:29:50 -04:00
Tim Abbott 6d8fc371fb Remove obsolete command to delete tutorial streams.
We don't have tutorial-bot anymore, so this is no longer needed.

(imported from commit b8a1bc9c2379a451e8cc1ed43b15f373bb3f2b06)
2013-10-31 15:29:50 -04:00
Tim Abbott c8d07cc64c Remove management commands from the mysql=>postgres migration.
(imported from commit 636f71b435a50e83fd8a9e40685d7c52b3b01f30)
2013-10-31 15:29:49 -04:00
Waseem Daher 1d784801cf More settings.LOCALSERVER -> settings.LOCAL_SERVER.
(imported from commit 2b5045d302b8ebb2a5c7175264b4ab9cfef2844d)
2013-10-31 13:31:03 -04:00
Steve Howell 70c1c31b3a Simplify callback protocol for register_json_consumer().
The register_json_consumer() function now expects its callback
function to accept a single argument, which is the payload, as
none of the callbacks cared about channel, method, and properties.

This change breaks down as follows:

    * A couple test stubs and subclasses were simplified.
    * All the consume() and consume_wrapper() functions in
      queue_processors.py were simplified.
    * Two callbacks via runtornado.py were simplified.  One
      of the callbacks was socket.respond_send_message, which
      had an additional caller, i.e. not register_json_consumer()
      calling back to it, and the caller was simplified not
      to pass None for the three removed arguments.

(imported from commit 792316e20be619458dd5036745233f37e6ffcf43)
2013-10-31 13:18:11 -04:00
Steve Howell ddc7608afe Add enqueue_file management command.
This reads lines from a file and pushes them on to one of our
worker queues.

(imported from commit 2ed3338c5065819f8bafb247a15799e2bc9e7d39)
2013-10-30 16:47:16 -04:00
Luke Faraone ab9b1be82c Rework management commands to handle deployment assignment
We now ensure `create_realm` adds you to a default deployment and that
`create_deployment` removes the old deployment association when
performed.

(imported from commit 5b94fb07b8e11332765b057dc640a5ed873ec99e)
2013-10-30 12:00:13 -04:00
Tim Abbott 901a5c342c Use os.path directly rather than sometimes importing it separately.
(imported from commit 48486c4ea64d02a15faeebb0f490d31e9b113d62)
2013-10-28 15:44:13 -04:00
Zev Benjamin 807116e955 Delay connecting to RabbitMQ until it's necessary
Previously we were connecting at import-time.

(imported from commit 9a638f0d238f3b6b00feb4aa524098a64953cb92)
2013-10-28 15:27:25 -04:00
Zev Benjamin ffb6266319 Refuse to run a queue processor if USING_RABBITMQ is False
(imported from commit 39beff47cdbb18ba39756989e6f07facbd16864f)
2013-10-28 14:30:53 -04:00
Zev Benjamin 8aa2c7b547 Make the purge_queue management command more efficient
Before we were removing items individually from the queue.  We now
directly use RabbitMQ's queue purging mechanism.

(imported from commit 62ab52c724c5a221b4c81a967154a4046a579f84)
2013-10-28 14:30:53 -04:00
Jessica McKellar fac9d34d6c digest: Add a management command to enqueue digest email recipients.
(imported from commit 70ff2c7a4dae654f4077041c45e2154b3ac7afb7)
2013-10-25 19:08:47 -04:00
Waseem Daher 66f48288b4 Rename LOCALSERVER -> LOCAL_SERVER.
(imported from commit b3abdd10d54d2ad7a9c463af9a291d2e2127707f)
2013-10-25 17:37:06 -04:00
Luke Faraone a1b44986e0 [schema] Add a base_{api,site}_url property to our deployments.
This will allow us to redirect clients to the correct local site.

To apply this migration, just run:
     python manage.py migrate zilencer 0002

(imported from commit 7bd39b5f035145b6b52e1b2cb2ad5f6720d598ce)
2013-10-25 15:18:37 -04:00
Luke Faraone 81d7dd1fda [schema] Support for authenticating Deployments via the API.
Here we introduce a new Django app, zilencer. The intent is to not have
this app enabled on LOCALSERVER instances, and for it to grow to include
all the functionality we want to have in our central server that isn't
relevant for local deployments.

Currently we have to modify functions in zerver/* to match; in the
future, it would be cool to have the relevant shared code broken out
into a separate library.

This commit inclues both the migration to create the models as well as a
data migration that (for non-LOCALSERVER) creates a single default
Deployment for zulip.com.

To apply this migration to your system, run:
   ./manage.py migrate zilencer

(imported from commit 86d5497ac120e03fa7f298a9cc08b192d5939b43)
2013-10-25 14:13:30 -04:00
Luke Faraone 52309b5789 Enable absolute imports for non-third-party code where it was omitted previously
(imported from commit 34856ac32c6b94b614273a6fe46c87a314058d9b)
2013-10-25 14:13:30 -04:00
Leo Franchi a9b9c0ec7b Add Email Mirror default settings and error out with readable warning if not configured
(imported from commit adac321d09561b51277ec1e314a1d480dc01e0c8)
2013-10-25 11:51:13 -04:00
Leo Franchi 2d276179d0 Refactor email-mirror to handle running on any machine
(imported from commit 2971449ceaacb564770e66874fc095f77e68d445)
2013-10-25 11:37:34 -04:00
Leo Franchi dc8bfde0b4 Add a check_apns_tokens manage.py command to flush old tokens
(imported from commit 15dcfaa1967bd749b582882c4a816a457ca6cc12)
2013-10-24 14:54:31 -04:00
Leo Franchi 05564ec0e2 [schema] Add a config option for enabling/disabling push notifications
(imported from commit 5649188cd5e0e4edb6e71559ec7d7fc5e92efaf8)
2013-10-24 14:54:30 -04:00
Zev Benjamin 5979af3a45 [manual] Add asynchronous message sender via sockjs-tornado
New dependency: sockjs-tornado

One known limitation is that we don't clean up sessions for
non-websockets transports.  This is a bug in Tornado so I'm going to
look at upgrading us to the latest version:
https://github.com/mrjoes/sockjs-tornado/issues/47

(imported from commit 31cdb7596dd5ee094ab006c31757db17dca8899b)
2013-10-22 18:45:11 -04:00
Steve Howell 03c856ce10 Add create-stream management command.
The current version should only be used for testing; for example,
if you want to create a bunch of streams for stress testing, you
can run this in a loop.

(imported from commit ec51a431fb9679fc18379e4c6ecdba66bc75a395)
2013-10-19 09:28:59 -04:00
Jessica McKellar b7419d50b4 create_realm: add a reminder to run set_default_streams.
(imported from commit 59cf9378f8f271a88a6fbda591814fd9142a8d93)
2013-10-17 13:47:51 -04:00
Jessica McKellar 6dbf7613a1 Supply a name when creating realms.
For our populate_db bulk creation, just use the domain.

(imported from commit 4fb756f6dfa2d8f90e55822e27891e84168d5d1c)
2013-10-17 13:47:51 -04:00
Jessica McKellar 648b3bed74 create_realm: don't let us create realms with invalid domains.
(imported from commit 95a75a941b18d0857027084f5f9ea4c011805c9d)
2013-10-17 13:47:51 -04:00
Leo Franchi 753aace8f8 Add a basic statistics module to analyze active user engagement
(imported from commit 4d7a21b9a09b97c88a80d0446fb16e53012507c9)
2013-10-16 18:39:44 -04:00
acrefoot 98897fe5e3 Queue the day 1 and day 2 Zulip followup emails
Handled by the queue processor for signups. Added a management command
that accomplishes the same task, in case it's needed for manually added users,
or in case we goof and need to remove queued emails for a given user.

This addresses Trac #1807

(imported from commit 6727b82a07fa6a3ea3d827860c9e60fd0602297a)
2013-10-10 19:32:21 -04:00
Jessica McKellar 6afdfeb1dc Filter obvious footers from email mirror messages.
(imported from commit 7e22b6823b5d0a07f7f350e4af65be66728a8f88)
2013-10-02 14:39:22 -04:00
Jessica McKellar f0911f51a0 email mirror: ignore decoding errors.
Empirically, we only get these for malformed emails where the charset
specified in a message part header does not match the true encoding of
the part. I checked what the resulting Zulip looked like for the
original offender, and it looked find with ignoring errors.

(imported from commit ac6ba65b611cb22d4ec547b75a585abce6fc50b0)
2013-09-27 11:34:57 -04:00
Tim Abbott e4496521e9 Update logging path for email-mirror.
(imported from commit 75f36dbb963b2b77ecda7ba138023f0f774047e0)
2013-09-25 16:52:41 -04:00
Jessica McKellar 0d897c469d Add a utility to report some rough recent client activity metrics.
(imported from commit 27b4a70871939b2728fcbe0ce5824326ff4decc2)
2013-09-25 14:32:53 -04:00
Zev Benjamin 871afde142 Prepare process_queue to be used with supervisor's numprocs
(imported from commit 5a652b93f1e8b32b5ed89d622035161abaedfb11)
2013-09-24 20:44:15 -04:00
Zev Benjamin e450fc80c9 Move existing queue processing workers to new framework
(imported from commit b448eeefcddb1840344bd337046be30714a7fdbc)
2013-09-24 20:44:15 -04:00
Zev Benjamin 6cf5ebc63d Add new system for defining and running queue-processing workers
(imported from commit 7e77da57b8ad0b70837785c85f546601ba5b1957)
2013-09-24 20:44:15 -04:00
Jessica McKellar d434bf4bb4 email mirror: use a message part's specific charset rather than the message charset.
The overall message charset may be null or not match the part's
charset. Even though it's unclear from the documentation,
experimentally using the charset for a message part seems to give you
the charset even for non-multipart emails.

(imported from commit 0e1d23073f4c53041f9760e66a6635f8a94893d1)
2013-09-24 16:19:53 -04:00
Jessica McKellar 7bb75e3acb Populate date_created for existing realms.
(imported from commit 6a10150905a23589f098778ccbd8075025ff900f)
2013-09-24 16:19:52 -04:00
Zev Benjamin 233290d578 Add a management command to purge a rabbitmq queue
This is useful in debugging when you just want to discard all the
messages in a queue because they have the wrong structure.

(imported from commit 8559ac74f11841430b4d0c801d5506ebcb74c3eb)
2013-09-23 11:25:58 -04:00
Jessica McKellar ee2cd51c4c generate_invite_links: work for open realms too.
(imported from commit ec2143d25cb2f6f4181ece7a25c79666334a1634)
2013-09-20 10:00:10 -04:00
Kevin Mehall 63cff6eb71 Make populate_db set the is_bot flag on bot accounts.
(imported from commit 93a507faa2b19fde05376fd73ed3a65149cfce68)
2013-09-18 16:16:17 -04:00
Jessica McKellar c2e704e4b6 Instead of erroring out, do our best to format HTML-only e-mails.
(imported from commit a58e65864ebb3ef539cf15902b59636cd3912b69)
2013-09-18 15:41:27 -04:00
Jessica McKellar 8ab3ae9da2 email mirror: handle non-ASCII message bodies.
(imported from commit 1a8e21b7ff5441c21147bab85f5a460a7567c5e3)
2013-09-18 15:41:27 -04:00
Jessica McKellar 28f80211ec email mirror: handle non-multipart e-mails when checking for attachments.
(imported from commit 2132d604bb353ef6e8ca2c371479b2b2146d8603)
2013-09-17 16:20:28 -04:00
Jessica McKellar 3db51fa17f Upload and display attachments sent through the email mirror.
(imported from commit b3ac24b71a2e7c354d69ecae9174316102ab8f84)
2013-09-17 10:00:01 -04:00
Jessica McKellar 2c0ba365ae email mirror: show more debugging information on errors.
(imported from commit 57f5b8b57062501443367953239526f1ec78eb7f)
2013-09-17 10:00:00 -04:00
Leo Franchi 2eb8c6aae3 Add documentation and a duration argument to analyze_user_activity.py
(imported from commit 5668ec02a23dcfeb29760b55d8acee17cfdea9cf)
2013-09-10 13:25:59 -04:00
Tim Abbott 9fcca3df4e [schema] Collect data on when users are active on site.
These engagement data will be useful both for making pretty graphs of
how addicted our users are as well as for allowing us to check whether
a new deployment is actually using the product or not.

This measures "number of minutes during which each user had checked
the app within the previous 15 minutes".  It should correctly not
count server-initiated reloads.

It's possible that we should use something less aggressive than
mousemove; I'm a little torn on that because you really can check the
app for new messages without doing anything active.

This is somewhat tested but there are a few outstanding issues:

* Mobile apps don't report these data.  It should be as easy as having
  them send in update_active_status queries with new_user_input=true.

* The semantics of this should be better documented (e.g. the
  management script should print out the spec above)x.

(imported from commit ec8b2dc96b180e1951df00490707ae916887178e)
2013-09-10 13:25:59 -04:00
Leo Franchi a33957db32 Handle mailchimp error 214 (duplicate email) in signup worker
(imported from commit cb34c153fc96bca7c8faed01d019aa2433fcf568)
2013-09-09 10:57:09 -04:00
acrefoot cf3e98d74a use staging for wdaher.com for the email script
(imported from commit fbad47386f5c48785f766886218370612db31c26)
2013-09-05 19:19:41 -04:00
acrefoot 592e1ead6b Actually use the realm specified by the email mirror when doing the markdown
(imported from commit 00b909b1d0c70a3523ab53d2f91815ab14ba2a6f)
2013-09-05 18:24:55 -04:00
Luke Faraone 085fdebc98 Use the X-Gm-Original-To header instead of iterating through To/Cc/Bcc
This will allow mail with an implicit destination (mailing lists, complex
forwarding) to be received by our system.

See http://support.google.com/a/bin/answer.py?hl=en&answer=2368151 for
documentation of Google's behaviour that adds this header.

(imported from commit f8fd500e3c27e12af5941c63c91d5c796a2cd24a)
2013-09-03 17:46:44 -04:00
Jessica McKellar acdfb6afa0 email mirror: make being one of many recipients okay.
Previously the email gateway had to be the only address in a recipient
field or we'd mis-parse the recipient.

This commit also makes the mirror correctly handle addresses of the
form "Jessica McKellar <jesstess@zulip.com>".

(imported from commit 7435f2b59b8f47dc599cc869f64597a730af7d12)
2013-08-29 18:13:21 -04:00
Jessica McKellar c4157ed2b6 Make email mirror testing easier by using a different mailbox locally.
The mirror will use INBOX when deployed and Test locally. Send an
e-mail to the Test mailbox by including the word "localhost" in the
subject; a GMail filter will place it in Test on receipt.

(imported from commit bacf9a9554c8c5e1f3ec8497761edf2c15d3745d)
2013-08-29 18:13:21 -04:00
Jessica McKellar 95c70967d2 rename-stream: support non-ASCII stream names.
(imported from commit 5fe2c803c0645bb6367f39bfe58acc478a66a0eb)
2013-08-28 10:23:39 -04:00
Jessica McKellar 8f3c5dc924 Add a management command to rename a stream.
(imported from commit b3acadc09b289b48e2ac298e50a5427545b6a473)
2013-08-28 10:23:39 -04:00
Kevin Mehall e2a13637f2 Management command for realm emoji
(imported from commit 4de3ac69c34bc6e92d180c672fa8276338fecf53)
2013-08-23 21:39:29 -04:00
Leo Franchi 8484cac5c9 Add a messages/flags POST api call to change a message's flags
(imported from commit b51ebd94c99f57f1cda19039023013209556e343)
2013-08-20 16:19:07 -04:00
Jessica McKellar f3a32d8382 Report email forwarding errors to our internal Zulip.
For now, just do this, and we'll reach out to realms having trouble
manually. We may eventually need to automatically reply to the e-mail,
reach out to a realm admin, etc.

(imported from commit 5c5ac354066f9e9be3fb928e1f8801613c22c1ac)
2013-08-16 16:09:18 -04:00
Jessica McKellar 557bdd2a6e email gateway: Make forwarding even "richly formatted" messages work.
(imported from commit b15f24fa6898eb2eaf7d27bf862e8bead76f544d)
2013-08-16 16:09:18 -04:00
Jessica McKellar cc21429b99 Add a management command to dump all messages on public streams for a realm.
(imported from commit f4f8bfece408b466af4db93b2da15cf69b68e0a3)
2013-08-13 18:16:30 -04:00
Jessica McKellar 15afdf65eb email forwarder: encode stream names so they are safe in email addresses.
(imported from commit 6699d656e3cae58fad062a1403fb9923429fde89)
2013-08-13 14:28:47 -04:00
Jessica McKellar c4a0797413 Decode MIME-encoded emails before forwarding them to Zulip.
Otherwise you'll see garbage.

(imported from commit 07e68a5e209fdea4f54470396a67986dd9fa4016)
2013-08-13 14:28:47 -04:00
Jessica McKellar a878f8a668 populate_db: make emailgateway@zulip.com a default user.
(imported from commit 000d03f59afbb1da125d5d610e1fff6b54d5dbd0)
2013-08-09 15:22:59 -04:00
Jessica McKellar a228bb6671 Implement a v1 email-to-zulip forwarder.
(imported from commit 5f91fd69ee2075413d0d4aedf71b878bf98c640e)
2013-08-09 15:22:58 -04:00
Jessica McKellar 639aac964e Add a management command to generate hashes for all streams that need them.
(imported from commit b3a05e47dfe69b44f984185e360d79bf04f7885b)
2013-08-09 14:59:27 -04:00
Tim Abbott 2ea1bb05a5 Rename ~/.humbugrc to ~/.zuliprc.
(imported from commit a0d53dd20097a56971874dc1d84c6f95267e84f2)
2013-08-08 10:22:31 -04:00
Tim Abbott 4c8fdec554 Rename humbug => zulip in comments.
(imported from commit 26f305fa9351135a1b874f3c2106f03aed7683cb)
2013-08-07 10:00:08 -04:00
Tim Abbott ae615dfb4d Rename humbug => zulip in some settings.
(imported from commit 4d10e6213cf828f16b833ecba98b950972c32442)
2013-08-07 10:00:08 -04:00
Tim Abbott d59ba39518 Rename humbug_finish to zulip_finish.
(imported from commit a73ef9302d16a4068c7d050d4882d2eba699488d)
2013-08-07 10:00:08 -04:00
Tim Abbott 661ca38b58 Change humbug => zulip in some local variables.
(imported from commit 88caa4a87ea0fd269ab741645c124c5d07d69c0a)
2013-08-07 10:00:08 -04:00
Tim Abbott 2db6dc6c88 Change humbug => zulip in populate_db variables.
(imported from commit 14f8e072fd87478e8877925912fde113a75aba2c)
2013-08-07 10:00:08 -04:00
Tim Abbott d8f78a5e2d Fix some remaining references to zephyr.
(imported from commit 2072d3451cc08bc32b997d7fafaafd7b6ac568df)
2013-08-07 10:00:08 -04:00
Tim Abbott 1f1af26048 Change Humbug => Zulip in text/comments.
(imported from commit 2f9d73431ae40e1b9e9e11bc2f4f62f566ae758a)
2013-08-07 10:00:07 -04:00
Tim Abbott e05979d4c4 Rename humbug_realm to zulip_realm.
(imported from commit b36dcc7e5240159357a0f30036394a2af9b85844)
2013-08-07 10:00:07 -04:00
Tim Abbott e111a2f9a5 [manual] Rename Django app from zephyr to zerver.
This needs to be deployed to both staging and prod at the same
off-peak time (and the schema migration run).

At the time it is deployed, we need to make a few changes directly in
the database:

(1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr';
(2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr';

(imported from commit eb3fd719571740189514ef0b884738cb30df1320)
2013-08-06 07:39:36 -04:00